You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by xiongqi wu <xi...@gmail.com> on 2018/11/07 01:30:17 UTC
Re: [VOTE] KIP-354 Time-based log compaction policy
bump
Xiongqi (Wesley) Wu
On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xi...@gmail.com> wrote:
>
> Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for feedback.
> Can I have more feedback or VOTE on this KIP?
>
>
> Xiongqi (Wesley) Wu
>
>
> On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <xi...@gmail.com> wrote:
>
>> Any other votes or comments?
>>
>> Xiongqi (Wesley) Wu
>>
>>
>> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <xi...@gmail.com> wrote:
>>
>>> Yes, more votes and code review.
>>>
>>> Xiongqi (Wesley) Wu
>>>
>>>
>>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann <br...@zendesk.com.invalid>
>>> wrote:
>>>
>>>> +1 (non binding) from on 0 then, and on the KIP.
>>>>
>>>> Where do we go from here? More votes?
>>>>
>>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <cm...@apache.org>
>>>> wrote:
>>>>
>>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
>>>> > > Thank you for comments. I will use '0' for now.
>>>> > >
>>>> > > If we create topics through admin client in the future, we might
>>>> perform
>>>> > > some useful checks. (but the assumption is all brokers in the same
>>>> > cluster
>>>> > > have the same default configurations value, otherwise,it might
>>>> still be
>>>> > > tricky to do such cross validation check.)
>>>> >
>>>> > This isn't something that we might do in the future-- this is
>>>> something we
>>>> > are doing now. We already have Create Topic policies which are
>>>> enforced by
>>>> > the broker. Check KIP-108 and KIP-170 for details. This is one of the
>>>> > motivations for getting rid of direct ZK access-- making sure that
>>>> these
>>>> > policies are applied.
>>>> >
>>>> > I agree that having different configurations on different brokers can
>>>> be
>>>> > confusing and frustrating . That's why more configurations are being
>>>> made
>>>> > dynamic using KIP-226. Dynamic configurations are stored centrally in
>>>> ZK,
>>>> > so they are the same on all brokers (modulo propagation delays). In
>>>> any
>>>> > case, this is a general issue, not specific to "create topics".
>>>> >
>>>> > cheers,
>>>> > Colin
>>>> >
>>>> >
>>>> > >
>>>> > >
>>>> > > Xiongqi (Wesley) Wu
>>>> > >
>>>> > >
>>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <cm...@apache.org>
>>>> > wrote:
>>>> > >
>>>> > > > I don't have a strong opinion. But I think we should probably be
>>>> > > > consistent with how segment.ms works, and just use 0.
>>>> > > >
>>>> > > > best,
>>>> > > > Colin
>>>> > > >
>>>> > > >
>>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
>>>> > > > > OK thanks for that clarification. I see why you're uncomfortable
>>>> > with 0
>>>> > > > now.
>>>> > > > >
>>>> > > > > I'm not really fussed. I just prefer consistency in
>>>> configuration
>>>> > > > options.
>>>> > > > >
>>>> > > > > Personally I lean towards treating 0 and 1 similarly in that
>>>> > scenario,
>>>> > > > > because it favours the person thinking about setting the
>>>> > configurations,
>>>> > > > > and a person doesn't care about a 1ms edge case especially when
>>>> the
>>>> > > > context
>>>> > > > > is the true minimum is tied to the log cleaner cadence.
>>>> > > > >
>>>> > > > > Introducing 0 to mean "disabled" because there is some
>>>> uniquness in
>>>> > > > > segment.ms not being able to be set to 0, reduces configuration
>>>> > > > consistency
>>>> > > > > in favour of capturing a MS gap in an edge case that nobody
>>>> would
>>>> > ever
>>>> > > > > notice. For someone to understand why everywhere else -1 is
>>>> used to
>>>> > > > > disable, but here 0 is used, they would need to learn about
>>>> > segment.ms
>>>> > > > > having a 1ms minimum and then after learning would think "who
>>>> cares
>>>> > about
>>>> > > > > 1ms?" in this context. I would anyway :)
>>>> > > > >
>>>> > > > > my 2c anyway. Will again defer to majority. Curious which way
>>>> Colin
>>>> > falls
>>>> > > > > now.
>>>> > > > >
>>>> > > > > Don't want to spend more time on this though, It's well into
>>>> > > > bikeshedding
>>>> > > > > territory now. :)
>>>> > > > >
>>>> > > > >
>>>> > > > >
>>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <xi...@gmail.com>
>>>> > wrote:
>>>> > > > >
>>>> > > > > > I want to honor the minimum value of segment.ms (which is
>>>> 1ms) to
>>>> > > > force
>>>> > > > > > roll an active segment.
>>>> > > > > > So if we set "max.compaction.lag.ms" any value > 0, the
>>>> minimum of
>>>> > > > > > max.compaction.lag.ms and segment.ms will be used to seal an
>>>> > active
>>>> > > > > > segment.
>>>> > > > > >
>>>> > > > > > If we set max.compaction.lag.ms to 0, the current
>>>> implementation
>>>> > will
>>>> > > > > > treat it as disabled.
>>>> > > > > >
>>>> > > > > > It is a little bit weird to treat max.compaction.lag=0 the
>>>> same as
>>>> > > > > > max.compaction.lag=1.
>>>> > > > > >
>>>> > > > > > There might be a reason why we set the minimum of segment.ms
>>>> to 1,
>>>> > > > and I
>>>> > > > > > don't want to break this assumption.
>>>> > > > > >
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > Xiongqi (Wesley) Wu
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
>>>> > <br...@zendesk.com.invalid>
>>>> > > > > > wrote:
>>>> > > > > >
>>>> > > > > > > You're rolling a new segment if the condition is met right?
>>>> So
>>>> > I'm
>>>> > > > > > > struggling to understand the relevance of segment.ms here.
>>>> > Maybe an
>>>> > > > > > > example
>>>> > > > > > > would help my understanding:
>>>> > > > > > >
>>>> > > > > > > segment.ms=9999999
>>>> > > > > > > *min.cleanable.dirty.ratio=1*
>>>> > > > > > > max.compaction.lag.ms=1
>>>> > > > > > >
>>>> > > > > > > When a duplicate message comes in, after 1ms the topic
>>>> should be
>>>> > > > eligible
>>>> > > > > > > for compaction when the log compaction thread gets around to
>>>> > > > evaluating
>>>> > > > > > the
>>>> > > > > > > topic.
>>>> > > > > > >
>>>> > > > > > > if we have
>>>> > > > > > > segment.ms=9999999
>>>> > > > > > > *min.cleanable.dirty.ratio=1*
>>>> > > > > > > max.compaction.lag.ms=0
>>>> > > > > > >
>>>> > > > > > > When a duplicate message comes in, after 0ms the topic
>>>> should be
>>>> > > > eligible
>>>> > > > > > > for compaction when the log compaction thread gets around to
>>>> > > > evaluating
>>>> > > > > > the
>>>> > > > > > > topic.
>>>> > > > > > >
>>>> > > > > > > In both of those cases the change would mean a new segment
>>>> is
>>>> > rolled
>>>> > > > so
>>>> > > > > > the
>>>> > > > > > > new message would be part of the compaction task. 0 and 1
>>>> are
>>>> > > > practically
>>>> > > > > > > the same meaning since neither is providing an actual
>>>> guarantee
>>>> > at
>>>> > > > such
>>>> > > > > > low
>>>> > > > > > > MS settings, but effectively tying it to both the frequency
>>>> of
>>>> > the
>>>> > > > log
>>>> > > > > > > cleaner running and the priority of the given topic being
>>>> the
>>>> > highest
>>>> > > > > > > priority of all topics that are evaluated for cleaning on
>>>> the
>>>> > next
>>>> > > > cycle.
>>>> > > > > > > You've captured that nuance with careful "skipped" wording
>>>> in
>>>> > the KIP
>>>> > > > > > > here "controls
>>>> > > > > > > the max time interval a message/segment can be skipped for
>>>> log
>>>> > > > > > compaction".
>>>> > > > > > >
>>>> > > > > > > How is 0 different to 1, practically? And how is it
>>>> relating to
>>>> > > > > > segment.ms
>>>> > > > > > > ?
>>>> > > > > > > Is it that you're proposing to have 0 mean "use segment.ms
>>>> > > > instead?" as
>>>> > > > > > a
>>>> > > > > > > kind of third option?
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
>>>> xiongqiwu@gmail.com>
>>>> > > > wrote:
>>>> > > > > > >
>>>> > > > > > > > To make it clear,
>>>> > > > > > > > I don't against using -1 as disabled, but we need to come
>>>> up
>>>> > with
>>>> > > > the
>>>> > > > > > > > meaning of "0".
>>>> > > > > > > > If "0" means immediate compaction, but the actual
>>>> compaction
>>>> > lag
>>>> > > > will
>>>> > > > > > be
>>>> > > > > > > > segment.ms.
>>>> > > > > > > > It has longer lag than setting the value to be half of
>>>> > segment.ms.
>>>> > > > > > > > We cannot provide "0" as max compaction lag.
>>>> > > > > > > >
>>>> > > > > > > > Here are two options.
>>>> > > > > > > > Option 1:
>>>> > > > > > > > Keep 0 as disabled
>>>> > > > > > > > Option 2:
>>>> > > > > > > > -1 (disabled), 0 (max compaction lag = segment.ms), and
>>>> > others.
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > > Xiongqi (Wesley) Wu
>>>> > > > > > > >
>>>> > > > > > > >
>>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
>>>> > > > <br...@zendesk.com.invalid>
>>>> > > > > > > > wrote:
>>>> > > > > > > >
>>>> > > > > > > > > -1 is consistent as "special" with these settings for
>>>> > example:
>>>> > > > > > > > >
>>>> > > > > > > > > log.retention.bytes
>>>> > > > > > > > > socket.received.buffer.bytes
>>>> > > > > > > > > socket.send.buffer.bytes
>>>> > > > > > > > > queued.max.request.bytes
>>>> > > > > > > > > retention.bytes
>>>> > > > > > > > > retention.ms
>>>> > > > > > > > >
>>>> > > > > > > > > and acks.
>>>> > > > > > > > >
>>>> > > > > > > > > Where it may mean no limit, use OS defaults, max (acks),
>>>> > etc. I
>>>> > > > don't
>>>> > > > > > > see
>>>> > > > > > > > > much convention of 0 meaning those things.
>>>> > > > > > > > >
>>>> > > > > > > > > There are some NULLs but it seems convetion there is
>>>> NULL is
>>>> > used
>>>> > > > > > where
>>>> > > > > > > > > there's another setting in the hierarchy that would be
>>>> used
>>>> > > > instead.
>>>> > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann <
>>>> > brann@zendesk.com>
>>>> > > > > > wrote:
>>>> > > > > > > > >
>>>> > > > > > > > > > If segment.ms can't be set to 0, then we're not being
>>>> > > > consistent
>>>> > > > > > > > > > by using 0 for this new setting? I throw out -1 for
>>>> > > > consideration
>>>> > > > > > > > > > again :)
>>>> > > > > > > > > >
>>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu <
>>>> > > > xiongqiwu@gmail.com>
>>>> > > > > > > > wrote:
>>>> > > > > > > > > >
>>>> > > > > > > > > >> Thanks. I will document after PR is merged.
>>>> > > > > > > > > >>
>>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "segment.ms" to
>>>> 1, we
>>>> > > > cannot
>>>> > > > > > set
>>>> > > > > > > "
>>>> > > > > > > > > >> segment.ms" to 0.
>>>> > > > > > > > > >>
>>>> > > > > > > > > >> I also updated the title of this KIP.
>>>> > > > > > > > > >>
>>>> > > > > > > > > >> Xiongqi (Wesley) Wu
>>>> > > > > > > > > >>
>>>> > > > > > > > > >>
>>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett Rann
>>>> > > > > > <brann@zendesk.com.invalid
>>>> > > > > > > >
>>>> > > > > > > > > >> wrote:
>>>> > > > > > > > > >>
>>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm in the
>>>> minority.
>>>> > :)
>>>> > > > Can
>>>> > > > > > we
>>>> > > > > > > > > >> > make sure 0 gets documented as meaning disabled
>>>> here:
>>>> > > > > > > > > >> >
>>>> https://kafka.apache.org/documentation/#brokerconfigs
>>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
>>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
>>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
>>>> > > > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
>>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
>>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
>>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>
>>>> > > > > > > > > >> <
>>>> https://kafka.apache.org/documentation/#brokerconfigs
>>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
>>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
>>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
>>>> > > > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
>>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
>>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
>>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>> ?
>>>> > > > > > > > > >> > And while there it would be good if segment.ms is
>>>> > > > documented
>>>> > > > > > > > > >> > that 0 is disabled too. (there's some hierarchy of
>>>> > configs
>>>> > > > for
>>>> > > > > > > that
>>>> > > > > > > > > too
>>>> > > > > > > > > >> > if its not set and null for others means disabled!)
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi wu <
>>>> > > > xiongqiwu@gmail.com>
>>>> > > > > > > > > wrote:
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> > > If we use 0 to indicate immediate compaction, the
>>>> > > > compaction
>>>> > > > > > lag
>>>> > > > > > > > is
>>>> > > > > > > > > >> > > determined by segment.ms in worst case. If
>>>> segment.ms
>>>> > is
>>>> > > > 24
>>>> > > > > > > > hours,
>>>> > > > > > > > > >> > > "immediate compaction" is a weaker guarantee than
>>>> > setting
>>>> > > > any
>>>> > > > > > > > value
>>>> > > > > > > > > >> less
>>>> > > > > > > > > >> > > than 24 hours. By the definition of "max
>>>> compaction
>>>> > lag",
>>>> > > > we
>>>> > > > > > > > cannot
>>>> > > > > > > > > >> have
>>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate "disable".
>>>> > > > > > > > > >> > >
>>>> > > > > > > > > >> > >
>>>> > > > > > > > > >> > >
>>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
>>>> > > > > > > > > >> > >
>>>> > > > > > > > > >> > >
>>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin McCabe <
>>>> > > > > > cmccabe@apache.org
>>>> > > > > > > >
>>>> > > > > > > > > >> wrote:
>>>> > > > > > > > > >> > >
>>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, Brett Rann
>>>> wrote:
>>>> > > > > > > > > >> > > > > > That's a fair point. We should make 0 =
>>>> > disable, to
>>>> > > > be
>>>> > > > > > > > > >> consistent
>>>> > > > > > > > > >> > > with
>>>> > > > > > > > > >> > > > > the other settings.
>>>> > > > > > > > > >> > > > >
>>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable and when
>>>> seeing
>>>> > it
>>>> > > > in a
>>>> > > > > > > > config
>>>> > > > > > > > > >> it's
>>>> > > > > > > > > >> > > > clear
>>>> > > > > > > > > >> > > > > that it's a special meaning. 0 doesn't have
>>>> to
>>>> > mean
>>>> > > > > > instant,
>>>> > > > > > > > it
>>>> > > > > > > > > >> just
>>>> > > > > > > > > >> > > > means
>>>> > > > > > > > > >> > > > > as quickly as possible. I don't think 0 is
>>>> > intuitive
>>>> > > > for
>>>> > > > > > > > > disabled
>>>> > > > > > > > > >> and
>>>> > > > > > > > > >> > > it
>>>> > > > > > > > > >> > > > > will be confusing. I wasn't aware segment.ms=0
>>>> ==
>>>> > > > > > disabled,
>>>> > > > > > > > > but I
>>>> > > > > > > > > >> > > think
>>>> > > > > > > > > >> > > > > that is also unintuitive.
>>>> > > > > > > > > >> > > >
>>>> > > > > > > > > >> > > > I think there is an argument for keeping these
>>>> two
>>>> > > > > > > > configurations
>>>> > > > > > > > > >> > > > consistent, since they are so similar. I agree
>>>> that
>>>> > 0
>>>> > > > was an
>>>> > > > > > > > > >> > unfortunate
>>>> > > > > > > > > >> > > > choice.,
>>>> > > > > > > > > >> > > >
>>>> > > > > > > > > >> > > > best,
>>>> > > > > > > > > >> > > > Colin
>>>> > > > > > > > > >> > > >
>>>> > > > > > > > > >> > > > >
>>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM Colin McCabe
>>>> <
>>>> > > > > > > > > cmccabe@apache.org>
>>>> > > > > > > > > >> > > wrote:
>>>> > > > > > > > > >> > > > >
>>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47, xiongqi wu
>>>> wrote:
>>>> > > > > > > > > >> > > > > > > Colin,
>>>> > > > > > > > > >> > > > > > > Thank you for comments.
>>>> > > > > > > > > >> > > > > > > see my inline reply below.
>>>> > > > > > > > > >> > > > > > >
>>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
>>>> > > > > > > > > >> > > > > > >
>>>> > > > > > > > > >> > > > > > >
>>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 PM Colin
>>>> McCabe <
>>>> > > > > > > > > >> cmccabe@apache.org>
>>>> > > > > > > > > >> > > > wrote:
>>>> > > > > > > > > >> > > > > > >
>>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > > The name seems a bit ambiguous. Our
>>>> > compaction
>>>> > > > > > > policies
>>>> > > > > > > > > are
>>>> > > > > > > > > >> > > already
>>>> > > > > > > > > >> > > > > > > > time-based, after all. It seems like
>>>> this
>>>> > > > change is
>>>> > > > > > > > > focused
>>>> > > > > > > > > >> > > around
>>>> > > > > > > > > >> > > > > > adding
>>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms." Perhaps
>>>> the KIP
>>>> > > > title
>>>> > > > > > > should
>>>> > > > > > > > > be
>>>> > > > > > > > > >> > > > something
>>>> > > > > > > > > >> > > > > > > > like "add maximum compaction lag time"?
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > > ==========> sure. I will change the
>>>> title.
>>>> > > > > > > > > >> > > > > > >
>>>> > > > > > > > > >> > > > > > > > The active segment is forced to roll
>>>> when
>>>> > > > either "
>>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
>>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (log.roll.ms and
>>>> > > > log.roll.hours)
>>>> > > > > > > has
>>>> > > > > > > > > >> > reached.
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > > If the max.compaction.lag.ms is low,
>>>> it
>>>> > seems
>>>> > > > like
>>>> > > > > > > > > segments
>>>> > > > > > > > > >> > will
>>>> > > > > > > > > >> > > > be
>>>> > > > > > > > > >> > > > > > > > rolled very frequently. This can be a
>>>> > source of
>>>> > > > > > > problems
>>>> > > > > > > > > in
>>>> > > > > > > > > >> the
>>>> > > > > > > > > >> > > > > > cluster,
>>>> > > > > > > > > >> > > > > > > > since creating many different small log
>>>> > segments
>>>> > > > > > > > consumes
>>>> > > > > > > > > a
>>>> > > > > > > > > >> > huge
>>>> > > > > > > > > >> > > > > > amount of
>>>> > > > > > > > > >> > > > > > > > cluster resources. Therefore, I would
>>>> > suggest
>>>> > > > > > adding a
>>>> > > > > > > > > >> > > broker-level
>>>> > > > > > > > > >> > > > > > > > configuration which allows us to set a
>>>> > minimum
>>>> > > > value
>>>> > > > > > > for
>>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If we let
>>>> users set
>>>> > it
>>>> > > > on a
>>>> > > > > > > > > >> per-topic
>>>> > > > > > > > > >> > > > basis,
>>>> > > > > > > > > >> > > > > > > > someone could set a value of 1 ms or
>>>> > something,
>>>> > > > and
>>>> > > > > > > > cause
>>>> > > > > > > > > >> > chaos.
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > > =========> this applies to segment.ms
>>>> as
>>>> > well.
>>>> > > > > > Today
>>>> > > > > > > > > users
>>>> > > > > > > > > >> can
>>>> > > > > > > > > >> > > > set "
>>>> > > > > > > > > >> > > > > > > segment.ms" to a very low value, and
>>>> cause a
>>>> > > > frequent
>>>> > > > > > > > > >> rolling of
>>>> > > > > > > > > >> > > > active
>>>> > > > > > > > > >> > > > > > > segments.
>>>> > > > > > > > > >> > > > > >
>>>> > > > > > > > > >> > > > > > Hi Xiongqi,
>>>> > > > > > > > > >> > > > > >
>>>> > > > > > > > > >> > > > > > I agree that this is an existing problem
>>>> with
>>>> > > > > > segment.ms.
>>>> > > > > > > > > >> However,
>>>> > > > > > > > > >> > > > that
>>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't fix it. As
>>>> you
>>>> > noted,
>>>> > > > > > there
>>>> > > > > > > > > will
>>>> > > > > > > > > >> be
>>>> > > > > > > > > >> > > more
>>>> > > > > > > > > >> > > > > > interest in these topic-level retention
>>>> > settings as
>>>> > > > a
>>>> > > > > > > result
>>>> > > > > > > > > of
>>>> > > > > > > > > >> > GDPR.
>>>> > > > > > > > > >> > > > It
>>>> > > > > > > > > >> > > > > > seems likely that pre-existing problems
>>>> will
>>>> > cause
>>>> > > > more
>>>> > > > > > > > > trouble.
>>>> > > > > > > > > >> > > > > >
>>>> > > > > > > > > >> > > > > > The fix seems relatively straightforward
>>>> here --
>>>> > > > add a
>>>> > > > > > > > > >> broker-level
>>>> > > > > > > > > >> > > > > > minimum segment.ms that overrides
>>>> per-topic
>>>> > > > minimums.
>>>> > > > > > We
>>>> > > > > > > > can
>>>> > > > > > > > > >> also
>>>> > > > > > > > > >> > > fail
>>>> > > > > > > > > >> > > > > > with a helpful error message when someone
>>>> > attempts
>>>> > > > to
>>>> > > > > > set
>>>> > > > > > > an
>>>> > > > > > > > > >> > invalid
>>>> > > > > > > > > >> > > > > > configuration.
>>>> > > > > > > > > >> > > > > >
>>>> > > > > > > > > >> > > > > > > In my option, the minimum of "
>>>> > > > max.compaction.lag.ms"
>>>> > > > > > > > should
>>>> > > > > > > > > >> be
>>>> > > > > > > > > >> > > > > > > based on the minimum of "segment.ms".
>>>> Since
>>>> > > > today the
>>>> > > > > > > > > >> minimum of
>>>> > > > > > > > > >> > > > > > segment.ms
>>>> > > > > > > > > >> > > > > > > is 1, "max.compaction.lag.ms" also
>>>> starts
>>>> > with
>>>> > > > 1. "0"
>>>> > > > > > > > means
>>>> > > > > > > > > >> > > > disable. I
>>>> > > > > > > > > >> > > > > > > can use -1 as disable, but it is hard to
>>>> > define
>>>> > > > the
>>>> > > > > > > > meaning
>>>> > > > > > > > > >> of 0
>>>> > > > > > > > > >> > > > because
>>>> > > > > > > > > >> > > > > > we
>>>> > > > > > > > > >> > > > > > > cannot just roll the active segment
>>>> > immediately.
>>>> > > > > > > > > >> > > > > >
>>>> > > > > > > > > >> > > > > > That's a fair point. We should make 0 =
>>>> > disable, to
>>>> > > > be
>>>> > > > > > > > > >> consistent
>>>> > > > > > > > > >> > > with
>>>> > > > > > > > > >> > > > the
>>>> > > > > > > > > >> > > > > > other settings.
>>>> > > > > > > > > >> > > > > >
>>>> > > > > > > > > >> > > > > > best,
>>>> > > > > > > > > >> > > > > > Colin
>>>> > > > > > > > > >> > > > > >
>>>> > > > > > > > > >> > > > > > >
>>>> > > > > > > > > >> > > > > > > > -- Note that an alternative
>>>> configuration
>>>> > is to
>>>> > > > use
>>>> > > > > > -1
>>>> > > > > > > > as
>>>> > > > > > > > > >> > > > "disabled"
>>>> > > > > > > > > >> > > > > > and
>>>> > > > > > > > > >> > > > > > > > 0
>>>> > > > > > > > > >> > > > > > > > > as "immediate compaction". Because
>>>> > compaction
>>>> > > > lag
>>>> > > > > > is
>>>> > > > > > > > > still
>>>> > > > > > > > > >> > > > determined
>>>> > > > > > > > > >> > > > > > > > > based on min.compaction.lag and how
>>>> long
>>>> > to
>>>> > > > roll
>>>> > > > > > an
>>>> > > > > > > > > active
>>>> > > > > > > > > >> > > > segment,
>>>> > > > > > > > > >> > > > > > > > the
>>>> > > > > > > > > >> > > > > > > > > actual lag for compaction is
>>>> undetermined
>>>> > if
>>>> > > > we
>>>> > > > > > use
>>>> > > > > > > > "0".
>>>> > > > > > > > > >> On
>>>> > > > > > > > > >> > the
>>>> > > > > > > > > >> > > > other
>>>> > > > > > > > > >> > > > > > > > > hand, we can already set
>>>> > > > > > "min.cleanable.dirty.ratio"
>>>> > > > > > > > to
>>>> > > > > > > > > >> > achieve
>>>> > > > > > > > > >> > > > the
>>>> > > > > > > > > >> > > > > > > > same
>>>> > > > > > > > > >> > > > > > > > > goal. So here we choose "0" as
>>>> "disabled".
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be the invalid
>>>> setting.
>>>> > > > > > Treating
>>>> > > > > > > 0
>>>> > > > > > > > > >> > > differently
>>>> > > > > > > > > >> > > > > > than
>>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > > =====> see my previous comment, I am
>>>> not
>>>> > > > strongly
>>>> > > > > > > > against,
>>>> > > > > > > > > >> but
>>>> > > > > > > > > >> > 0
>>>> > > > > > > > > >> > > is
>>>> > > > > > > > > >> > > > > > not a
>>>> > > > > > > > > >> > > > > > > valid configuration in my option. So I
>>>> use
>>>> > "0" as
>>>> > > > > > > disabled
>>>> > > > > > > > > >> state.
>>>> > > > > > > > > >> > > > > > >
>>>> > > > > > > > > >> > > > > > > best,
>>>> > > > > > > > > >> > > > > > > > Colin
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at 15:04, xiongqi
>>>> wu
>>>> > wrote:
>>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
>>>> > > > > > > > > >> > > > > > > > > KIP:
>>>> > > > > > > > > >> > > > > > > > >
>>>> > > > > > > > >
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>>>> > > > > > > > <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
>>>> > > > > > > > > >> <
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>>>> > > > > > > > <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>
>>>> > > > > > > > > >> > > <
>>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>>>> > > > > > > > <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
>>>> > > > > > > > > >> <
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>>>> > > > > > > > <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>>
>>>> > > > > > > > > >> > > > > > <
>>>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>>>> > > > > > > > <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
>>>> > > > > > > > > >> <
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>>>> > > > > > > > <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>
>>>> > > > > > > > > >> > > <
>>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>>>> > > > > > > > <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
>>>> > > > > > > > > >> <
>>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>>>> > > > > > > > <
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>>>
>>>> > > > > > > > > >> > > > > > > > > %3A+Time-based+log+compaction+policy
>>>> > > > > > > > > >> > > > > > > > >
>>>> > > > > > > > > >> > > > > > > > > Implementation:
>>>> > > > > > > > > >> > > > > > > > >
>>>> > > > > > > > > >> > > > > > > > >
>>>> https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>
>>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>>
>>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>
>>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>>>
>>>> > > > > > > > > >> > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>
>>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>>
>>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>
>>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>>>>
>>>> > > > > > > > > >> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>
>>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>>
>>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>
>>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>>>
>>>> > > > > > > > > >> > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>
>>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>>
>>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>
>>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>
>>>> > > > > > <https://github.com/apache/kafka/pull/5611
>>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
>>>> > > > > > > > > >> > > > > > > > >
>>>> > > > > > > > > >> > > > > > > > >
>>>> > > > > > > > > >> > > > > > > > >
>>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
>>>> > > > > > > > > >> > > > > > > >
>>>> > > > > > > > > >> > > > > >
>>>> > > > > > > > > >> > > >
>>>> > > > > > > > > >> > >
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> > --
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> > Brett Rann
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> > Senior DevOps Engineer
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> > Zendesk International Ltd
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000 Australia
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
>>>> > > > > > > > > >> >
>>>> > > > > > > > > >>
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > > --
>>>> > > > > > > > > >
>>>> > > > > > > > > > Brett Rann
>>>> > > > > > > > > >
>>>> > > > > > > > > > Senior DevOps Engineer
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > > Zendesk International Ltd
>>>> > > > > > > > > >
>>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
>>>> > > > > > > > > >
>>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
>>>> > > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > > --
>>>> > > > > > > > >
>>>> > > > > > > > > Brett Rann
>>>> > > > > > > > >
>>>> > > > > > > > > Senior DevOps Engineer
>>>> > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > > Zendesk International Ltd
>>>> > > > > > > > >
>>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
>>>> > > > > > > > >
>>>> > > > > > > > > Mobile: +61 (0) 418 826 017
>>>> > > > > > > > >
>>>> > > > > > > >
>>>> > > > > > >
>>>> > > > > >
>>>> > > >
>>>> >
>>>>
>>>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by xiongqi wu <xi...@gmail.com>.
Colin,
Thanks for comments.
Out of ordered message timestamp is a very good point.
We can combine max.compaction.lag.ms with
log.message.timestamp.difference.max.ms to achieve what we want in an
environment that message timestamp can be shifted a lot.
There are similar discussions regarding log.retention.ms and
log.message.timestamp.difference.max.ms in KAFKA-4340.
I agree that we can always use first message timestamp not the maxTimestamp
of the previous log segment to make it simple.
I have updated the KIP.
Xiongqi (Wesley) Wu
On Tue, Dec 4, 2018 at 5:13 PM Colin McCabe <cm...@apache.org> wrote:
> Hi Xiongqi,
>
> Thinking about this a little bit more, it seems like we don't have any
> guarantees just by looking at the timestamp of the first message in a log
> segment. Similarly, we don't have any guarantees just by looking at the
> maxTimestamp of the previous log segment. Old data could appear anywhere--
> you could put data that was years old in the middle of a segment from 2018.
>
> However, if log.message.timestamp.difference.max.ms is set, then we can
> make some actual guarantees that old data gets purged-- which is what the
> GDPR requires, of course.
>
> Overall, maybe we can make KIP-354 simpler by just always looking at the
> timestamp of the first log message. I don't think looking at the
> maxTimestamp of the previous segment is any more accurate. Aside from
> that, looks good, since we can get what we need with the combination of
> this and log.message.timestamp.difference.max.ms.
>
> best,
> Colin
>
>
> On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote:
> > Thanks for binding and non-binding votes.
> > Can I get one more binding vote?
> >
> > Thanks in advance!
> >
> > Xiongqi (Wesley) Wu
> >
> >
> > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <ma...@frmr.me> wrote:
> >
> > > I'm a +1 (non-binding) — This looks like it would have saved us a lot
> of
> > > pain in an issue we had to debug recently. I can't go into details, but
> > > figuring out how to achieve this effect gave me quite a headache. :)
> > >
> > > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Can I have one more vote on this KIP?
> > > > Any comment is appreciated.
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
> > > >
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > > >
> > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xi...@gmail.com>
> wrote:
> > > >
> > > > > Thanks Dong.
> > > > > I have updated the KIP.
> > > > >
> > > > > Xiongqi (Wesley) Wu
> > > > >
> > > > >
> > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com>
> wrote:
> > > > >
> > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> > > > >>
> > > > >> One minor comment: it may be a bit better to clarify in the public
> > > > >> interface section that the value of the newly added metric is
> > > determined
> > > > >> based by applying that formula across all compactable segments.
> For
> > > > >> example:
> > > > >>
> > > > >> The maximum value of Math.max(now -
> > > > >> earliest_timestamp_in_ms_of_uncompacted_segment -
> > > max.compaction.lag.ms
> > > > ,
> > > > >> 0)/1000 across all compactable partitions, where the
> > > > >> max.compaction.lag.ms
> > > > >> can be overridden on per-topic basis.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > Thanks Joel.
> > > > >> > Tracking the delay at second granularity makes sense
> > > > >> > I have updated KIP.
> > > > >> >
> > > > >> > Xiongqi (Wesley) Wu
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > +1 with one suggestion on the proposed metric. You should
> probably
> > > > >> > include
> > > > >> > > the unit. So for e.g., max-compaction-delay-secs.
> > > > >> > >
> > > > >> > > Joel
> > > > >> > >
> > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <
> xiongqiwu@gmail.com>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > bump
> > > > >> > > > Xiongqi (Wesley) Wu
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <
> xiongqiwu@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for
> > > > >> feedback.
> > > > >> > > > > Can I have more feedback or VOTE on this KIP?
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> > > > xiongqiwu@gmail.com>
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > >> Any other votes or comments?
> > > > >> > > > >>
> > > > >> > > > >> Xiongqi (Wesley) Wu
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> > > > xiongqiwu@gmail.com
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > >>
> > > > >> > > > >>> Yes, more votes and code review.
> > > > >> > > > >>>
> > > > >> > > > >>> Xiongqi (Wesley) Wu
> > > > >> > > > >>>
> > > > >> > > > >>>
> > > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > > > >> > > <brann@zendesk.com.invalid
> > > > >> > > > >
> > > > >> > > > >>> wrote:
> > > > >> > > > >>>
> > > > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > > > >> > > > >>>>
> > > > >> > > > >>>> Where do we go from here? More votes?
> > > > >> > > > >>>>
> > > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> > > > >> cmccabe@apache.org>
> > > > >> > > > >>>> wrote:
> > > > >> > > > >>>>
> > > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > > > >> > > > >>>> > > Thank you for comments. I will use '0' for now.
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > > If we create topics through admin client in the
> future,
> > > > we
> > > > >> > might
> > > > >> > > > >>>> perform
> > > > >> > > > >>>> > > some useful checks. (but the assumption is all
> brokers
> > > in
> > > > >> the
> > > > >> > > same
> > > > >> > > > >>>> > cluster
> > > > >> > > > >>>> > > have the same default configurations value,
> > > otherwise,it
> > > > >> might
> > > > >> > > > >>>> still be
> > > > >> > > > >>>> > > tricky to do such cross validation check.)
> > > > >> > > > >>>> >
> > > > >> > > > >>>> > This isn't something that we might do in the future--
> > > this
> > > > is
> > > > >> > > > >>>> something we
> > > > >> > > > >>>> > are doing now. We already have Create Topic policies
> > > which
> > > > >> are
> > > > >> > > > >>>> enforced by
> > > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details.
> This
> > > is
> > > > >> one
> > > > >> > of
> > > > >> > > > the
> > > > >> > > > >>>> > motivations for getting rid of direct ZK access--
> making
> > > > sure
> > > > >> > that
> > > > >> > > > >>>> these
> > > > >> > > > >>>> > policies are applied.
> > > > >> > > > >>>> >
> > > > >> > > > >>>> > I agree that having different configurations on
> different
> > > > >> > brokers
> > > > >> > > > can
> > > > >> > > > >>>> be
> > > > >> > > > >>>> > confusing and frustrating . That's why more
> > > configurations
> > > > >> are
> > > > >> > > being
> > > > >> > > > >>>> made
> > > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are
> stored
> > > > >> > centrally
> > > > >> > > > in
> > > > >> > > > >>>> ZK,
> > > > >> > > > >>>> > so they are the same on all brokers (modulo
> propagation
> > > > >> delays).
> > > > >> > > In
> > > > >> > > > >>>> any
> > > > >> > > > >>>> > case, this is a general issue, not specific to
> "create
> > > > >> topics".
> > > > >> > > > >>>> >
> > > > >> > > > >>>> > cheers,
> > > > >> > > > >>>> > Colin
> > > > >> > > > >>>> >
> > > > >> > > > >>>> >
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> > > > >> > > cmccabe@apache.org
> > > > >> > > > >
> > > > >> > > > >>>> > wrote:
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > > > I don't have a strong opinion. But I think we
> should
> > > > >> > probably
> > > > >> > > be
> > > > >> > > > >>>> > > > consistent with how segment.ms works, and just
> use
> > > 0.
> > > > >> > > > >>>> > > >
> > > > >> > > > >>>> > > > best,
> > > > >> > > > >>>> > > > Colin
> > > > >> > > > >>>> > > >
> > > > >> > > > >>>> > > >
> > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> > > > >> > > > >>>> > > > > OK thanks for that clarification. I see why
> you're
> > > > >> > > > uncomfortable
> > > > >> > > > >>>> > with 0
> > > > >> > > > >>>> > > > now.
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer
> consistency in
> > > > >> > > > >>>> configuration
> > > > >> > > > >>>> > > > options.
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > Personally I lean towards treating 0 and 1
> > > similarly
> > > > in
> > > > >> > that
> > > > >> > > > >>>> > scenario,
> > > > >> > > > >>>> > > > > because it favours the person thinking about
> > > setting
> > > > >> the
> > > > >> > > > >>>> > configurations,
> > > > >> > > > >>>> > > > > and a person doesn't care about a 1ms edge case
> > > > >> especially
> > > > >> > > > when
> > > > >> > > > >>>> the
> > > > >> > > > >>>> > > > context
> > > > >> > > > >>>> > > > > is the true minimum is tied to the log cleaner
> > > > cadence.
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because there
> is
> > > > some
> > > > >> > > > >>>> uniquness in
> > > > >> > > > >>>> > > > > segment.ms not being able to be set to 0,
> reduces
> > > > >> > > > configuration
> > > > >> > > > >>>> > > > consistency
> > > > >> > > > >>>> > > > > in favour of capturing a MS gap in an edge case
> > > that
> > > > >> > nobody
> > > > >> > > > >>>> would
> > > > >> > > > >>>> > ever
> > > > >> > > > >>>> > > > > notice. For someone to understand why
> everywhere
> > > else
> > > > >> -1
> > > > >> > is
> > > > >> > > > >>>> used to
> > > > >> > > > >>>> > > > > disable, but here 0 is used, they would need to
> > > learn
> > > > >> > about
> > > > >> > > > >>>> > segment.ms
> > > > >> > > > >>>> > > > > having a 1ms minimum and then after learning
> would
> > > > >> think
> > > > >> > > "who
> > > > >> > > > >>>> cares
> > > > >> > > > >>>> > about
> > > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to majority.
> Curious
> > > > >> which
> > > > >> > > way
> > > > >> > > > >>>> Colin
> > > > >> > > > >>>> > falls
> > > > >> > > > >>>> > > > > now.
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > Don't want to spend more time on this though,
> It's
> > > > well
> > > > >> > into
> > > > >> > > > >>>> > > > bikeshedding
> > > > >> > > > >>>> > > > > territory now. :)
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > > > >> > > > xiongqiwu@gmail.com>
> > > > >> > > > >>>> > wrote:
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > > I want to honor the minimum value of
> segment.ms
> > > > >> (which
> > > > >> > is
> > > > >> > > > >>>> 1ms) to
> > > > >> > > > >>>> > > > force
> > > > >> > > > >>>> > > > > > roll an active segment.
> > > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any
> value >
> > > > 0,
> > > > >> the
> > > > >> > > > >>>> minimum of
> > > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be
> > > used
> > > > to
> > > > >> > seal
> > > > >> > > > an
> > > > >> > > > >>>> > active
> > > > >> > > > >>>> > > > > > segment.
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the
> > > current
> > > > >> > > > >>>> implementation
> > > > >> > > > >>>> > will
> > > > >> > > > >>>> > > > > > treat it as disabled.
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > It is a little bit weird to treat
> > > > >> max.compaction.lag=0
> > > > >> > the
> > > > >> > > > >>>> same as
> > > > >> > > > >>>> > > > > > max.compaction.lag=1.
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > There might be a reason why we set the
> minimum of
> > > > >> > > > segment.ms
> > > > >> > > > >>>> to 1,
> > > > >> > > > >>>> > > > and I
> > > > >> > > > >>>> > > > > > don't want to break this assumption.
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > > > >> > > > >>>> > <br...@zendesk.com.invalid>
> > > > >> > > > >>>> > > > > > wrote:
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > > You're rolling a new segment if the
> condition
> > > is
> > > > >> met
> > > > >> > > > right?
> > > > >> > > > >>>> So
> > > > >> > > > >>>> > I'm
> > > > >> > > > >>>> > > > > > > struggling to understand the relevance of
> > > > >> segment.ms
> > > > >> > > > here.
> > > > >> > > > >>>> > Maybe an
> > > > >> > > > >>>> > > > > > > example
> > > > >> > > > >>>> > > > > > > would help my understanding:
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after
> 1ms
> > > the
> > > > >> topic
> > > > >> > > > >>>> should be
> > > > >> > > > >>>> > > > eligible
> > > > >> > > > >>>> > > > > > > for compaction when the log compaction
> thread
> > > > gets
> > > > >> > > around
> > > > >> > > > to
> > > > >> > > > >>>> > > > evaluating
> > > > >> > > > >>>> > > > > > the
> > > > >> > > > >>>> > > > > > > topic.
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > if we have
> > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after
> 0ms
> > > the
> > > > >> topic
> > > > >> > > > >>>> should be
> > > > >> > > > >>>> > > > eligible
> > > > >> > > > >>>> > > > > > > for compaction when the log compaction
> thread
> > > > gets
> > > > >> > > around
> > > > >> > > > to
> > > > >> > > > >>>> > > > evaluating
> > > > >> > > > >>>> > > > > > the
> > > > >> > > > >>>> > > > > > > topic.
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > In both of those cases the change would
> mean a
> > > > new
> > > > >> > > segment
> > > > >> > > > >>>> is
> > > > >> > > > >>>> > rolled
> > > > >> > > > >>>> > > > so
> > > > >> > > > >>>> > > > > > the
> > > > >> > > > >>>> > > > > > > new message would be part of the compaction
> > > > task. 0
> > > > >> > and
> > > > >> > > 1
> > > > >> > > > >>>> are
> > > > >> > > > >>>> > > > practically
> > > > >> > > > >>>> > > > > > > the same meaning since neither is
> providing an
> > > > >> actual
> > > > >> > > > >>>> guarantee
> > > > >> > > > >>>> > at
> > > > >> > > > >>>> > > > such
> > > > >> > > > >>>> > > > > > low
> > > > >> > > > >>>> > > > > > > MS settings, but effectively tying it to
> both
> > > the
> > > > >> > > > frequency
> > > > >> > > > >>>> of
> > > > >> > > > >>>> > the
> > > > >> > > > >>>> > > > log
> > > > >> > > > >>>> > > > > > > cleaner running and the priority of the
> given
> > > > topic
> > > > >> > > being
> > > > >> > > > >>>> the
> > > > >> > > > >>>> > highest
> > > > >> > > > >>>> > > > > > > priority of all topics that are evaluated
> for
> > > > >> cleaning
> > > > >> > > on
> > > > >> > > > >>>> the
> > > > >> > > > >>>> > next
> > > > >> > > > >>>> > > > cycle.
> > > > >> > > > >>>> > > > > > > You've captured that nuance with careful
> > > > "skipped"
> > > > >> > > wording
> > > > >> > > > >>>> in
> > > > >> > > > >>>> > the KIP
> > > > >> > > > >>>> > > > > > > here "controls
> > > > >> > > > >>>> > > > > > > the max time interval a message/segment
> can be
> > > > >> skipped
> > > > >> > > for
> > > > >> > > > >>>> log
> > > > >> > > > >>>> > > > > > compaction".
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > How is 0 different to 1, practically? And
> how
> > > is
> > > > it
> > > > >> > > > >>>> relating to
> > > > >> > > > >>>> > > > > > segment.ms
> > > > >> > > > >>>> > > > > > > ?
> > > > >> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean
> "use
> > > > >> > > > segment.ms
> > > > >> > > > >>>> > > > instead?" as
> > > > >> > > > >>>> > > > > > a
> > > > >> > > > >>>> > > > > > > kind of third option?
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu
> <
> > > > >> > > > >>>> xiongqiwu@gmail.com>
> > > > >> > > > >>>> > > > wrote:
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > > To make it clear,
> > > > >> > > > >>>> > > > > > > > I don't against using -1 as disabled,
> but we
> > > > >> need to
> > > > >> > > > come
> > > > >> > > > >>>> up
> > > > >> > > > >>>> > with
> > > > >> > > > >>>> > > > the
> > > > >> > > > >>>> > > > > > > > meaning of "0".
> > > > >> > > > >>>> > > > > > > > If "0" means immediate compaction, but
> the
> > > > actual
> > > > >> > > > >>>> compaction
> > > > >> > > > >>>> > lag
> > > > >> > > > >>>> > > > will
> > > > >> > > > >>>> > > > > > be
> > > > >> > > > >>>> > > > > > > > segment.ms.
> > > > >> > > > >>>> > > > > > > > It has longer lag than setting the value
> to
> > > be
> > > > >> half
> > > > >> > of
> > > > >> > > > >>>> > segment.ms.
> > > > >> > > > >>>> > > > > > > > We cannot provide "0" as max compaction
> lag.
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > Here are two options.
> > > > >> > > > >>>> > > > > > > > Option 1:
> > > > >> > > > >>>> > > > > > > > Keep 0 as disabled
> > > > >> > > > >>>> > > > > > > > Option 2:
> > > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag =
> > > > >> segment.ms),
> > > > >> > > and
> > > > >> > > > >>>> > others.
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> > > > >> > > > >>>> > > > <br...@zendesk.com.invalid>
> > > > >> > > > >>>> > > > > > > > wrote:
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special" with
> these
> > > > >> settings
> > > > >> > > for
> > > > >> > > > >>>> > example:
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > log.retention.bytes
> > > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > > >> > > > >>>> > > > > > > > > queued.max.request.bytes
> > > > >> > > > >>>> > > > > > > > > retention.bytes
> > > > >> > > > >>>> > > > > > > > > retention.ms
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > and acks.
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS
> > > defaults,
> > > > >> max
> > > > >> > > > (acks),
> > > > >> > > > >>>> > etc. I
> > > > >> > > > >>>> > > > don't
> > > > >> > > > >>>> > > > > > > see
> > > > >> > > > >>>> > > > > > > > > much convention of 0 meaning those
> things.
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > There are some NULLs but it seems
> convetion
> > > > >> there
> > > > >> > is
> > > > >> > > > >>>> NULL is
> > > > >> > > > >>>> > used
> > > > >> > > > >>>> > > > > > where
> > > > >> > > > >>>> > > > > > > > > there's another setting in the
> hierarchy
> > > that
> > > > >> > would
> > > > >> > > be
> > > > >> > > > >>>> used
> > > > >> > > > >>>> > > > instead.
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett
> Rann
> > > <
> > > > >> > > > >>>> > brann@zendesk.com>
> > > > >> > > > >>>> > > > > > wrote:
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0,
> then
> > > > we're
> > > > >> not
> > > > >> > > > being
> > > > >> > > > >>>> > > > consistent
> > > > >> > > > >>>> > > > > > > > > > by using 0 for this new setting? I
> throw
> > > > out
> > > > >> -1
> > > > >> > > for
> > > > >> > > > >>>> > > > consideration
> > > > >> > > > >>>> > > > > > > > > > again :)
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM
> xiongqi
> > > wu
> > > > <
> > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > >> > > > >>>> > > > > > > > wrote:
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is
> > > > merged.
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "
> > > > >> segment.ms"
> > > > >> > > to
> > > > >> > > > >>>> 1, we
> > > > >> > > > >>>> > > > cannot
> > > > >> > > > >>>> > > > > > set
> > > > >> > > > >>>> > > > > > > "
> > > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> I also updated the title of this
> KIP.
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett
> > > Rann
> > > > >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1
> since i'm
> > > > in
> > > > >> the
> > > > >> > > > >>>> minority.
> > > > >> > > > >>>> > :)
> > > > >> > > > >>>> > > > Can
> > > > >> > > > >>>> > > > > > we
> > > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as
> meaning
> > > > >> > disabled
> > > > >> > > > >>>> here:
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >> > > > >>>> > > > > > <
> > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >> > > > >>>> > > > > > <
> > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >> > > > >>>> > > > > > <
> > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >> > > > >>>> > > > > > <
> > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>>
> > > > ?
> > > > >> > > > >>>> > > > > > > > > >> > And while there it would be good
> if
> > > > >> > segment.ms
> > > > >> > > > is
> > > > >> > > > >>>> > > > documented
> > > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's
> some
> > > > >> > hierarchy
> > > > >> > > > of
> > > > >> > > > >>>> > configs
> > > > >> > > > >>>> > > > for
> > > > >> > > > >>>> > > > > > > that
> > > > >> > > > >>>> > > > > > > > > too
> > > > >> > > > >>>> > > > > > > > > >> > if its not set and null for others
> > > means
> > > > >> > > > disabled!)
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM
> xiongqi
> > > > wu
> > > > >> <
> > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > >> > > > >>>> > > > > > > > > wrote:
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate
> immediate
> > > > >> > compaction,
> > > > >> > > > the
> > > > >> > > > >>>> > > > compaction
> > > > >> > > > >>>> > > > > > lag
> > > > >> > > > >>>> > > > > > > > is
> > > > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in
> worst
> > > > >> case. If
> > > > >> > > > >>>> segment.ms
> > > > >> > > > >>>> > is
> > > > >> > > > >>>> > > > 24
> > > > >> > > > >>>> > > > > > > > hours,
> > > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a
> weaker
> > > > >> > guarantee
> > > > >> > > > than
> > > > >> > > > >>>> > setting
> > > > >> > > > >>>> > > > any
> > > > >> > > > >>>> > > > > > > > value
> > > > >> > > > >>>> > > > > > > > > >> less
> > > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the
> definition of
> > > > "max
> > > > >> > > > >>>> compaction
> > > > >> > > > >>>> > lag",
> > > > >> > > > >>>> > > > we
> > > > >> > > > >>>> > > > > > > > cannot
> > > > >> > > > >>>> > > > > > > > > >> have
> > > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate
> > > > >> "disable".
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM
> Colin
> > > > >> > McCabe <
> > > > >> > > > >>>> > > > > > cmccabe@apache.org
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11,
> > > Brett
> > > > >> Rann
> > > > >> > > > >>>> wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We
> should
> > > > >> make
> > > > >> > 0 =
> > > > >> > > > >>>> > disable, to
> > > > >> > > > >>>> > > > be
> > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for
> disable
> > > > and
> > > > >> > when
> > > > >> > > > >>>> seeing
> > > > >> > > > >>>> > it
> > > > >> > > > >>>> > > > in a
> > > > >> > > > >>>> > > > > > > > config
> > > > >> > > > >>>> > > > > > > > > >> it's
> > > > >> > > > >>>> > > > > > > > > >> > > > clear
> > > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special
> meaning. 0
> > > > >> doesn't
> > > > >> > > have
> > > > >> > > > >>>> to
> > > > >> > > > >>>> > mean
> > > > >> > > > >>>> > > > > > instant,
> > > > >> > > > >>>> > > > > > > > it
> > > > >> > > > >>>> > > > > > > > > >> just
> > > > >> > > > >>>> > > > > > > > > >> > > > means
> > > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I
> don't
> > > > >> think 0
> > > > >> > > is
> > > > >> > > > >>>> > intuitive
> > > > >> > > > >>>> > > > for
> > > > >> > > > >>>> > > > > > > > > disabled
> > > > >> > > > >>>> > > > > > > > > >> and
> > > > >> > > > >>>> > > > > > > > > >> > > it
> > > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't
> > > aware
> > > > >> > > > segment.ms=0
> > > > >> > > > >>>> ==
> > > > >> > > > >>>> > > > > > disabled,
> > > > >> > > > >>>> > > > > > > > > but I
> > > > >> > > > >>>> > > > > > > > > >> > > think
> > > > >> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an argument
> for
> > > > >> keeping
> > > > >> > > > these
> > > > >> > > > >>>> two
> > > > >> > > > >>>> > > > > > > > configurations
> > > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they are so
> > > > >> similar. I
> > > > >> > > > agree
> > > > >> > > > >>>> that
> > > > >> > > > >>>> > 0
> > > > >> > > > >>>> > > > was an
> > > > >> > > > >>>> > > > > > > > > >> > unfortunate
> > > > >> > > > >>>> > > > > > > > > >> > > > choice.,
> > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > best,
> > > > >> > > > >>>> > > > > > > > > >> > > > Colin
> > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at
> 11:38 AM
> > > > >> Colin
> > > > >> > > > McCabe
> > > > >> > > > >>>> <
> > > > >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > > > >> > > > >>>> > > > > > > > > >> > > wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at
> 17:47,
> > > > >> > xiongqi
> > > > >> > > wu
> > > > >> > > > >>>> wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply
> below.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at
> 5:24
> > > PM
> > > > >> > Colin
> > > > >> > > > >>>> McCabe <
> > > > >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > > >> > > > >>>> > > > > > > > > >> > > > wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit
> > > > >> ambiguous.
> > > > >> > Our
> > > > >> > > > >>>> > compaction
> > > > >> > > > >>>> > > > > > > policies
> > > > >> > > > >>>> > > > > > > > > are
> > > > >> > > > >>>> > > > > > > > > >> > > already
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after
> all. It
> > > > >> seems
> > > > >> > > like
> > > > >> > > > >>>> this
> > > > >> > > > >>>> > > > change is
> > > > >> > > > >>>> > > > > > > > > focused
> > > > >> > > > >>>> > > > > > > > > >> > > around
> > > > >> > > > >>>> > > > > > > > > >> > > > > > adding
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “
> max.compaction.lag.ms
> > > ."
> > > > >> > Perhaps
> > > > >> > > > >>>> the KIP
> > > > >> > > > >>>> > > > title
> > > > >> > > > >>>> > > > > > > should
> > > > >> > > > >>>> > > > > > > > > be
> > > > >> > > > >>>> > > > > > > > > >> > > > something
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum
> > > compaction
> > > > >> lag
> > > > >> > > > time"?
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I
> will
> > > > >> change
> > > > >> > the
> > > > >> > > > >>>> title.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is
> > > forced
> > > > >> to
> > > > >> > > roll
> > > > >> > > > >>>> when
> > > > >> > > > >>>> > > > either "
> > > > >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (
> > > > >> log.roll.ms
> > > > >> > and
> > > > >> > > > >>>> > > > log.roll.hours)
> > > > >> > > > >>>> > > > > > > has
> > > > >> > > > >>>> > > > > > > > > >> > reached.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> > > > max.compaction.lag.ms
> > > > >> is
> > > > >> > > low,
> > > > >> > > > >>>> it
> > > > >> > > > >>>> > seems
> > > > >> > > > >>>> > > > like
> > > > >> > > > >>>> > > > > > > > > segments
> > > > >> > > > >>>> > > > > > > > > >> > will
> > > > >> > > > >>>> > > > > > > > > >> > > > be
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very
> frequently.
> > > This
> > > > >> can
> > > > >> > > be a
> > > > >> > > > >>>> > source of
> > > > >> > > > >>>> > > > > > > problems
> > > > >> > > > >>>> > > > > > > > > in
> > > > >> > > > >>>> > > > > > > > > >> the
> > > > >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating many
> > > > different
> > > > >> > small
> > > > >> > > > log
> > > > >> > > > >>>> > segments
> > > > >> > > > >>>> > > > > > > > consumes
> > > > >> > > > >>>> > > > > > > > > a
> > > > >> > > > >>>> > > > > > > > > >> > huge
> > > > >> > > > >>>> > > > > > > > > >> > > > > > amount of
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources.
> > > > Therefore,
> > > > >> I
> > > > >> > > would
> > > > >> > > > >>>> > suggest
> > > > >> > > > >>>> > > > > > adding a
> > > > >> > > > >>>> > > > > > > > > >> > > broker-level
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration which
> allows
> > > > us
> > > > >> to
> > > > >> > > set
> > > > >> > > > a
> > > > >> > > > >>>> > minimum
> > > > >> > > > >>>> > > > value
> > > > >> > > > >>>> > > > > > > for
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms.
> If
> > > > we
> > > > >> let
> > > > >> > > > >>>> users set
> > > > >> > > > >>>> > it
> > > > >> > > > >>>> > > > on a
> > > > >> > > > >>>> > > > > > > > > >> per-topic
> > > > >> > > > >>>> > > > > > > > > >> > > > basis,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a
> value
> > > of
> > > > >> 1 ms
> > > > >> > > or
> > > > >> > > > >>>> > something,
> > > > >> > > > >>>> > > > and
> > > > >> > > > >>>> > > > > > > > cause
> > > > >> > > > >>>> > > > > > > > > >> > chaos.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this
> applies to
> > > > >> > > > segment.ms
> > > > >> > > > >>>> as
> > > > >> > > > >>>> > well.
> > > > >> > > > >>>> > > > > > Today
> > > > >> > > > >>>> > > > > > > > > users
> > > > >> > > > >>>> > > > > > > > > >> can
> > > > >> > > > >>>> > > > > > > > > >> > > > set "
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very
> low
> > > > >> value,
> > > > >> > and
> > > > >> > > > >>>> cause a
> > > > >> > > > >>>> > > > frequent
> > > > >> > > > >>>> > > > > > > > > >> rolling of
> > > > >> > > > >>>> > > > > > > > > >> > > > active
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an
> > > existing
> > > > >> > > problem
> > > > >> > > > >>>> with
> > > > >> > > > >>>> > > > > > segment.ms.
> > > > >> > > > >>>> > > > > > > > > >> However,
> > > > >> > > > >>>> > > > > > > > > >> > > > that
> > > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we
> shouldn't
> > > > fix
> > > > >> > it.
> > > > >> > > As
> > > > >> > > > >>>> you
> > > > >> > > > >>>> > noted,
> > > > >> > > > >>>> > > > > > there
> > > > >> > > > >>>> > > > > > > > > will
> > > > >> > > > >>>> > > > > > > > > >> be
> > > > >> > > > >>>> > > > > > > > > >> > > more
> > > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these
> topic-level
> > > > >> > retention
> > > > >> > > > >>>> > settings as
> > > > >> > > > >>>> > > > a
> > > > >> > > > >>>> > > > > > > result
> > > > >> > > > >>>> > > > > > > > > of
> > > > >> > > > >>>> > > > > > > > > >> > GDPR.
> > > > >> > > > >>>> > > > > > > > > >> > > > It
> > > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that
> pre-existing
> > > > >> > problems
> > > > >> > > > >>>> will
> > > > >> > > > >>>> > cause
> > > > >> > > > >>>> > > > more
> > > > >> > > > >>>> > > > > > > > > trouble.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> > > > >> > > straightforward
> > > > >> > > > >>>> here --
> > > > >> > > > >>>> > > > add a
> > > > >> > > > >>>> > > > > > > > > >> broker-level
> > > > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that
> > > > overrides
> > > > >> > > > >>>> per-topic
> > > > >> > > > >>>> > > > minimums.
> > > > >> > > > >>>> > > > > > We
> > > > >> > > > >>>> > > > > > > > can
> > > > >> > > > >>>> > > > > > > > > >> also
> > > > >> > > > >>>> > > > > > > > > >> > > fail
> > > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error
> message
> > > > when
> > > > >> > > someone
> > > > >> > > > >>>> > attempts
> > > > >> > > > >>>> > > > to
> > > > >> > > > >>>> > > > > > set
> > > > >> > > > >>>> > > > > > > an
> > > > >> > > > >>>> > > > > > > > > >> > invalid
> > > > >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the
> minimum
> > > of "
> > > > >> > > > >>>> > > > max.compaction.lag.ms"
> > > > >> > > > >>>> > > > > > > > should
> > > > >> > > > >>>> > > > > > > > > >> be
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of
> "
> > > > >> > segment.ms
> > > > >> > > ".
> > > > >> > > > >>>> Since
> > > > >> > > > >>>> > > > today the
> > > > >> > > > >>>> > > > > > > > > >> minimum of
> > > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "
> > > max.compaction.lag.ms"
> > > > >> also
> > > > >> > > > >>>> starts
> > > > >> > > > >>>> > with
> > > > >> > > > >>>> > > > 1. "0"
> > > > >> > > > >>>> > > > > > > > means
> > > > >> > > > >>>> > > > > > > > > >> > > > disable. I
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable,
> but
> > > it
> > > > is
> > > > >> > hard
> > > > >> > > > to
> > > > >> > > > >>>> > define
> > > > >> > > > >>>> > > > the
> > > > >> > > > >>>> > > > > > > > meaning
> > > > >> > > > >>>> > > > > > > > > >> of 0
> > > > >> > > > >>>> > > > > > > > > >> > > > because
> > > > >> > > > >>>> > > > > > > > > >> > > > > > we
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the
> active
> > > > >> segment
> > > > >> > > > >>>> > immediately.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We
> should
> > > > >> make
> > > > >> > 0 =
> > > > >> > > > >>>> > disable, to
> > > > >> > > > >>>> > > > be
> > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > best,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > Colin
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an
> > > alternative
> > > > >> > > > >>>> configuration
> > > > >> > > > >>>> > is to
> > > > >> > > > >>>> > > > use
> > > > >> > > > >>>> > > > > > -1
> > > > >> > > > >>>> > > > > > > > as
> > > > >> > > > >>>> > > > > > > > > >> > > > "disabled"
> > > > >> > > > >>>> > > > > > > > > >> > > > > > and
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate
> > > compaction".
> > > > >> > > Because
> > > > >> > > > >>>> > compaction
> > > > >> > > > >>>> > > > lag
> > > > >> > > > >>>> > > > > > is
> > > > >> > > > >>>> > > > > > > > > still
> > > > >> > > > >>>> > > > > > > > > >> > > > determined
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> > > > min.compaction.lag
> > > > >> and
> > > > >> > > how
> > > > >> > > > >>>> long
> > > > >> > > > >>>> > to
> > > > >> > > > >>>> > > > roll
> > > > >> > > > >>>> > > > > > an
> > > > >> > > > >>>> > > > > > > > > active
> > > > >> > > > >>>> > > > > > > > > >> > > > segment,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > the
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for
> > > compaction
> > > > is
> > > > >> > > > >>>> undetermined
> > > > >> > > > >>>> > if
> > > > >> > > > >>>> > > > we
> > > > >> > > > >>>> > > > > > use
> > > > >> > > > >>>> > > > > > > > "0".
> > > > >> > > > >>>> > > > > > > > > >> On
> > > > >> > > > >>>> > > > > > > > > >> > the
> > > > >> > > > >>>> > > > > > > > > >> > > > other
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can
> already set
> > > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > > >> > > > >>>> > > > > > > > to
> > > > >> > > > >>>> > > > > > > > > >> > achieve
> > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > same
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we
> choose
> > > > "0"
> > > > >> as
> > > > >> > > > >>>> "disabled".
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to
> be
> > > the
> > > > >> > invalid
> > > > >> > > > >>>> setting.
> > > > >> > > > >>>> > > > > > Treating
> > > > >> > > > >>>> > > > > > > 0
> > > > >> > > > >>>> > > > > > > > > >> > > differently
> > > > >> > > > >>>> > > > > > > > > >> > > > > > than
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous
> > > > >> comment, I
> > > > >> > > am
> > > > >> > > > >>>> not
> > > > >> > > > >>>> > > > strongly
> > > > >> > > > >>>> > > > > > > > against,
> > > > >> > > > >>>> > > > > > > > > >> but
> > > > >> > > > >>>> > > > > > > > > >> > 0
> > > > >> > > > >>>> > > > > > > > > >> > > is
> > > > >> > > > >>>> > > > > > > > > >> > > > > > not a
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in
> my
> > > > >> option.
> > > > >> > So
> > > > >> > > I
> > > > >> > > > >>>> use
> > > > >> > > > >>>> > "0" as
> > > > >> > > > >>>> > > > > > > disabled
> > > > >> > > > >>>> > > > > > > > > >> state.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > best,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018,
> at
> > > > 15:04,
> > > > >> > > > xiongqi
> > > > >> > > > >>>> wu
> > > > >> > > > >>>> > wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this
> KIP.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>>
> > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > >> > > > >>>> > > >
> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > >> > > > >>>> > > > > > >
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>>
> > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > >> > > > >>>> > > >
> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > >>>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > %3A+Time-based+log+compaction+policy
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > >> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > >> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > >> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > --
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > Brett Rann
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC
> 3000
> > > > >> > > Australia
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > --
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > Brett Rann
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC
> 3000
> > > > >> Australia
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > --
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Brett Rann
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> > > > >> Australia
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > >
> > > > >> > > > >>>> >
> > > > >> > > > >>>>
> > > > >> > > > >>>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by xiongqi wu <xi...@gmail.com>.
Thanks Colin, Dong, and Joel for voting.
I have masked this KIP as accepted.
The pull request is up:
https://github.com/apache/kafka/pull/6009
Xiongqi (Wesley) Wu
On Thu, Dec 6, 2018 at 10:47 AM Joel Koshy <jj...@gmail.com> wrote:
> +1 on the updated KIP.
>
> On Wed, Dec 5, 2018 at 11:56 AM Dong Lin <li...@gmail.com> wrote:
>
> > Thanks for the update. +1 (binding)
> >
> > On Wed, Dec 5, 2018 at 8:19 AM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Thanks, Xiongqi Wu. +1 (binding)
> > >
> > > regards,
> > > Colin
> > >
> > >
> > > On Tue, Dec 4, 2018, at 20:58, xiongqi (wesley) wu wrote:
> > > > Colin,
> > > >
> > > > Thanks for comments.
> > > > Out of ordered message timestamp is a very good point.
> > > > We can combine max.compaction.lag.ms with
> > > > log.message.timestamp.difference.max.ms to achieve what we want in
> an
> > > > environment that message timestamp can be shifted a lot.
> > > >
> > > > There are similar discussions regarding log.retention.ms and
> > > > log.message.timestamp.difference.max.ms in KAFKA-4340.
> > > >
> > > > I agree that we can always use first message timestamp not the
> > > maxTimestamp
> > > > of the previous log segment to make it simple.
> > > >
> > > > I have updated the KIP.
> > > >
> > > > Xiongqi (wesley) Wu
> > > >
> > > >
> > > > On Tue, Dec 4, 2018 at 5:13 PM Colin McCabe <cm...@apache.org>
> > wrote:
> > > >
> > > > > Hi Xiongqi,
> > > > >
> > > > > Thinking about this a little bit more, it seems like we don't have
> > any
> > > > > guarantees just by looking at the timestamp of the first message
> in a
> > > log
> > > > > segment. Similarly, we don't have any guarantees just by looking
> at
> > > the
> > > > > maxTimestamp of the previous log segment. Old data could appear
> > > anywhere--
> > > > > you could put data that was years old in the middle of a segment
> from
> > > 2018.
> > > > >
> > > > > However, if log.message.timestamp.difference.max.ms is set, then
> we
> > > can
> > > > > make some actual guarantees that old data gets purged-- which is
> what
> > > the
> > > > > GDPR requires, of course.
> > > > >
> > > > > Overall, maybe we can make KIP-354 simpler by just always looking
> at
> > > the
> > > > > timestamp of the first log message. I don't think looking at the
> > > > > maxTimestamp of the previous segment is any more accurate. Aside
> > from
> > > > > that, looks good, since we can get what we need with the
> combination
> > of
> > > > > this and log.message.timestamp.difference.max.ms.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote:
> > > > > > Thanks for binding and non-binding votes.
> > > > > > Can I get one more binding vote?
> > > > > >
> > > > > > Thanks in advance!
> > > > > >
> > > > > > Xiongqi (Wesley) Wu
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <ma...@frmr.me>
> wrote:
> > > > > >
> > > > > > > I'm a +1 (non-binding) — This looks like it would have saved
> us a
> > > lot
> > > > > of
> > > > > > > pain in an issue we had to debug recently. I can't go into
> > > details, but
> > > > > > > figuring out how to achieve this effect gave me quite a
> headache.
> > > :)
> > > > > > >
> > > > > > > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <
> xiongqiwu@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Can I have one more vote on this KIP?
> > > > > > > > Any comment is appreciated.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
> > > > > > > >
> > > > > > > >
> > > > > > > > Xiongqi (Wesley) Wu
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <
> xiongqiwu@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Dong.
> > > > > > > > > I have updated the KIP.
> > > > > > > > >
> > > > > > > > > Xiongqi (Wesley) Wu
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <
> lindong28@gmail.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> > > > > > > > >>
> > > > > > > > >> One minor comment: it may be a bit better to clarify in
> the
> > > public
> > > > > > > > >> interface section that the value of the newly added metric
> > is
> > > > > > > determined
> > > > > > > > >> based by applying that formula across all compactable
> > > segments.
> > > > > For
> > > > > > > > >> example:
> > > > > > > > >>
> > > > > > > > >> The maximum value of Math.max(now -
> > > > > > > > >> earliest_timestamp_in_ms_of_uncompacted_segment -
> > > > > > > max.compaction.lag.ms
> > > > > > > > ,
> > > > > > > > >> 0)/1000 across all compactable partitions, where the
> > > > > > > > >> max.compaction.lag.ms
> > > > > > > > >> can be overridden on per-topic basis.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <
> > > xiongqiwu@gmail.com>
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> > Thanks Joel.
> > > > > > > > >> > Tracking the delay at second granularity makes sense
> > > > > > > > >> > I have updated KIP.
> > > > > > > > >> >
> > > > > > > > >> > Xiongqi (Wesley) Wu
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <
> > > jjkoshy.w@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > +1 with one suggestion on the proposed metric. You
> > should
> > > > > probably
> > > > > > > > >> > include
> > > > > > > > >> > > the unit. So for e.g., max-compaction-delay-secs.
> > > > > > > > >> > >
> > > > > > > > >> > > Joel
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <
> > > > > xiongqiwu@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > > bump
> > > > > > > > >> > > > Xiongqi (Wesley) Wu
> > > > > > > > >> > > >
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <
> > > > > xiongqiwu@gmail.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and
> > Xiaohe
> > > for
> > > > > > > > >> feedback.
> > > > > > > > >> > > > > Can I have more feedback or VOTE on this KIP?
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Xiongqi (Wesley) Wu
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> > > > > > > > xiongqiwu@gmail.com>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >> Any other votes or comments?
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> Xiongqi (Wesley) Wu
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> > > > > > > > xiongqiwu@gmail.com
> > > > > > > > >> >
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > >>
> > > > > > > > >> > > > >>> Yes, more votes and code review.
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> Xiongqi (Wesley) Wu
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > > > > > > > >> > > <brann@zendesk.com.invalid
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >>> wrote:
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > > >>>> +1 (non binding) from on 0 then, and on the
> KIP.
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> Where do we go from here? More votes?
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> > > > > > > > >> cmccabe@apache.org>
> > > > > > > > >> > > > >>>> wrote:
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu
> > wrote:
> > > > > > > > >> > > > >>>> > > Thank you for comments. I will use '0' for
> > now.
> > > > > > > > >> > > > >>>> > >
> > > > > > > > >> > > > >>>> > > If we create topics through admin client in
> > the
> > > > > future,
> > > > > > > > we
> > > > > > > > >> > might
> > > > > > > > >> > > > >>>> perform
> > > > > > > > >> > > > >>>> > > some useful checks. (but the assumption is
> > all
> > > > > brokers
> > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> > > same
> > > > > > > > >> > > > >>>> > cluster
> > > > > > > > >> > > > >>>> > > have the same default configurations value,
> > > > > > > otherwise,it
> > > > > > > > >> might
> > > > > > > > >> > > > >>>> still be
> > > > > > > > >> > > > >>>> > > tricky to do such cross validation check.)
> > > > > > > > >> > > > >>>> >
> > > > > > > > >> > > > >>>> > This isn't something that we might do in the
> > > future--
> > > > > > > this
> > > > > > > > is
> > > > > > > > >> > > > >>>> something we
> > > > > > > > >> > > > >>>> > are doing now. We already have Create Topic
> > > policies
> > > > > > > which
> > > > > > > > >> are
> > > > > > > > >> > > > >>>> enforced by
> > > > > > > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for
> > > details.
> > > > > This
> > > > > > > is
> > > > > > > > >> one
> > > > > > > > >> > of
> > > > > > > > >> > > > the
> > > > > > > > >> > > > >>>> > motivations for getting rid of direct ZK
> > access--
> > > > > making
> > > > > > > > sure
> > > > > > > > >> > that
> > > > > > > > >> > > > >>>> these
> > > > > > > > >> > > > >>>> > policies are applied.
> > > > > > > > >> > > > >>>> >
> > > > > > > > >> > > > >>>> > I agree that having different configurations
> on
> > > > > different
> > > > > > > > >> > brokers
> > > > > > > > >> > > > can
> > > > > > > > >> > > > >>>> be
> > > > > > > > >> > > > >>>> > confusing and frustrating . That's why more
> > > > > > > configurations
> > > > > > > > >> are
> > > > > > > > >> > > being
> > > > > > > > >> > > > >>>> made
> > > > > > > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations
> > are
> > > > > stored
> > > > > > > > >> > centrally
> > > > > > > > >> > > > in
> > > > > > > > >> > > > >>>> ZK,
> > > > > > > > >> > > > >>>> > so they are the same on all brokers (modulo
> > > > > propagation
> > > > > > > > >> delays).
> > > > > > > > >> > > In
> > > > > > > > >> > > > >>>> any
> > > > > > > > >> > > > >>>> > case, this is a general issue, not specific
> to
> > > > > "create
> > > > > > > > >> topics".
> > > > > > > > >> > > > >>>> >
> > > > > > > > >> > > > >>>> > cheers,
> > > > > > > > >> > > > >>>> > Colin
> > > > > > > > >> > > > >>>> >
> > > > > > > > >> > > > >>>> >
> > > > > > > > >> > > > >>>> > >
> > > > > > > > >> > > > >>>> > >
> > > > > > > > >> > > > >>>> > > Xiongqi (Wesley) Wu
> > > > > > > > >> > > > >>>> > >
> > > > > > > > >> > > > >>>> > >
> > > > > > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin
> > McCabe <
> > > > > > > > >> > > cmccabe@apache.org
> > > > > > > > >> > > > >
> > > > > > > > >> > > > >>>> > wrote:
> > > > > > > > >> > > > >>>> > >
> > > > > > > > >> > > > >>>> > > > I don't have a strong opinion. But I
> think
> > we
> > > > > should
> > > > > > > > >> > probably
> > > > > > > > >> > > be
> > > > > > > > >> > > > >>>> > > > consistent with how segment.ms works,
> and
> > > just
> > > > > use
> > > > > > > 0.
> > > > > > > > >> > > > >>>> > > >
> > > > > > > > >> > > > >>>> > > > best,
> > > > > > > > >> > > > >>>> > > > Colin
> > > > > > > > >> > > > >>>> > > >
> > > > > > > > >> > > > >>>> > > >
> > > > > > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann
> > > wrote:
> > > > > > > > >> > > > >>>> > > > > OK thanks for that clarification. I see
> > why
> > > > > you're
> > > > > > > > >> > > > uncomfortable
> > > > > > > > >> > > > >>>> > with 0
> > > > > > > > >> > > > >>>> > > > now.
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer
> > > > > consistency in
> > > > > > > > >> > > > >>>> configuration
> > > > > > > > >> > > > >>>> > > > options.
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > > Personally I lean towards treating 0
> and
> > 1
> > > > > > > similarly
> > > > > > > > in
> > > > > > > > >> > that
> > > > > > > > >> > > > >>>> > scenario,
> > > > > > > > >> > > > >>>> > > > > because it favours the person thinking
> > > about
> > > > > > > setting
> > > > > > > > >> the
> > > > > > > > >> > > > >>>> > configurations,
> > > > > > > > >> > > > >>>> > > > > and a person doesn't care about a 1ms
> > edge
> > > case
> > > > > > > > >> especially
> > > > > > > > >> > > > when
> > > > > > > > >> > > > >>>> the
> > > > > > > > >> > > > >>>> > > > context
> > > > > > > > >> > > > >>>> > > > > is the true minimum is tied to the log
> > > cleaner
> > > > > > > > cadence.
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled"
> because
> > > there
> > > > > is
> > > > > > > > some
> > > > > > > > >> > > > >>>> uniquness in
> > > > > > > > >> > > > >>>> > > > > segment.ms not being able to be set to
> > 0,
> > > > > reduces
> > > > > > > > >> > > > configuration
> > > > > > > > >> > > > >>>> > > > consistency
> > > > > > > > >> > > > >>>> > > > > in favour of capturing a MS gap in an
> > edge
> > > case
> > > > > > > that
> > > > > > > > >> > nobody
> > > > > > > > >> > > > >>>> would
> > > > > > > > >> > > > >>>> > ever
> > > > > > > > >> > > > >>>> > > > > notice. For someone to understand why
> > > > > everywhere
> > > > > > > else
> > > > > > > > >> -1
> > > > > > > > >> > is
> > > > > > > > >> > > > >>>> used to
> > > > > > > > >> > > > >>>> > > > > disable, but here 0 is used, they would
> > > need to
> > > > > > > learn
> > > > > > > > >> > about
> > > > > > > > >> > > > >>>> > segment.ms
> > > > > > > > >> > > > >>>> > > > > having a 1ms minimum and then after
> > > learning
> > > > > would
> > > > > > > > >> think
> > > > > > > > >> > > "who
> > > > > > > > >> > > > >>>> cares
> > > > > > > > >> > > > >>>> > about
> > > > > > > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway
> :)
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to
> > majority.
> > > > > Curious
> > > > > > > > >> which
> > > > > > > > >> > > way
> > > > > > > > >> > > > >>>> Colin
> > > > > > > > >> > > > >>>> > falls
> > > > > > > > >> > > > >>>> > > > > now.
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > > Don't want to spend more time on this
> > > though,
> > > > > It's
> > > > > > > > well
> > > > > > > > >> > into
> > > > > > > > >> > > > >>>> > > > bikeshedding
> > > > > > > > >> > > > >>>> > > > > territory now. :)
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi
> > wu <
> > > > > > > > >> > > > xiongqiwu@gmail.com>
> > > > > > > > >> > > > >>>> > wrote:
> > > > > > > > >> > > > >>>> > > > >
> > > > > > > > >> > > > >>>> > > > > > I want to honor the minimum value of
> > > > > segment.ms
> > > > > > > > >> (which
> > > > > > > > >> > is
> > > > > > > > >> > > > >>>> 1ms) to
> > > > > > > > >> > > > >>>> > > > force
> > > > > > > > >> > > > >>>> > > > > > roll an active segment.
> > > > > > > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms"
> > any
> > > > > value >
> > > > > > > > 0,
> > > > > > > > >> the
> > > > > > > > >> > > > >>>> minimum of
> > > > > > > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms
> > > will be
> > > > > > > used
> > > > > > > > to
> > > > > > > > >> > seal
> > > > > > > > >> > > > an
> > > > > > > > >> > > > >>>> > active
> > > > > > > > >> > > > >>>> > > > > > segment.
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to
> 0,
> > > the
> > > > > > > current
> > > > > > > > >> > > > >>>> implementation
> > > > > > > > >> > > > >>>> > will
> > > > > > > > >> > > > >>>> > > > > > treat it as disabled.
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > > It is a little bit weird to treat
> > > > > > > > >> max.compaction.lag=0
> > > > > > > > >> > the
> > > > > > > > >> > > > >>>> same as
> > > > > > > > >> > > > >>>> > > > > > max.compaction.lag=1.
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > > There might be a reason why we set
> the
> > > > > minimum of
> > > > > > > > >> > > > segment.ms
> > > > > > > > >> > > > >>>> to 1,
> > > > > > > > >> > > > >>>> > > > and I
> > > > > > > > >> > > > >>>> > > > > > don't want to break this assumption.
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett
> > Rann
> > > > > > > > >> > > > >>>> > <br...@zendesk.com.invalid>
> > > > > > > > >> > > > >>>> > > > > > wrote:
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > You're rolling a new segment if the
> > > > > condition
> > > > > > > is
> > > > > > > > >> met
> > > > > > > > >> > > > right?
> > > > > > > > >> > > > >>>> So
> > > > > > > > >> > > > >>>> > I'm
> > > > > > > > >> > > > >>>> > > > > > > struggling to understand the
> > relevance
> > > of
> > > > > > > > >> segment.ms
> > > > > > > > >> > > > here.
> > > > > > > > >> > > > >>>> > Maybe an
> > > > > > > > >> > > > >>>> > > > > > > example
> > > > > > > > >> > > > >>>> > > > > > > would help my understanding:
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in,
> > > after
> > > > > 1ms
> > > > > > > the
> > > > > > > > >> topic
> > > > > > > > >> > > > >>>> should be
> > > > > > > > >> > > > >>>> > > > eligible
> > > > > > > > >> > > > >>>> > > > > > > for compaction when the log
> > compaction
> > > > > thread
> > > > > > > > gets
> > > > > > > > >> > > around
> > > > > > > > >> > > > to
> > > > > > > > >> > > > >>>> > > > evaluating
> > > > > > > > >> > > > >>>> > > > > > the
> > > > > > > > >> > > > >>>> > > > > > > topic.
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > if we have
> > > > > > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in,
> > > after
> > > > > 0ms
> > > > > > > the
> > > > > > > > >> topic
> > > > > > > > >> > > > >>>> should be
> > > > > > > > >> > > > >>>> > > > eligible
> > > > > > > > >> > > > >>>> > > > > > > for compaction when the log
> > compaction
> > > > > thread
> > > > > > > > gets
> > > > > > > > >> > > around
> > > > > > > > >> > > > to
> > > > > > > > >> > > > >>>> > > > evaluating
> > > > > > > > >> > > > >>>> > > > > > the
> > > > > > > > >> > > > >>>> > > > > > > topic.
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > In both of those cases the change
> > would
> > > > > mean a
> > > > > > > > new
> > > > > > > > >> > > segment
> > > > > > > > >> > > > >>>> is
> > > > > > > > >> > > > >>>> > rolled
> > > > > > > > >> > > > >>>> > > > so
> > > > > > > > >> > > > >>>> > > > > > the
> > > > > > > > >> > > > >>>> > > > > > > new message would be part of the
> > > compaction
> > > > > > > > task. 0
> > > > > > > > >> > and
> > > > > > > > >> > > 1
> > > > > > > > >> > > > >>>> are
> > > > > > > > >> > > > >>>> > > > practically
> > > > > > > > >> > > > >>>> > > > > > > the same meaning since neither is
> > > > > providing an
> > > > > > > > >> actual
> > > > > > > > >> > > > >>>> guarantee
> > > > > > > > >> > > > >>>> > at
> > > > > > > > >> > > > >>>> > > > such
> > > > > > > > >> > > > >>>> > > > > > low
> > > > > > > > >> > > > >>>> > > > > > > MS settings, but effectively tying
> it
> > > to
> > > > > both
> > > > > > > the
> > > > > > > > >> > > > frequency
> > > > > > > > >> > > > >>>> of
> > > > > > > > >> > > > >>>> > the
> > > > > > > > >> > > > >>>> > > > log
> > > > > > > > >> > > > >>>> > > > > > > cleaner running and the priority of
> > the
> > > > > given
> > > > > > > > topic
> > > > > > > > >> > > being
> > > > > > > > >> > > > >>>> the
> > > > > > > > >> > > > >>>> > highest
> > > > > > > > >> > > > >>>> > > > > > > priority of all topics that are
> > > evaluated
> > > > > for
> > > > > > > > >> cleaning
> > > > > > > > >> > > on
> > > > > > > > >> > > > >>>> the
> > > > > > > > >> > > > >>>> > next
> > > > > > > > >> > > > >>>> > > > cycle.
> > > > > > > > >> > > > >>>> > > > > > > You've captured that nuance with
> > > careful
> > > > > > > > "skipped"
> > > > > > > > >> > > wording
> > > > > > > > >> > > > >>>> in
> > > > > > > > >> > > > >>>> > the KIP
> > > > > > > > >> > > > >>>> > > > > > > here "controls
> > > > > > > > >> > > > >>>> > > > > > > the max time interval a
> > message/segment
> > > > > can be
> > > > > > > > >> skipped
> > > > > > > > >> > > for
> > > > > > > > >> > > > >>>> log
> > > > > > > > >> > > > >>>> > > > > > compaction".
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > How is 0 different to 1,
> practically?
> > > And
> > > > > how
> > > > > > > is
> > > > > > > > it
> > > > > > > > >> > > > >>>> relating to
> > > > > > > > >> > > > >>>> > > > > > segment.ms
> > > > > > > > >> > > > >>>> > > > > > > ?
> > > > > > > > >> > > > >>>> > > > > > > Is it that you're proposing to
> have 0
> > > mean
> > > > > "use
> > > > > > > > >> > > > segment.ms
> > > > > > > > >> > > > >>>> > > > instead?" as
> > > > > > > > >> > > > >>>> > > > > > a
> > > > > > > > >> > > > >>>> > > > > > > kind of third option?
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM
> > > xiongqi wu
> > > > > <
> > > > > > > > >> > > > >>>> xiongqiwu@gmail.com>
> > > > > > > > >> > > > >>>> > > > wrote:
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > To make it clear,
> > > > > > > > >> > > > >>>> > > > > > > > I don't against using -1 as
> > disabled,
> > > > > but we
> > > > > > > > >> need to
> > > > > > > > >> > > > come
> > > > > > > > >> > > > >>>> up
> > > > > > > > >> > > > >>>> > with
> > > > > > > > >> > > > >>>> > > > the
> > > > > > > > >> > > > >>>> > > > > > > > meaning of "0".
> > > > > > > > >> > > > >>>> > > > > > > > If "0" means immediate
> compaction,
> > > but
> > > > > the
> > > > > > > > actual
> > > > > > > > >> > > > >>>> compaction
> > > > > > > > >> > > > >>>> > lag
> > > > > > > > >> > > > >>>> > > > will
> > > > > > > > >> > > > >>>> > > > > > be
> > > > > > > > >> > > > >>>> > > > > > > > segment.ms.
> > > > > > > > >> > > > >>>> > > > > > > > It has longer lag than setting
> the
> > > value
> > > > > to
> > > > > > > be
> > > > > > > > >> half
> > > > > > > > >> > of
> > > > > > > > >> > > > >>>> > segment.ms.
> > > > > > > > >> > > > >>>> > > > > > > > We cannot provide "0" as max
> > > compaction
> > > > > lag.
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > Here are two options.
> > > > > > > > >> > > > >>>> > > > > > > > Option 1:
> > > > > > > > >> > > > >>>> > > > > > > > Keep 0 as disabled
> > > > > > > > >> > > > >>>> > > > > > > > Option 2:
> > > > > > > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction
> > lag
> > > =
> > > > > > > > >> segment.ms),
> > > > > > > > >> > > and
> > > > > > > > >> > > > >>>> > others.
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM
> > Brett
> > > Rann
> > > > > > > > >> > > > >>>> > > > <br...@zendesk.com.invalid>
> > > > > > > > >> > > > >>>> > > > > > > > wrote:
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special"
> > with
> > > > > these
> > > > > > > > >> settings
> > > > > > > > >> > > for
> > > > > > > > >> > > > >>>> > example:
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > log.retention.bytes
> > > > > > > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > > > > > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > > > > > > >> > > > >>>> > > > > > > > > queued.max.request.bytes
> > > > > > > > >> > > > >>>> > > > > > > > > retention.bytes
> > > > > > > > >> > > > >>>> > > > > > > > > retention.ms
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > and acks.
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use
> > OS
> > > > > > > defaults,
> > > > > > > > >> max
> > > > > > > > >> > > > (acks),
> > > > > > > > >> > > > >>>> > etc. I
> > > > > > > > >> > > > >>>> > > > don't
> > > > > > > > >> > > > >>>> > > > > > > see
> > > > > > > > >> > > > >>>> > > > > > > > > much convention of 0 meaning
> > those
> > > > > things.
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > There are some NULLs but it
> seems
> > > > > convetion
> > > > > > > > >> there
> > > > > > > > >> > is
> > > > > > > > >> > > > >>>> NULL is
> > > > > > > > >> > > > >>>> > used
> > > > > > > > >> > > > >>>> > > > > > where
> > > > > > > > >> > > > >>>> > > > > > > > > there's another setting in the
> > > > > hierarchy
> > > > > > > that
> > > > > > > > >> > would
> > > > > > > > >> > > be
> > > > > > > > >> > > > >>>> used
> > > > > > > > >> > > > >>>> > > > instead.
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM
> > > Brett
> > > > > Rann
> > > > > > > <
> > > > > > > > >> > > > >>>> > brann@zendesk.com>
> > > > > > > > >> > > > >>>> > > > > > wrote:
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set
> to
> > 0,
> > > > > then
> > > > > > > > we're
> > > > > > > > >> not
> > > > > > > > >> > > > being
> > > > > > > > >> > > > >>>> > > > consistent
> > > > > > > > >> > > > >>>> > > > > > > > > > by using 0 for this new
> > setting?
> > > I
> > > > > throw
> > > > > > > > out
> > > > > > > > >> -1
> > > > > > > > >> > > for
> > > > > > > > >> > > > >>>> > > > consideration
> > > > > > > > >> > > > >>>> > > > > > > > > > again :)
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03
> AM
> > > > > xiongqi
> > > > > > > wu
> > > > > > > > <
> > > > > > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > > > > > >> > > > >>>> > > > > > > > wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document
> after
> > > PR is
> > > > > > > > merged.
> > > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the
> minimum
> > > of "
> > > > > > > > >> segment.ms"
> > > > > > > > >> > > to
> > > > > > > > >> > > > >>>> 1, we
> > > > > > > > >> > > > >>>> > > > cannot
> > > > > > > > >> > > > >>>> > > > > > set
> > > > > > > > >> > > > >>>> > > > > > > "
> > > > > > > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > > >> I also updated the title of
> > this
> > > > > KIP.
> > > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34
> PM
> > > Brett
> > > > > > > Rann
> > > > > > > > >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on
> -1
> > > > > since i'm
> > > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> > > > >>>> minority.
> > > > > > > > >> > > > >>>> > :)
> > > > > > > > >> > > > >>>> > > > Can
> > > > > > > > >> > > > >>>> > > > > > we
> > > > > > > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets
> documented
> > as
> > > > > meaning
> > > > > > > > >> > disabled
> > > > > > > > >> > > > >>>> here:
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>>
> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > > >> > > > >>>> > <
> > > > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > > >> > > > >>>> > <
> > > > > https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > > >> > > > >>>> > <
> > > > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > > >> > > > >>>> > <
> > > > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > >> > > > >>>>
> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > > >> > > > >>>> > <
> > > > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > > >> > > > >>>> > <
> > > > > https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > > >> > > > >>>> > <
> > > > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > > >> > > > >>>> > <
> > > > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >>>>
> > > > > > > > ?
> > > > > > > > >> > > > >>>> > > > > > > > > >> > And while there it would
> be
> > > good
> > > > > if
> > > > > > > > >> > segment.ms
> > > > > > > > >> > > > is
> > > > > > > > >> > > > >>>> > > > documented
> > > > > > > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too.
> > > (there's
> > > > > some
> > > > > > > > >> > hierarchy
> > > > > > > > >> > > > of
> > > > > > > > >> > > > >>>> > configs
> > > > > > > > >> > > > >>>> > > > for
> > > > > > > > >> > > > >>>> > > > > > > that
> > > > > > > > >> > > > >>>> > > > > > > > > too
> > > > > > > > >> > > > >>>> > > > > > > > > >> > if its not set and null
> for
> > > others
> > > > > > > means
> > > > > > > > >> > > > disabled!)
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at
> 4:44
> > AM
> > > > > xiongqi
> > > > > > > > wu
> > > > > > > > >> <
> > > > > > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > > > > > >> > > > >>>> > > > > > > > > wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate
> > > > > immediate
> > > > > > > > >> > compaction,
> > > > > > > > >> > > > the
> > > > > > > > >> > > > >>>> > > > compaction
> > > > > > > > >> > > > >>>> > > > > > lag
> > > > > > > > >> > > > >>>> > > > > > > > is
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > determined by
> segment.ms
> > in
> > > > > worst
> > > > > > > > >> case. If
> > > > > > > > >> > > > >>>> segment.ms
> > > > > > > > >> > > > >>>> > is
> > > > > > > > >> > > > >>>> > > > 24
> > > > > > > > >> > > > >>>> > > > > > > > hours,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction"
> is
> > a
> > > > > weaker
> > > > > > > > >> > guarantee
> > > > > > > > >> > > > than
> > > > > > > > >> > > > >>>> > setting
> > > > > > > > >> > > > >>>> > > > any
> > > > > > > > >> > > > >>>> > > > > > > > value
> > > > > > > > >> > > > >>>> > > > > > > > > >> less
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the
> > > > > definition of
> > > > > > > > "max
> > > > > > > > >> > > > >>>> compaction
> > > > > > > > >> > > > >>>> > lag",
> > > > > > > > >> > > > >>>> > > > we
> > > > > > > > >> > > > >>>> > > > > > > > cannot
> > > > > > > > >> > > > >>>> > > > > > > > > >> have
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to
> > > indicate
> > > > > > > > >> "disable".
> > > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at
> > 8:34
> > > AM
> > > > > Colin
> > > > > > > > >> > McCabe <
> > > > > > > > >> > > > >>>> > > > > > cmccabe@apache.org
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018,
> at
> > > 22:11,
> > > > > > > Brett
> > > > > > > > >> Rann
> > > > > > > > >> > > > >>>> wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair
> point.
> > > We
> > > > > should
> > > > > > > > >> make
> > > > > > > > >> > 0 =
> > > > > > > > >> > > > >>>> > disable, to
> > > > > > > > >> > > > >>>> > > > be
> > > > > > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere
> > for
> > > > > disable
> > > > > > > > and
> > > > > > > > >> > when
> > > > > > > > >> > > > >>>> seeing
> > > > > > > > >> > > > >>>> > it
> > > > > > > > >> > > > >>>> > > > in a
> > > > > > > > >> > > > >>>> > > > > > > > config
> > > > > > > > >> > > > >>>> > > > > > > > > >> it's
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > clear
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special
> > > > > meaning. 0
> > > > > > > > >> doesn't
> > > > > > > > >> > > have
> > > > > > > > >> > > > >>>> to
> > > > > > > > >> > > > >>>> > mean
> > > > > > > > >> > > > >>>> > > > > > instant,
> > > > > > > > >> > > > >>>> > > > > > > > it
> > > > > > > > >> > > > >>>> > > > > > > > > >> just
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > means
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as
> > possible.
> > > I
> > > > > don't
> > > > > > > > >> think 0
> > > > > > > > >> > > is
> > > > > > > > >> > > > >>>> > intuitive
> > > > > > > > >> > > > >>>> > > > for
> > > > > > > > >> > > > >>>> > > > > > > > > disabled
> > > > > > > > >> > > > >>>> > > > > > > > > >> and
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > it
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I
> > > wasn't
> > > > > > > aware
> > > > > > > > >> > > > segment.ms=0
> > > > > > > > >> > > > >>>> ==
> > > > > > > > >> > > > >>>> > > > > > disabled,
> > > > > > > > >> > > > >>>> > > > > > > > > but I
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > think
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > that is also
> > > unintuitive.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an
> > > argument
> > > > > for
> > > > > > > > >> keeping
> > > > > > > > >> > > > these
> > > > > > > > >> > > > >>>> two
> > > > > > > > >> > > > >>>> > > > > > > > configurations
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they
> > > are so
> > > > > > > > >> similar. I
> > > > > > > > >> > > > agree
> > > > > > > > >> > > > >>>> that
> > > > > > > > >> > > > >>>> > 0
> > > > > > > > >> > > > >>>> > > > was an
> > > > > > > > >> > > > >>>> > > > > > > > > >> > unfortunate
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > choice.,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > best,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > Colin
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018
> at
> > > > > 11:38 AM
> > > > > > > > >> Colin
> > > > > > > > >> > > > McCabe
> > > > > > > > >> > > > >>>> <
> > > > > > > > >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4,
> 2018,
> > > at
> > > > > 17:47,
> > > > > > > > >> > xiongqi
> > > > > > > > >> > > wu
> > > > > > > > >> > > > >>>> wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for
> > > comments.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline
> > reply
> > > > > below.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley)
> > Wu
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4,
> > 2018
> > > at
> > > > > 5:24
> > > > > > > PM
> > > > > > > > >> > Colin
> > > > > > > > >> > > > >>>> McCabe <
> > > > > > > > >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for
> this
> > > KIP.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name
> seems a
> > > bit
> > > > > > > > >> ambiguous.
> > > > > > > > >> > Our
> > > > > > > > >> > > > >>>> > compaction
> > > > > > > > >> > > > >>>> > > > > > > policies
> > > > > > > > >> > > > >>>> > > > > > > > > are
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > already
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based,
> > after
> > > > > all. It
> > > > > > > > >> seems
> > > > > > > > >> > > like
> > > > > > > > >> > > > >>>> this
> > > > > > > > >> > > > >>>> > > > change is
> > > > > > > > >> > > > >>>> > > > > > > > > focused
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > around
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > adding
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “
> > > > > max.compaction.lag.ms
> > > > > > > ."
> > > > > > > > >> > Perhaps
> > > > > > > > >> > > > >>>> the KIP
> > > > > > > > >> > > > >>>> > > > title
> > > > > > > > >> > > > >>>> > > > > > > should
> > > > > > > > >> > > > >>>> > > > > > > > > be
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > something
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add
> > maximum
> > > > > > > compaction
> > > > > > > > >> lag
> > > > > > > > >> > > > time"?
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========>
> > sure.
> > > I
> > > > > will
> > > > > > > > >> change
> > > > > > > > >> > the
> > > > > > > > >> > > > >>>> title.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active
> > > segment is
> > > > > > > forced
> > > > > > > > >> to
> > > > > > > > >> > > roll
> > > > > > > > >> > > > >>>> when
> > > > > > > > >> > > > >>>> > > > either "
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > max.compaction.lag.ms
> > > "
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "
> > segment.ms"
> > > (
> > > > > > > > >> log.roll.ms
> > > > > > > > >> > and
> > > > > > > > >> > > > >>>> > > > log.roll.hours)
> > > > > > > > >> > > > >>>> > > > > > > has
> > > > > > > > >> > > > >>>> > > > > > > > > >> > reached.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> > > > > > > > max.compaction.lag.ms
> > > > > > > > >> is
> > > > > > > > >> > > low,
> > > > > > > > >> > > > >>>> it
> > > > > > > > >> > > > >>>> > seems
> > > > > > > > >> > > > >>>> > > > like
> > > > > > > > >> > > > >>>> > > > > > > > > segments
> > > > > > > > >> > > > >>>> > > > > > > > > >> > will
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > be
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very
> > > > > frequently.
> > > > > > > This
> > > > > > > > >> can
> > > > > > > > >> > > be a
> > > > > > > > >> > > > >>>> > source of
> > > > > > > > >> > > > >>>> > > > > > > problems
> > > > > > > > >> > > > >>>> > > > > > > > > in
> > > > > > > > >> > > > >>>> > > > > > > > > >> the
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating
> > > many
> > > > > > > > different
> > > > > > > > >> > small
> > > > > > > > >> > > > log
> > > > > > > > >> > > > >>>> > segments
> > > > > > > > >> > > > >>>> > > > > > > > consumes
> > > > > > > > >> > > > >>>> > > > > > > > > a
> > > > > > > > >> > > > >>>> > > > > > > > > >> > huge
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > amount of
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster
> > resources.
> > > > > > > > Therefore,
> > > > > > > > >> I
> > > > > > > > >> > > would
> > > > > > > > >> > > > >>>> > suggest
> > > > > > > > >> > > > >>>> > > > > > adding a
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > broker-level
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration
> > > which
> > > > > allows
> > > > > > > > us
> > > > > > > > >> to
> > > > > > > > >> > > set
> > > > > > > > >> > > > a
> > > > > > > > >> > > > >>>> > minimum
> > > > > > > > >> > > > >>>> > > > value
> > > > > > > > >> > > > >>>> > > > > > > for
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > max.compaction.lag.ms.
> > > > > If
> > > > > > > > we
> > > > > > > > >> let
> > > > > > > > >> > > > >>>> users set
> > > > > > > > >> > > > >>>> > it
> > > > > > > > >> > > > >>>> > > > on a
> > > > > > > > >> > > > >>>> > > > > > > > > >> per-topic
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > basis,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could
> > set
> > > a
> > > > > value
> > > > > > > of
> > > > > > > > >> 1 ms
> > > > > > > > >> > > or
> > > > > > > > >> > > > >>>> > something,
> > > > > > > > >> > > > >>>> > > > and
> > > > > > > > >> > > > >>>> > > > > > > > cause
> > > > > > > > >> > > > >>>> > > > > > > > > >> > chaos.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========>
> this
> > > > > applies to
> > > > > > > > >> > > > segment.ms
> > > > > > > > >> > > > >>>> as
> > > > > > > > >> > > > >>>> > well.
> > > > > > > > >> > > > >>>> > > > > > Today
> > > > > > > > >> > > > >>>> > > > > > > > > users
> > > > > > > > >> > > > >>>> > > > > > > > > >> can
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > set "
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to
> a
> > > very
> > > > > low
> > > > > > > > >> value,
> > > > > > > > >> > and
> > > > > > > > >> > > > >>>> cause a
> > > > > > > > >> > > > >>>> > > > frequent
> > > > > > > > >> > > > >>>> > > > > > > > > >> rolling of
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > active
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this
> is
> > > an
> > > > > > > existing
> > > > > > > > >> > > problem
> > > > > > > > >> > > > >>>> with
> > > > > > > > >> > > > >>>> > > > > > segment.ms.
> > > > > > > > >> > > > >>>> > > > > > > > > >> However,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > that
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that
> we
> > > > > shouldn't
> > > > > > > > fix
> > > > > > > > >> > it.
> > > > > > > > >> > > As
> > > > > > > > >> > > > >>>> you
> > > > > > > > >> > > > >>>> > noted,
> > > > > > > > >> > > > >>>> > > > > > there
> > > > > > > > >> > > > >>>> > > > > > > > > will
> > > > > > > > >> > > > >>>> > > > > > > > > >> be
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > more
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these
> > > > > topic-level
> > > > > > > > >> > retention
> > > > > > > > >> > > > >>>> > settings as
> > > > > > > > >> > > > >>>> > > > a
> > > > > > > > >> > > > >>>> > > > > > > result
> > > > > > > > >> > > > >>>> > > > > > > > > of
> > > > > > > > >> > > > >>>> > > > > > > > > >> > GDPR.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > It
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that
> > > > > pre-existing
> > > > > > > > >> > problems
> > > > > > > > >> > > > >>>> will
> > > > > > > > >> > > > >>>> > cause
> > > > > > > > >> > > > >>>> > > > more
> > > > > > > > >> > > > >>>> > > > > > > > > trouble.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems
> > > relatively
> > > > > > > > >> > > straightforward
> > > > > > > > >> > > > >>>> here --
> > > > > > > > >> > > > >>>> > > > add a
> > > > > > > > >> > > > >>>> > > > > > > > > >> broker-level
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > minimum
> segment.ms
> > > that
> > > > > > > > overrides
> > > > > > > > >> > > > >>>> per-topic
> > > > > > > > >> > > > >>>> > > > minimums.
> > > > > > > > >> > > > >>>> > > > > > We
> > > > > > > > >> > > > >>>> > > > > > > > can
> > > > > > > > >> > > > >>>> > > > > > > > > >> also
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > fail
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful
> error
> > > > > message
> > > > > > > > when
> > > > > > > > >> > > someone
> > > > > > > > >> > > > >>>> > attempts
> > > > > > > > >> > > > >>>> > > > to
> > > > > > > > >> > > > >>>> > > > > > set
> > > > > > > > >> > > > >>>> > > > > > > an
> > > > > > > > >> > > > >>>> > > > > > > > > >> > invalid
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option,
> the
> > > > > minimum
> > > > > > > of "
> > > > > > > > >> > > > >>>> > > > max.compaction.lag.ms"
> > > > > > > > >> > > > >>>> > > > > > > > should
> > > > > > > > >> > > > >>>> > > > > > > > > >> be
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the
> > > minimum of
> > > > > "
> > > > > > > > >> > segment.ms
> > > > > > > > >> > > ".
> > > > > > > > >> > > > >>>> Since
> > > > > > > > >> > > > >>>> > > > today the
> > > > > > > > >> > > > >>>> > > > > > > > > >> minimum of
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "
> > > > > > > max.compaction.lag.ms"
> > > > > > > > >> also
> > > > > > > > >> > > > >>>> starts
> > > > > > > > >> > > > >>>> > with
> > > > > > > > >> > > > >>>> > > > 1. "0"
> > > > > > > > >> > > > >>>> > > > > > > > means
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > disable. I
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as
> > > disable,
> > > > > but
> > > > > > > it
> > > > > > > > is
> > > > > > > > >> > hard
> > > > > > > > >> > > > to
> > > > > > > > >> > > > >>>> > define
> > > > > > > > >> > > > >>>> > > > the
> > > > > > > > >> > > > >>>> > > > > > > > meaning
> > > > > > > > >> > > > >>>> > > > > > > > > >> of 0
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > because
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > we
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll
> > the
> > > > > active
> > > > > > > > >> segment
> > > > > > > > >> > > > >>>> > immediately.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair
> point.
> > > We
> > > > > should
> > > > > > > > >> make
> > > > > > > > >> > 0 =
> > > > > > > > >> > > > >>>> > disable, to
> > > > > > > > >> > > > >>>> > > > be
> > > > > > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > best,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Colin
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that
> an
> > > > > > > alternative
> > > > > > > > >> > > > >>>> configuration
> > > > > > > > >> > > > >>>> > is to
> > > > > > > > >> > > > >>>> > > > use
> > > > > > > > >> > > > >>>> > > > > > -1
> > > > > > > > >> > > > >>>> > > > > > > > as
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > "disabled"
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > and
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as
> "immediate
> > > > > > > compaction".
> > > > > > > > >> > > Because
> > > > > > > > >> > > > >>>> > compaction
> > > > > > > > >> > > > >>>> > > > lag
> > > > > > > > >> > > > >>>> > > > > > is
> > > > > > > > >> > > > >>>> > > > > > > > > still
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > determined
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> > > > > > > > min.compaction.lag
> > > > > > > > >> and
> > > > > > > > >> > > how
> > > > > > > > >> > > > >>>> long
> > > > > > > > >> > > > >>>> > to
> > > > > > > > >> > > > >>>> > > > roll
> > > > > > > > >> > > > >>>> > > > > > an
> > > > > > > > >> > > > >>>> > > > > > > > > active
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > segment,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > the
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag
> for
> > > > > > > compaction
> > > > > > > > is
> > > > > > > > >> > > > >>>> undetermined
> > > > > > > > >> > > > >>>> > if
> > > > > > > > >> > > > >>>> > > > we
> > > > > > > > >> > > > >>>> > > > > > use
> > > > > > > > >> > > > >>>> > > > > > > > "0".
> > > > > > > > >> > > > >>>> > > > > > > > > >> On
> > > > > > > > >> > > > >>>> > > > > > > > > >> > the
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > other
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can
> > > > > already set
> > > > > > > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > > > > > > >> > > > >>>> > > > > > > > to
> > > > > > > > >> > > > >>>> > > > > > > > > >> > achieve
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > same
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So
> here
> > we
> > > > > choose
> > > > > > > > "0"
> > > > > > > > >> as
> > > > > > > > >> > > > >>>> "disabled".
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer
> > -1
> > > to
> > > > > be
> > > > > > > the
> > > > > > > > >> > invalid
> > > > > > > > >> > > > >>>> setting.
> > > > > > > > >> > > > >>>> > > > > > Treating
> > > > > > > > >> > > > >>>> > > > > > > 0
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > differently
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > than
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems
> strange
> > > to me.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my
> > > previous
> > > > > > > > >> comment, I
> > > > > > > > >> > > am
> > > > > > > > >> > > > >>>> not
> > > > > > > > >> > > > >>>> > > > strongly
> > > > > > > > >> > > > >>>> > > > > > > > against,
> > > > > > > > >> > > > >>>> > > > > > > > > >> but
> > > > > > > > >> > > > >>>> > > > > > > > > >> > 0
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > is
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > not a
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > valid
> > configuration
> > > in
> > > > > my
> > > > > > > > >> option.
> > > > > > > > >> > So
> > > > > > > > >> > > I
> > > > > > > > >> > > > >>>> use
> > > > > > > > >> > > > >>>> > "0" as
> > > > > > > > >> > > > >>>> > > > > > > disabled
> > > > > > > > >> > > > >>>> > > > > > > > > >> state.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > best,
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4,
> > > 2018,
> > > > > at
> > > > > > > > 15:04,
> > > > > > > > >> > > > xiongqi
> > > > > > > > >> > > > >>>> wu
> > > > > > > > >> > > > >>>> > wrote:
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE
> for
> > > this
> > > > > KIP.
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > >> > > > >>>> >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> >>>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > > > >> > > > >>>> > > >
> > > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > >> > > > >>>> >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> >>>>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > >> > > > >>>> >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> >>>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > > > >> > > > >>>> > > >
> > > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > >> > > > >>>> >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > > >> > > > >>>>
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > <
> > > > > > > > >> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > > > >>>> > <
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > %3A+Time-based+log+compaction+policy
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > Implementation:
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> >>>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > > > >> https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> > >>>>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > > > > > >> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> >>>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > > > >> https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> >>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > > >> > > > >>>> > > > > > > > <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > > >> > > > >>>> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> > >>>>>>
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi
> > > (Wesley) Wu
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > --
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > Brett Rann
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street,
> > Melbourne
> > > VIC
> > > > > 3000
> > > > > > > > >> > > Australia
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826
> 017
> > > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > > --
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > > Brett Rann
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne
> > VIC
> > > > > 3000
> > > > > > > > >> Australia
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > --
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > Brett Rann
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne
> VIC
> > > 3000
> > > > > > > > >> Australia
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > > >> > > > >>>> > > > > > >
> > > > > > > > >> > > > >>>> > > > > >
> > > > > > > > >> > > > >>>> > > >
> > > > > > > > >> > > > >>>> >
> > > > > > > > >> > > > >>>>
> > > > > > > > >> > > > >>>
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by Joel Koshy <jj...@gmail.com>.
+1 on the updated KIP.
On Wed, Dec 5, 2018 at 11:56 AM Dong Lin <li...@gmail.com> wrote:
> Thanks for the update. +1 (binding)
>
> On Wed, Dec 5, 2018 at 8:19 AM Colin McCabe <cm...@apache.org> wrote:
>
> > Thanks, Xiongqi Wu. +1 (binding)
> >
> > regards,
> > Colin
> >
> >
> > On Tue, Dec 4, 2018, at 20:58, xiongqi (wesley) wu wrote:
> > > Colin,
> > >
> > > Thanks for comments.
> > > Out of ordered message timestamp is a very good point.
> > > We can combine max.compaction.lag.ms with
> > > log.message.timestamp.difference.max.ms to achieve what we want in an
> > > environment that message timestamp can be shifted a lot.
> > >
> > > There are similar discussions regarding log.retention.ms and
> > > log.message.timestamp.difference.max.ms in KAFKA-4340.
> > >
> > > I agree that we can always use first message timestamp not the
> > maxTimestamp
> > > of the previous log segment to make it simple.
> > >
> > > I have updated the KIP.
> > >
> > > Xiongqi (wesley) Wu
> > >
> > >
> > > On Tue, Dec 4, 2018 at 5:13 PM Colin McCabe <cm...@apache.org>
> wrote:
> > >
> > > > Hi Xiongqi,
> > > >
> > > > Thinking about this a little bit more, it seems like we don't have
> any
> > > > guarantees just by looking at the timestamp of the first message in a
> > log
> > > > segment. Similarly, we don't have any guarantees just by looking at
> > the
> > > > maxTimestamp of the previous log segment. Old data could appear
> > anywhere--
> > > > you could put data that was years old in the middle of a segment from
> > 2018.
> > > >
> > > > However, if log.message.timestamp.difference.max.ms is set, then we
> > can
> > > > make some actual guarantees that old data gets purged-- which is what
> > the
> > > > GDPR requires, of course.
> > > >
> > > > Overall, maybe we can make KIP-354 simpler by just always looking at
> > the
> > > > timestamp of the first log message. I don't think looking at the
> > > > maxTimestamp of the previous segment is any more accurate. Aside
> from
> > > > that, looks good, since we can get what we need with the combination
> of
> > > > this and log.message.timestamp.difference.max.ms.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote:
> > > > > Thanks for binding and non-binding votes.
> > > > > Can I get one more binding vote?
> > > > >
> > > > > Thanks in advance!
> > > > >
> > > > > Xiongqi (Wesley) Wu
> > > > >
> > > > >
> > > > > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <ma...@frmr.me> wrote:
> > > > >
> > > > > > I'm a +1 (non-binding) — This looks like it would have saved us a
> > lot
> > > > of
> > > > > > pain in an issue we had to debug recently. I can't go into
> > details, but
> > > > > > figuring out how to achieve this effect gave me quite a headache.
> > :)
> > > > > >
> > > > > > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xi...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Can I have one more vote on this KIP?
> > > > > > > Any comment is appreciated.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
> > > > > > >
> > > > > > >
> > > > > > > Xiongqi (Wesley) Wu
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xiongqiwu@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks Dong.
> > > > > > > > I have updated the KIP.
> > > > > > > >
> > > > > > > > Xiongqi (Wesley) Wu
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <lindong28@gmail.com
> >
> > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> > > > > > > >>
> > > > > > > >> One minor comment: it may be a bit better to clarify in the
> > public
> > > > > > > >> interface section that the value of the newly added metric
> is
> > > > > > determined
> > > > > > > >> based by applying that formula across all compactable
> > segments.
> > > > For
> > > > > > > >> example:
> > > > > > > >>
> > > > > > > >> The maximum value of Math.max(now -
> > > > > > > >> earliest_timestamp_in_ms_of_uncompacted_segment -
> > > > > > max.compaction.lag.ms
> > > > > > > ,
> > > > > > > >> 0)/1000 across all compactable partitions, where the
> > > > > > > >> max.compaction.lag.ms
> > > > > > > >> can be overridden on per-topic basis.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <
> > xiongqiwu@gmail.com>
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >> > Thanks Joel.
> > > > > > > >> > Tracking the delay at second granularity makes sense
> > > > > > > >> > I have updated KIP.
> > > > > > > >> >
> > > > > > > >> > Xiongqi (Wesley) Wu
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <
> > jjkoshy.w@gmail.com>
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > > +1 with one suggestion on the proposed metric. You
> should
> > > > probably
> > > > > > > >> > include
> > > > > > > >> > > the unit. So for e.g., max-compaction-delay-secs.
> > > > > > > >> > >
> > > > > > > >> > > Joel
> > > > > > > >> > >
> > > > > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <
> > > > xiongqiwu@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >> > >
> > > > > > > >> > > > bump
> > > > > > > >> > > > Xiongqi (Wesley) Wu
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <
> > > > xiongqiwu@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and
> Xiaohe
> > for
> > > > > > > >> feedback.
> > > > > > > >> > > > > Can I have more feedback or VOTE on this KIP?
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > Xiongqi (Wesley) Wu
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> > > > > > > xiongqiwu@gmail.com>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > >> Any other votes or comments?
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> Xiongqi (Wesley) Wu
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> > > > > > > xiongqiwu@gmail.com
> > > > > > > >> >
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>> Yes, more votes and code review.
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> Xiongqi (Wesley) Wu
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > > > > > > >> > > <brann@zendesk.com.invalid
> > > > > > > >> > > > >
> > > > > > > >> > > > >>> wrote:
> > > > > > > >> > > > >>>
> > > > > > > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> Where do we go from here? More votes?
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> > > > > > > >> cmccabe@apache.org>
> > > > > > > >> > > > >>>> wrote:
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu
> wrote:
> > > > > > > >> > > > >>>> > > Thank you for comments. I will use '0' for
> now.
> > > > > > > >> > > > >>>> > >
> > > > > > > >> > > > >>>> > > If we create topics through admin client in
> the
> > > > future,
> > > > > > > we
> > > > > > > >> > might
> > > > > > > >> > > > >>>> perform
> > > > > > > >> > > > >>>> > > some useful checks. (but the assumption is
> all
> > > > brokers
> > > > > > in
> > > > > > > >> the
> > > > > > > >> > > same
> > > > > > > >> > > > >>>> > cluster
> > > > > > > >> > > > >>>> > > have the same default configurations value,
> > > > > > otherwise,it
> > > > > > > >> might
> > > > > > > >> > > > >>>> still be
> > > > > > > >> > > > >>>> > > tricky to do such cross validation check.)
> > > > > > > >> > > > >>>> >
> > > > > > > >> > > > >>>> > This isn't something that we might do in the
> > future--
> > > > > > this
> > > > > > > is
> > > > > > > >> > > > >>>> something we
> > > > > > > >> > > > >>>> > are doing now. We already have Create Topic
> > policies
> > > > > > which
> > > > > > > >> are
> > > > > > > >> > > > >>>> enforced by
> > > > > > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for
> > details.
> > > > This
> > > > > > is
> > > > > > > >> one
> > > > > > > >> > of
> > > > > > > >> > > > the
> > > > > > > >> > > > >>>> > motivations for getting rid of direct ZK
> access--
> > > > making
> > > > > > > sure
> > > > > > > >> > that
> > > > > > > >> > > > >>>> these
> > > > > > > >> > > > >>>> > policies are applied.
> > > > > > > >> > > > >>>> >
> > > > > > > >> > > > >>>> > I agree that having different configurations on
> > > > different
> > > > > > > >> > brokers
> > > > > > > >> > > > can
> > > > > > > >> > > > >>>> be
> > > > > > > >> > > > >>>> > confusing and frustrating . That's why more
> > > > > > configurations
> > > > > > > >> are
> > > > > > > >> > > being
> > > > > > > >> > > > >>>> made
> > > > > > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations
> are
> > > > stored
> > > > > > > >> > centrally
> > > > > > > >> > > > in
> > > > > > > >> > > > >>>> ZK,
> > > > > > > >> > > > >>>> > so they are the same on all brokers (modulo
> > > > propagation
> > > > > > > >> delays).
> > > > > > > >> > > In
> > > > > > > >> > > > >>>> any
> > > > > > > >> > > > >>>> > case, this is a general issue, not specific to
> > > > "create
> > > > > > > >> topics".
> > > > > > > >> > > > >>>> >
> > > > > > > >> > > > >>>> > cheers,
> > > > > > > >> > > > >>>> > Colin
> > > > > > > >> > > > >>>> >
> > > > > > > >> > > > >>>> >
> > > > > > > >> > > > >>>> > >
> > > > > > > >> > > > >>>> > >
> > > > > > > >> > > > >>>> > > Xiongqi (Wesley) Wu
> > > > > > > >> > > > >>>> > >
> > > > > > > >> > > > >>>> > >
> > > > > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin
> McCabe <
> > > > > > > >> > > cmccabe@apache.org
> > > > > > > >> > > > >
> > > > > > > >> > > > >>>> > wrote:
> > > > > > > >> > > > >>>> > >
> > > > > > > >> > > > >>>> > > > I don't have a strong opinion. But I think
> we
> > > > should
> > > > > > > >> > probably
> > > > > > > >> > > be
> > > > > > > >> > > > >>>> > > > consistent with how segment.ms works, and
> > just
> > > > use
> > > > > > 0.
> > > > > > > >> > > > >>>> > > >
> > > > > > > >> > > > >>>> > > > best,
> > > > > > > >> > > > >>>> > > > Colin
> > > > > > > >> > > > >>>> > > >
> > > > > > > >> > > > >>>> > > >
> > > > > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann
> > wrote:
> > > > > > > >> > > > >>>> > > > > OK thanks for that clarification. I see
> why
> > > > you're
> > > > > > > >> > > > uncomfortable
> > > > > > > >> > > > >>>> > with 0
> > > > > > > >> > > > >>>> > > > now.
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer
> > > > consistency in
> > > > > > > >> > > > >>>> configuration
> > > > > > > >> > > > >>>> > > > options.
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > > Personally I lean towards treating 0 and
> 1
> > > > > > similarly
> > > > > > > in
> > > > > > > >> > that
> > > > > > > >> > > > >>>> > scenario,
> > > > > > > >> > > > >>>> > > > > because it favours the person thinking
> > about
> > > > > > setting
> > > > > > > >> the
> > > > > > > >> > > > >>>> > configurations,
> > > > > > > >> > > > >>>> > > > > and a person doesn't care about a 1ms
> edge
> > case
> > > > > > > >> especially
> > > > > > > >> > > > when
> > > > > > > >> > > > >>>> the
> > > > > > > >> > > > >>>> > > > context
> > > > > > > >> > > > >>>> > > > > is the true minimum is tied to the log
> > cleaner
> > > > > > > cadence.
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because
> > there
> > > > is
> > > > > > > some
> > > > > > > >> > > > >>>> uniquness in
> > > > > > > >> > > > >>>> > > > > segment.ms not being able to be set to
> 0,
> > > > reduces
> > > > > > > >> > > > configuration
> > > > > > > >> > > > >>>> > > > consistency
> > > > > > > >> > > > >>>> > > > > in favour of capturing a MS gap in an
> edge
> > case
> > > > > > that
> > > > > > > >> > nobody
> > > > > > > >> > > > >>>> would
> > > > > > > >> > > > >>>> > ever
> > > > > > > >> > > > >>>> > > > > notice. For someone to understand why
> > > > everywhere
> > > > > > else
> > > > > > > >> -1
> > > > > > > >> > is
> > > > > > > >> > > > >>>> used to
> > > > > > > >> > > > >>>> > > > > disable, but here 0 is used, they would
> > need to
> > > > > > learn
> > > > > > > >> > about
> > > > > > > >> > > > >>>> > segment.ms
> > > > > > > >> > > > >>>> > > > > having a 1ms minimum and then after
> > learning
> > > > would
> > > > > > > >> think
> > > > > > > >> > > "who
> > > > > > > >> > > > >>>> cares
> > > > > > > >> > > > >>>> > about
> > > > > > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to
> majority.
> > > > Curious
> > > > > > > >> which
> > > > > > > >> > > way
> > > > > > > >> > > > >>>> Colin
> > > > > > > >> > > > >>>> > falls
> > > > > > > >> > > > >>>> > > > > now.
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > > Don't want to spend more time on this
> > though,
> > > > It's
> > > > > > > well
> > > > > > > >> > into
> > > > > > > >> > > > >>>> > > > bikeshedding
> > > > > > > >> > > > >>>> > > > > territory now. :)
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi
> wu <
> > > > > > > >> > > > xiongqiwu@gmail.com>
> > > > > > > >> > > > >>>> > wrote:
> > > > > > > >> > > > >>>> > > > >
> > > > > > > >> > > > >>>> > > > > > I want to honor the minimum value of
> > > > segment.ms
> > > > > > > >> (which
> > > > > > > >> > is
> > > > > > > >> > > > >>>> 1ms) to
> > > > > > > >> > > > >>>> > > > force
> > > > > > > >> > > > >>>> > > > > > roll an active segment.
> > > > > > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms"
> any
> > > > value >
> > > > > > > 0,
> > > > > > > >> the
> > > > > > > >> > > > >>>> minimum of
> > > > > > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms
> > will be
> > > > > > used
> > > > > > > to
> > > > > > > >> > seal
> > > > > > > >> > > > an
> > > > > > > >> > > > >>>> > active
> > > > > > > >> > > > >>>> > > > > > segment.
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0,
> > the
> > > > > > current
> > > > > > > >> > > > >>>> implementation
> > > > > > > >> > > > >>>> > will
> > > > > > > >> > > > >>>> > > > > > treat it as disabled.
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > > It is a little bit weird to treat
> > > > > > > >> max.compaction.lag=0
> > > > > > > >> > the
> > > > > > > >> > > > >>>> same as
> > > > > > > >> > > > >>>> > > > > > max.compaction.lag=1.
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > > There might be a reason why we set the
> > > > minimum of
> > > > > > > >> > > > segment.ms
> > > > > > > >> > > > >>>> to 1,
> > > > > > > >> > > > >>>> > > > and I
> > > > > > > >> > > > >>>> > > > > > don't want to break this assumption.
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett
> Rann
> > > > > > > >> > > > >>>> > <br...@zendesk.com.invalid>
> > > > > > > >> > > > >>>> > > > > > wrote:
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > > > > > You're rolling a new segment if the
> > > > condition
> > > > > > is
> > > > > > > >> met
> > > > > > > >> > > > right?
> > > > > > > >> > > > >>>> So
> > > > > > > >> > > > >>>> > I'm
> > > > > > > >> > > > >>>> > > > > > > struggling to understand the
> relevance
> > of
> > > > > > > >> segment.ms
> > > > > > > >> > > > here.
> > > > > > > >> > > > >>>> > Maybe an
> > > > > > > >> > > > >>>> > > > > > > example
> > > > > > > >> > > > >>>> > > > > > > would help my understanding:
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in,
> > after
> > > > 1ms
> > > > > > the
> > > > > > > >> topic
> > > > > > > >> > > > >>>> should be
> > > > > > > >> > > > >>>> > > > eligible
> > > > > > > >> > > > >>>> > > > > > > for compaction when the log
> compaction
> > > > thread
> > > > > > > gets
> > > > > > > >> > > around
> > > > > > > >> > > > to
> > > > > > > >> > > > >>>> > > > evaluating
> > > > > > > >> > > > >>>> > > > > > the
> > > > > > > >> > > > >>>> > > > > > > topic.
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > if we have
> > > > > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in,
> > after
> > > > 0ms
> > > > > > the
> > > > > > > >> topic
> > > > > > > >> > > > >>>> should be
> > > > > > > >> > > > >>>> > > > eligible
> > > > > > > >> > > > >>>> > > > > > > for compaction when the log
> compaction
> > > > thread
> > > > > > > gets
> > > > > > > >> > > around
> > > > > > > >> > > > to
> > > > > > > >> > > > >>>> > > > evaluating
> > > > > > > >> > > > >>>> > > > > > the
> > > > > > > >> > > > >>>> > > > > > > topic.
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > In both of those cases the change
> would
> > > > mean a
> > > > > > > new
> > > > > > > >> > > segment
> > > > > > > >> > > > >>>> is
> > > > > > > >> > > > >>>> > rolled
> > > > > > > >> > > > >>>> > > > so
> > > > > > > >> > > > >>>> > > > > > the
> > > > > > > >> > > > >>>> > > > > > > new message would be part of the
> > compaction
> > > > > > > task. 0
> > > > > > > >> > and
> > > > > > > >> > > 1
> > > > > > > >> > > > >>>> are
> > > > > > > >> > > > >>>> > > > practically
> > > > > > > >> > > > >>>> > > > > > > the same meaning since neither is
> > > > providing an
> > > > > > > >> actual
> > > > > > > >> > > > >>>> guarantee
> > > > > > > >> > > > >>>> > at
> > > > > > > >> > > > >>>> > > > such
> > > > > > > >> > > > >>>> > > > > > low
> > > > > > > >> > > > >>>> > > > > > > MS settings, but effectively tying it
> > to
> > > > both
> > > > > > the
> > > > > > > >> > > > frequency
> > > > > > > >> > > > >>>> of
> > > > > > > >> > > > >>>> > the
> > > > > > > >> > > > >>>> > > > log
> > > > > > > >> > > > >>>> > > > > > > cleaner running and the priority of
> the
> > > > given
> > > > > > > topic
> > > > > > > >> > > being
> > > > > > > >> > > > >>>> the
> > > > > > > >> > > > >>>> > highest
> > > > > > > >> > > > >>>> > > > > > > priority of all topics that are
> > evaluated
> > > > for
> > > > > > > >> cleaning
> > > > > > > >> > > on
> > > > > > > >> > > > >>>> the
> > > > > > > >> > > > >>>> > next
> > > > > > > >> > > > >>>> > > > cycle.
> > > > > > > >> > > > >>>> > > > > > > You've captured that nuance with
> > careful
> > > > > > > "skipped"
> > > > > > > >> > > wording
> > > > > > > >> > > > >>>> in
> > > > > > > >> > > > >>>> > the KIP
> > > > > > > >> > > > >>>> > > > > > > here "controls
> > > > > > > >> > > > >>>> > > > > > > the max time interval a
> message/segment
> > > > can be
> > > > > > > >> skipped
> > > > > > > >> > > for
> > > > > > > >> > > > >>>> log
> > > > > > > >> > > > >>>> > > > > > compaction".
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > How is 0 different to 1, practically?
> > And
> > > > how
> > > > > > is
> > > > > > > it
> > > > > > > >> > > > >>>> relating to
> > > > > > > >> > > > >>>> > > > > > segment.ms
> > > > > > > >> > > > >>>> > > > > > > ?
> > > > > > > >> > > > >>>> > > > > > > Is it that you're proposing to have 0
> > mean
> > > > "use
> > > > > > > >> > > > segment.ms
> > > > > > > >> > > > >>>> > > > instead?" as
> > > > > > > >> > > > >>>> > > > > > a
> > > > > > > >> > > > >>>> > > > > > > kind of third option?
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM
> > xiongqi wu
> > > > <
> > > > > > > >> > > > >>>> xiongqiwu@gmail.com>
> > > > > > > >> > > > >>>> > > > wrote:
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > To make it clear,
> > > > > > > >> > > > >>>> > > > > > > > I don't against using -1 as
> disabled,
> > > > but we
> > > > > > > >> need to
> > > > > > > >> > > > come
> > > > > > > >> > > > >>>> up
> > > > > > > >> > > > >>>> > with
> > > > > > > >> > > > >>>> > > > the
> > > > > > > >> > > > >>>> > > > > > > > meaning of "0".
> > > > > > > >> > > > >>>> > > > > > > > If "0" means immediate compaction,
> > but
> > > > the
> > > > > > > actual
> > > > > > > >> > > > >>>> compaction
> > > > > > > >> > > > >>>> > lag
> > > > > > > >> > > > >>>> > > > will
> > > > > > > >> > > > >>>> > > > > > be
> > > > > > > >> > > > >>>> > > > > > > > segment.ms.
> > > > > > > >> > > > >>>> > > > > > > > It has longer lag than setting the
> > value
> > > > to
> > > > > > be
> > > > > > > >> half
> > > > > > > >> > of
> > > > > > > >> > > > >>>> > segment.ms.
> > > > > > > >> > > > >>>> > > > > > > > We cannot provide "0" as max
> > compaction
> > > > lag.
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > Here are two options.
> > > > > > > >> > > > >>>> > > > > > > > Option 1:
> > > > > > > >> > > > >>>> > > > > > > > Keep 0 as disabled
> > > > > > > >> > > > >>>> > > > > > > > Option 2:
> > > > > > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction
> lag
> > =
> > > > > > > >> segment.ms),
> > > > > > > >> > > and
> > > > > > > >> > > > >>>> > others.
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM
> Brett
> > Rann
> > > > > > > >> > > > >>>> > > > <br...@zendesk.com.invalid>
> > > > > > > >> > > > >>>> > > > > > > > wrote:
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special"
> with
> > > > these
> > > > > > > >> settings
> > > > > > > >> > > for
> > > > > > > >> > > > >>>> > example:
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > log.retention.bytes
> > > > > > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > > > > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > > > > > >> > > > >>>> > > > > > > > > queued.max.request.bytes
> > > > > > > >> > > > >>>> > > > > > > > > retention.bytes
> > > > > > > >> > > > >>>> > > > > > > > > retention.ms
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > and acks.
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use
> OS
> > > > > > defaults,
> > > > > > > >> max
> > > > > > > >> > > > (acks),
> > > > > > > >> > > > >>>> > etc. I
> > > > > > > >> > > > >>>> > > > don't
> > > > > > > >> > > > >>>> > > > > > > see
> > > > > > > >> > > > >>>> > > > > > > > > much convention of 0 meaning
> those
> > > > things.
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > There are some NULLs but it seems
> > > > convetion
> > > > > > > >> there
> > > > > > > >> > is
> > > > > > > >> > > > >>>> NULL is
> > > > > > > >> > > > >>>> > used
> > > > > > > >> > > > >>>> > > > > > where
> > > > > > > >> > > > >>>> > > > > > > > > there's another setting in the
> > > > hierarchy
> > > > > > that
> > > > > > > >> > would
> > > > > > > >> > > be
> > > > > > > >> > > > >>>> used
> > > > > > > >> > > > >>>> > > > instead.
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM
> > Brett
> > > > Rann
> > > > > > <
> > > > > > > >> > > > >>>> > brann@zendesk.com>
> > > > > > > >> > > > >>>> > > > > > wrote:
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to
> 0,
> > > > then
> > > > > > > we're
> > > > > > > >> not
> > > > > > > >> > > > being
> > > > > > > >> > > > >>>> > > > consistent
> > > > > > > >> > > > >>>> > > > > > > > > > by using 0 for this new
> setting?
> > I
> > > > throw
> > > > > > > out
> > > > > > > >> -1
> > > > > > > >> > > for
> > > > > > > >> > > > >>>> > > > consideration
> > > > > > > >> > > > >>>> > > > > > > > > > again :)
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM
> > > > xiongqi
> > > > > > wu
> > > > > > > <
> > > > > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > > > > >> > > > >>>> > > > > > > > wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document after
> > PR is
> > > > > > > merged.
> > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum
> > of "
> > > > > > > >> segment.ms"
> > > > > > > >> > > to
> > > > > > > >> > > > >>>> 1, we
> > > > > > > >> > > > >>>> > > > cannot
> > > > > > > >> > > > >>>> > > > > > set
> > > > > > > >> > > > >>>> > > > > > > "
> > > > > > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > > >> I also updated the title of
> this
> > > > KIP.
> > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM
> > Brett
> > > > > > Rann
> > > > > > > >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1
> > > > since i'm
> > > > > > > in
> > > > > > > >> the
> > > > > > > >> > > > >>>> minority.
> > > > > > > >> > > > >>>> > :)
> > > > > > > >> > > > >>>> > > > Can
> > > > > > > >> > > > >>>> > > > > > we
> > > > > > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented
> as
> > > > meaning
> > > > > > > >> > disabled
> > > > > > > >> > > > >>>> here:
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>>
> > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >> > > > >>>> > <
> > > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >> > > > >>>> > <
> > > > https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >> > > > >>>> > <
> > > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >> > > > >>>> > <
> > > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > >> > > > >>>>
> > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >> > > > >>>> > <
> > > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >> > > > >>>> > <
> > > > https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >> > > > >>>> > <
> > > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > > >> > > > >>>> > <
> > > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >>>>
> > > > > > > ?
> > > > > > > >> > > > >>>> > > > > > > > > >> > And while there it would be
> > good
> > > > if
> > > > > > > >> > segment.ms
> > > > > > > >> > > > is
> > > > > > > >> > > > >>>> > > > documented
> > > > > > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too.
> > (there's
> > > > some
> > > > > > > >> > hierarchy
> > > > > > > >> > > > of
> > > > > > > >> > > > >>>> > configs
> > > > > > > >> > > > >>>> > > > for
> > > > > > > >> > > > >>>> > > > > > > that
> > > > > > > >> > > > >>>> > > > > > > > > too
> > > > > > > >> > > > >>>> > > > > > > > > >> > if its not set and null for
> > others
> > > > > > means
> > > > > > > >> > > > disabled!)
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44
> AM
> > > > xiongqi
> > > > > > > wu
> > > > > > > >> <
> > > > > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > > > > >> > > > >>>> > > > > > > > > wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate
> > > > immediate
> > > > > > > >> > compaction,
> > > > > > > >> > > > the
> > > > > > > >> > > > >>>> > > > compaction
> > > > > > > >> > > > >>>> > > > > > lag
> > > > > > > >> > > > >>>> > > > > > > > is
> > > > > > > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms
> in
> > > > worst
> > > > > > > >> case. If
> > > > > > > >> > > > >>>> segment.ms
> > > > > > > >> > > > >>>> > is
> > > > > > > >> > > > >>>> > > > 24
> > > > > > > >> > > > >>>> > > > > > > > hours,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is
> a
> > > > weaker
> > > > > > > >> > guarantee
> > > > > > > >> > > > than
> > > > > > > >> > > > >>>> > setting
> > > > > > > >> > > > >>>> > > > any
> > > > > > > >> > > > >>>> > > > > > > > value
> > > > > > > >> > > > >>>> > > > > > > > > >> less
> > > > > > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the
> > > > definition of
> > > > > > > "max
> > > > > > > >> > > > >>>> compaction
> > > > > > > >> > > > >>>> > lag",
> > > > > > > >> > > > >>>> > > > we
> > > > > > > >> > > > >>>> > > > > > > > cannot
> > > > > > > >> > > > >>>> > > > > > > > > >> have
> > > > > > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to
> > indicate
> > > > > > > >> "disable".
> > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at
> 8:34
> > AM
> > > > Colin
> > > > > > > >> > McCabe <
> > > > > > > >> > > > >>>> > > > > > cmccabe@apache.org
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at
> > 22:11,
> > > > > > Brett
> > > > > > > >> Rann
> > > > > > > >> > > > >>>> wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point.
> > We
> > > > should
> > > > > > > >> make
> > > > > > > >> > 0 =
> > > > > > > >> > > > >>>> > disable, to
> > > > > > > >> > > > >>>> > > > be
> > > > > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere
> for
> > > > disable
> > > > > > > and
> > > > > > > >> > when
> > > > > > > >> > > > >>>> seeing
> > > > > > > >> > > > >>>> > it
> > > > > > > >> > > > >>>> > > > in a
> > > > > > > >> > > > >>>> > > > > > > > config
> > > > > > > >> > > > >>>> > > > > > > > > >> it's
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > clear
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special
> > > > meaning. 0
> > > > > > > >> doesn't
> > > > > > > >> > > have
> > > > > > > >> > > > >>>> to
> > > > > > > >> > > > >>>> > mean
> > > > > > > >> > > > >>>> > > > > > instant,
> > > > > > > >> > > > >>>> > > > > > > > it
> > > > > > > >> > > > >>>> > > > > > > > > >> just
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > means
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as
> possible.
> > I
> > > > don't
> > > > > > > >> think 0
> > > > > > > >> > > is
> > > > > > > >> > > > >>>> > intuitive
> > > > > > > >> > > > >>>> > > > for
> > > > > > > >> > > > >>>> > > > > > > > > disabled
> > > > > > > >> > > > >>>> > > > > > > > > >> and
> > > > > > > >> > > > >>>> > > > > > > > > >> > > it
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I
> > wasn't
> > > > > > aware
> > > > > > > >> > > > segment.ms=0
> > > > > > > >> > > > >>>> ==
> > > > > > > >> > > > >>>> > > > > > disabled,
> > > > > > > >> > > > >>>> > > > > > > > > but I
> > > > > > > >> > > > >>>> > > > > > > > > >> > > think
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > that is also
> > unintuitive.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an
> > argument
> > > > for
> > > > > > > >> keeping
> > > > > > > >> > > > these
> > > > > > > >> > > > >>>> two
> > > > > > > >> > > > >>>> > > > > > > > configurations
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they
> > are so
> > > > > > > >> similar. I
> > > > > > > >> > > > agree
> > > > > > > >> > > > >>>> that
> > > > > > > >> > > > >>>> > 0
> > > > > > > >> > > > >>>> > > > was an
> > > > > > > >> > > > >>>> > > > > > > > > >> > unfortunate
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > choice.,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > best,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > Colin
> > > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at
> > > > 11:38 AM
> > > > > > > >> Colin
> > > > > > > >> > > > McCabe
> > > > > > > >> > > > >>>> <
> > > > > > > >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018,
> > at
> > > > 17:47,
> > > > > > > >> > xiongqi
> > > > > > > >> > > wu
> > > > > > > >> > > > >>>> wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for
> > comments.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline
> reply
> > > > below.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley)
> Wu
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4,
> 2018
> > at
> > > > 5:24
> > > > > > PM
> > > > > > > >> > Colin
> > > > > > > >> > > > >>>> McCabe <
> > > > > > > >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this
> > KIP.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a
> > bit
> > > > > > > >> ambiguous.
> > > > > > > >> > Our
> > > > > > > >> > > > >>>> > compaction
> > > > > > > >> > > > >>>> > > > > > > policies
> > > > > > > >> > > > >>>> > > > > > > > > are
> > > > > > > >> > > > >>>> > > > > > > > > >> > > already
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based,
> after
> > > > all. It
> > > > > > > >> seems
> > > > > > > >> > > like
> > > > > > > >> > > > >>>> this
> > > > > > > >> > > > >>>> > > > change is
> > > > > > > >> > > > >>>> > > > > > > > > focused
> > > > > > > >> > > > >>>> > > > > > > > > >> > > around
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > adding
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “
> > > > max.compaction.lag.ms
> > > > > > ."
> > > > > > > >> > Perhaps
> > > > > > > >> > > > >>>> the KIP
> > > > > > > >> > > > >>>> > > > title
> > > > > > > >> > > > >>>> > > > > > > should
> > > > > > > >> > > > >>>> > > > > > > > > be
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > something
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add
> maximum
> > > > > > compaction
> > > > > > > >> lag
> > > > > > > >> > > > time"?
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========>
> sure.
> > I
> > > > will
> > > > > > > >> change
> > > > > > > >> > the
> > > > > > > >> > > > >>>> title.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active
> > segment is
> > > > > > forced
> > > > > > > >> to
> > > > > > > >> > > roll
> > > > > > > >> > > > >>>> when
> > > > > > > >> > > > >>>> > > > either "
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> max.compaction.lag.ms
> > "
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "
> segment.ms"
> > (
> > > > > > > >> log.roll.ms
> > > > > > > >> > and
> > > > > > > >> > > > >>>> > > > log.roll.hours)
> > > > > > > >> > > > >>>> > > > > > > has
> > > > > > > >> > > > >>>> > > > > > > > > >> > reached.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> > > > > > > max.compaction.lag.ms
> > > > > > > >> is
> > > > > > > >> > > low,
> > > > > > > >> > > > >>>> it
> > > > > > > >> > > > >>>> > seems
> > > > > > > >> > > > >>>> > > > like
> > > > > > > >> > > > >>>> > > > > > > > > segments
> > > > > > > >> > > > >>>> > > > > > > > > >> > will
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > be
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very
> > > > frequently.
> > > > > > This
> > > > > > > >> can
> > > > > > > >> > > be a
> > > > > > > >> > > > >>>> > source of
> > > > > > > >> > > > >>>> > > > > > > problems
> > > > > > > >> > > > >>>> > > > > > > > > in
> > > > > > > >> > > > >>>> > > > > > > > > >> the
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating
> > many
> > > > > > > different
> > > > > > > >> > small
> > > > > > > >> > > > log
> > > > > > > >> > > > >>>> > segments
> > > > > > > >> > > > >>>> > > > > > > > consumes
> > > > > > > >> > > > >>>> > > > > > > > > a
> > > > > > > >> > > > >>>> > > > > > > > > >> > huge
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > amount of
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster
> resources.
> > > > > > > Therefore,
> > > > > > > >> I
> > > > > > > >> > > would
> > > > > > > >> > > > >>>> > suggest
> > > > > > > >> > > > >>>> > > > > > adding a
> > > > > > > >> > > > >>>> > > > > > > > > >> > > broker-level
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration
> > which
> > > > allows
> > > > > > > us
> > > > > > > >> to
> > > > > > > >> > > set
> > > > > > > >> > > > a
> > > > > > > >> > > > >>>> > minimum
> > > > > > > >> > > > >>>> > > > value
> > > > > > > >> > > > >>>> > > > > > > for
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > max.compaction.lag.ms.
> > > > If
> > > > > > > we
> > > > > > > >> let
> > > > > > > >> > > > >>>> users set
> > > > > > > >> > > > >>>> > it
> > > > > > > >> > > > >>>> > > > on a
> > > > > > > >> > > > >>>> > > > > > > > > >> per-topic
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > basis,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could
> set
> > a
> > > > value
> > > > > > of
> > > > > > > >> 1 ms
> > > > > > > >> > > or
> > > > > > > >> > > > >>>> > something,
> > > > > > > >> > > > >>>> > > > and
> > > > > > > >> > > > >>>> > > > > > > > cause
> > > > > > > >> > > > >>>> > > > > > > > > >> > chaos.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this
> > > > applies to
> > > > > > > >> > > > segment.ms
> > > > > > > >> > > > >>>> as
> > > > > > > >> > > > >>>> > well.
> > > > > > > >> > > > >>>> > > > > > Today
> > > > > > > >> > > > >>>> > > > > > > > > users
> > > > > > > >> > > > >>>> > > > > > > > > >> can
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > set "
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a
> > very
> > > > low
> > > > > > > >> value,
> > > > > > > >> > and
> > > > > > > >> > > > >>>> cause a
> > > > > > > >> > > > >>>> > > > frequent
> > > > > > > >> > > > >>>> > > > > > > > > >> rolling of
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > active
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is
> > an
> > > > > > existing
> > > > > > > >> > > problem
> > > > > > > >> > > > >>>> with
> > > > > > > >> > > > >>>> > > > > > segment.ms.
> > > > > > > >> > > > >>>> > > > > > > > > >> However,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > that
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we
> > > > shouldn't
> > > > > > > fix
> > > > > > > >> > it.
> > > > > > > >> > > As
> > > > > > > >> > > > >>>> you
> > > > > > > >> > > > >>>> > noted,
> > > > > > > >> > > > >>>> > > > > > there
> > > > > > > >> > > > >>>> > > > > > > > > will
> > > > > > > >> > > > >>>> > > > > > > > > >> be
> > > > > > > >> > > > >>>> > > > > > > > > >> > > more
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these
> > > > topic-level
> > > > > > > >> > retention
> > > > > > > >> > > > >>>> > settings as
> > > > > > > >> > > > >>>> > > > a
> > > > > > > >> > > > >>>> > > > > > > result
> > > > > > > >> > > > >>>> > > > > > > > > of
> > > > > > > >> > > > >>>> > > > > > > > > >> > GDPR.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > It
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that
> > > > pre-existing
> > > > > > > >> > problems
> > > > > > > >> > > > >>>> will
> > > > > > > >> > > > >>>> > cause
> > > > > > > >> > > > >>>> > > > more
> > > > > > > >> > > > >>>> > > > > > > > > trouble.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems
> > relatively
> > > > > > > >> > > straightforward
> > > > > > > >> > > > >>>> here --
> > > > > > > >> > > > >>>> > > > add a
> > > > > > > >> > > > >>>> > > > > > > > > >> broker-level
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms
> > that
> > > > > > > overrides
> > > > > > > >> > > > >>>> per-topic
> > > > > > > >> > > > >>>> > > > minimums.
> > > > > > > >> > > > >>>> > > > > > We
> > > > > > > >> > > > >>>> > > > > > > > can
> > > > > > > >> > > > >>>> > > > > > > > > >> also
> > > > > > > >> > > > >>>> > > > > > > > > >> > > fail
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error
> > > > message
> > > > > > > when
> > > > > > > >> > > someone
> > > > > > > >> > > > >>>> > attempts
> > > > > > > >> > > > >>>> > > > to
> > > > > > > >> > > > >>>> > > > > > set
> > > > > > > >> > > > >>>> > > > > > > an
> > > > > > > >> > > > >>>> > > > > > > > > >> > invalid
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the
> > > > minimum
> > > > > > of "
> > > > > > > >> > > > >>>> > > > max.compaction.lag.ms"
> > > > > > > >> > > > >>>> > > > > > > > should
> > > > > > > >> > > > >>>> > > > > > > > > >> be
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the
> > minimum of
> > > > "
> > > > > > > >> > segment.ms
> > > > > > > >> > > ".
> > > > > > > >> > > > >>>> Since
> > > > > > > >> > > > >>>> > > > today the
> > > > > > > >> > > > >>>> > > > > > > > > >> minimum of
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "
> > > > > > max.compaction.lag.ms"
> > > > > > > >> also
> > > > > > > >> > > > >>>> starts
> > > > > > > >> > > > >>>> > with
> > > > > > > >> > > > >>>> > > > 1. "0"
> > > > > > > >> > > > >>>> > > > > > > > means
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > disable. I
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as
> > disable,
> > > > but
> > > > > > it
> > > > > > > is
> > > > > > > >> > hard
> > > > > > > >> > > > to
> > > > > > > >> > > > >>>> > define
> > > > > > > >> > > > >>>> > > > the
> > > > > > > >> > > > >>>> > > > > > > > meaning
> > > > > > > >> > > > >>>> > > > > > > > > >> of 0
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > because
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > we
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll
> the
> > > > active
> > > > > > > >> segment
> > > > > > > >> > > > >>>> > immediately.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point.
> > We
> > > > should
> > > > > > > >> make
> > > > > > > >> > 0 =
> > > > > > > >> > > > >>>> > disable, to
> > > > > > > >> > > > >>>> > > > be
> > > > > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > best,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Colin
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an
> > > > > > alternative
> > > > > > > >> > > > >>>> configuration
> > > > > > > >> > > > >>>> > is to
> > > > > > > >> > > > >>>> > > > use
> > > > > > > >> > > > >>>> > > > > > -1
> > > > > > > >> > > > >>>> > > > > > > > as
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > "disabled"
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > and
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate
> > > > > > compaction".
> > > > > > > >> > > Because
> > > > > > > >> > > > >>>> > compaction
> > > > > > > >> > > > >>>> > > > lag
> > > > > > > >> > > > >>>> > > > > > is
> > > > > > > >> > > > >>>> > > > > > > > > still
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > determined
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> > > > > > > min.compaction.lag
> > > > > > > >> and
> > > > > > > >> > > how
> > > > > > > >> > > > >>>> long
> > > > > > > >> > > > >>>> > to
> > > > > > > >> > > > >>>> > > > roll
> > > > > > > >> > > > >>>> > > > > > an
> > > > > > > >> > > > >>>> > > > > > > > > active
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > segment,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > the
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for
> > > > > > compaction
> > > > > > > is
> > > > > > > >> > > > >>>> undetermined
> > > > > > > >> > > > >>>> > if
> > > > > > > >> > > > >>>> > > > we
> > > > > > > >> > > > >>>> > > > > > use
> > > > > > > >> > > > >>>> > > > > > > > "0".
> > > > > > > >> > > > >>>> > > > > > > > > >> On
> > > > > > > >> > > > >>>> > > > > > > > > >> > the
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > other
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can
> > > > already set
> > > > > > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > > > > > >> > > > >>>> > > > > > > > to
> > > > > > > >> > > > >>>> > > > > > > > > >> > achieve
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > same
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here
> we
> > > > choose
> > > > > > > "0"
> > > > > > > >> as
> > > > > > > >> > > > >>>> "disabled".
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer
> -1
> > to
> > > > be
> > > > > > the
> > > > > > > >> > invalid
> > > > > > > >> > > > >>>> setting.
> > > > > > > >> > > > >>>> > > > > > Treating
> > > > > > > >> > > > >>>> > > > > > > 0
> > > > > > > >> > > > >>>> > > > > > > > > >> > > differently
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > than
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange
> > to me.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my
> > previous
> > > > > > > >> comment, I
> > > > > > > >> > > am
> > > > > > > >> > > > >>>> not
> > > > > > > >> > > > >>>> > > > strongly
> > > > > > > >> > > > >>>> > > > > > > > against,
> > > > > > > >> > > > >>>> > > > > > > > > >> but
> > > > > > > >> > > > >>>> > > > > > > > > >> > 0
> > > > > > > >> > > > >>>> > > > > > > > > >> > > is
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > not a
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > valid
> configuration
> > in
> > > > my
> > > > > > > >> option.
> > > > > > > >> > So
> > > > > > > >> > > I
> > > > > > > >> > > > >>>> use
> > > > > > > >> > > > >>>> > "0" as
> > > > > > > >> > > > >>>> > > > > > > disabled
> > > > > > > >> > > > >>>> > > > > > > > > >> state.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > best,
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4,
> > 2018,
> > > > at
> > > > > > > 15:04,
> > > > > > > >> > > > xiongqi
> > > > > > > >> > > > >>>> wu
> > > > > > > >> > > > >>>> > wrote:
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for
> > this
> > > > KIP.
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> >>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > >> > > > >>>> >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> >>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > > >> > > > >>>> > > >
> > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> >>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > >> > > > >>>> >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> >>>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> >>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > >> > > > >>>> >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> >>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > > >> > > > >>>> > > >
> > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> >>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > >> > > > >>>> >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > > > > >> > > > >>>>
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > <
> > > > > > > >> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > > > >>>> > <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > > >> > >>>>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > >> > > > %3A+Time-based+log+compaction+policy
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> Implementation:
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > > >> https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> >>>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > > > > >> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > > >> https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > > >> > > > >>>> > > > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > > >> > > > >>>> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611
> >>>>>>
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi
> > (Wesley) Wu
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> > --
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> > Brett Rann
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street,
> Melbourne
> > VIC
> > > > 3000
> > > > > > > >> > > Australia
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > > --
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > > Brett Rann
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne
> VIC
> > > > 3000
> > > > > > > >> Australia
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > --
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > Brett Rann
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC
> > 3000
> > > > > > > >> Australia
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > > > > >> > > > >>>> > > > > > > > >
> > > > > > > >> > > > >>>> > > > > > > >
> > > > > > > >> > > > >>>> > > > > > >
> > > > > > > >> > > > >>>> > > > > >
> > > > > > > >> > > > >>>> > > >
> > > > > > > >> > > > >>>> >
> > > > > > > >> > > > >>>>
> > > > > > > >> > > > >>>
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by Dong Lin <li...@gmail.com>.
Thanks for the update. +1 (binding)
On Wed, Dec 5, 2018 at 8:19 AM Colin McCabe <cm...@apache.org> wrote:
> Thanks, Xiongqi Wu. +1 (binding)
>
> regards,
> Colin
>
>
> On Tue, Dec 4, 2018, at 20:58, xiongqi (wesley) wu wrote:
> > Colin,
> >
> > Thanks for comments.
> > Out of ordered message timestamp is a very good point.
> > We can combine max.compaction.lag.ms with
> > log.message.timestamp.difference.max.ms to achieve what we want in an
> > environment that message timestamp can be shifted a lot.
> >
> > There are similar discussions regarding log.retention.ms and
> > log.message.timestamp.difference.max.ms in KAFKA-4340.
> >
> > I agree that we can always use first message timestamp not the
> maxTimestamp
> > of the previous log segment to make it simple.
> >
> > I have updated the KIP.
> >
> > Xiongqi (wesley) Wu
> >
> >
> > On Tue, Dec 4, 2018 at 5:13 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi Xiongqi,
> > >
> > > Thinking about this a little bit more, it seems like we don't have any
> > > guarantees just by looking at the timestamp of the first message in a
> log
> > > segment. Similarly, we don't have any guarantees just by looking at
> the
> > > maxTimestamp of the previous log segment. Old data could appear
> anywhere--
> > > you could put data that was years old in the middle of a segment from
> 2018.
> > >
> > > However, if log.message.timestamp.difference.max.ms is set, then we
> can
> > > make some actual guarantees that old data gets purged-- which is what
> the
> > > GDPR requires, of course.
> > >
> > > Overall, maybe we can make KIP-354 simpler by just always looking at
> the
> > > timestamp of the first log message. I don't think looking at the
> > > maxTimestamp of the previous segment is any more accurate. Aside from
> > > that, looks good, since we can get what we need with the combination of
> > > this and log.message.timestamp.difference.max.ms.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote:
> > > > Thanks for binding and non-binding votes.
> > > > Can I get one more binding vote?
> > > >
> > > > Thanks in advance!
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > > >
> > > > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <ma...@frmr.me> wrote:
> > > >
> > > > > I'm a +1 (non-binding) — This looks like it would have saved us a
> lot
> > > of
> > > > > pain in an issue we had to debug recently. I can't go into
> details, but
> > > > > figuring out how to achieve this effect gave me quite a headache.
> :)
> > > > >
> > > > > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Can I have one more vote on this KIP?
> > > > > > Any comment is appreciated.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
> > > > > >
> > > > > >
> > > > > > Xiongqi (Wesley) Wu
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xi...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Thanks Dong.
> > > > > > > I have updated the KIP.
> > > > > > >
> > > > > > > Xiongqi (Wesley) Wu
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> > > > > > >>
> > > > > > >> One minor comment: it may be a bit better to clarify in the
> public
> > > > > > >> interface section that the value of the newly added metric is
> > > > > determined
> > > > > > >> based by applying that formula across all compactable
> segments.
> > > For
> > > > > > >> example:
> > > > > > >>
> > > > > > >> The maximum value of Math.max(now -
> > > > > > >> earliest_timestamp_in_ms_of_uncompacted_segment -
> > > > > max.compaction.lag.ms
> > > > > > ,
> > > > > > >> 0)/1000 across all compactable partitions, where the
> > > > > > >> max.compaction.lag.ms
> > > > > > >> can be overridden on per-topic basis.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <
> xiongqiwu@gmail.com>
> > > > > wrote:
> > > > > > >>
> > > > > > >> > Thanks Joel.
> > > > > > >> > Tracking the delay at second granularity makes sense
> > > > > > >> > I have updated KIP.
> > > > > > >> >
> > > > > > >> > Xiongqi (Wesley) Wu
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <
> jjkoshy.w@gmail.com>
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > > +1 with one suggestion on the proposed metric. You should
> > > probably
> > > > > > >> > include
> > > > > > >> > > the unit. So for e.g., max-compaction-delay-secs.
> > > > > > >> > >
> > > > > > >> > > Joel
> > > > > > >> > >
> > > > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <
> > > xiongqiwu@gmail.com>
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > > bump
> > > > > > >> > > > Xiongqi (Wesley) Wu
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <
> > > xiongqiwu@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > > >
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe
> for
> > > > > > >> feedback.
> > > > > > >> > > > > Can I have more feedback or VOTE on this KIP?
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > Xiongqi (Wesley) Wu
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> > > > > > xiongqiwu@gmail.com>
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > >> Any other votes or comments?
> > > > > > >> > > > >>
> > > > > > >> > > > >> Xiongqi (Wesley) Wu
> > > > > > >> > > > >>
> > > > > > >> > > > >>
> > > > > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> > > > > > xiongqiwu@gmail.com
> > > > > > >> >
> > > > > > >> > > > wrote:
> > > > > > >> > > > >>
> > > > > > >> > > > >>> Yes, more votes and code review.
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> Xiongqi (Wesley) Wu
> > > > > > >> > > > >>>
> > > > > > >> > > > >>>
> > > > > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > > > > > >> > > <brann@zendesk.com.invalid
> > > > > > >> > > > >
> > > > > > >> > > > >>> wrote:
> > > > > > >> > > > >>>
> > > > > > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> Where do we go from here? More votes?
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> > > > > > >> cmccabe@apache.org>
> > > > > > >> > > > >>>> wrote:
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > > > > > >> > > > >>>> > > Thank you for comments. I will use '0' for now.
> > > > > > >> > > > >>>> > >
> > > > > > >> > > > >>>> > > If we create topics through admin client in the
> > > future,
> > > > > > we
> > > > > > >> > might
> > > > > > >> > > > >>>> perform
> > > > > > >> > > > >>>> > > some useful checks. (but the assumption is all
> > > brokers
> > > > > in
> > > > > > >> the
> > > > > > >> > > same
> > > > > > >> > > > >>>> > cluster
> > > > > > >> > > > >>>> > > have the same default configurations value,
> > > > > otherwise,it
> > > > > > >> might
> > > > > > >> > > > >>>> still be
> > > > > > >> > > > >>>> > > tricky to do such cross validation check.)
> > > > > > >> > > > >>>> >
> > > > > > >> > > > >>>> > This isn't something that we might do in the
> future--
> > > > > this
> > > > > > is
> > > > > > >> > > > >>>> something we
> > > > > > >> > > > >>>> > are doing now. We already have Create Topic
> policies
> > > > > which
> > > > > > >> are
> > > > > > >> > > > >>>> enforced by
> > > > > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for
> details.
> > > This
> > > > > is
> > > > > > >> one
> > > > > > >> > of
> > > > > > >> > > > the
> > > > > > >> > > > >>>> > motivations for getting rid of direct ZK access--
> > > making
> > > > > > sure
> > > > > > >> > that
> > > > > > >> > > > >>>> these
> > > > > > >> > > > >>>> > policies are applied.
> > > > > > >> > > > >>>> >
> > > > > > >> > > > >>>> > I agree that having different configurations on
> > > different
> > > > > > >> > brokers
> > > > > > >> > > > can
> > > > > > >> > > > >>>> be
> > > > > > >> > > > >>>> > confusing and frustrating . That's why more
> > > > > configurations
> > > > > > >> are
> > > > > > >> > > being
> > > > > > >> > > > >>>> made
> > > > > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are
> > > stored
> > > > > > >> > centrally
> > > > > > >> > > > in
> > > > > > >> > > > >>>> ZK,
> > > > > > >> > > > >>>> > so they are the same on all brokers (modulo
> > > propagation
> > > > > > >> delays).
> > > > > > >> > > In
> > > > > > >> > > > >>>> any
> > > > > > >> > > > >>>> > case, this is a general issue, not specific to
> > > "create
> > > > > > >> topics".
> > > > > > >> > > > >>>> >
> > > > > > >> > > > >>>> > cheers,
> > > > > > >> > > > >>>> > Colin
> > > > > > >> > > > >>>> >
> > > > > > >> > > > >>>> >
> > > > > > >> > > > >>>> > >
> > > > > > >> > > > >>>> > >
> > > > > > >> > > > >>>> > > Xiongqi (Wesley) Wu
> > > > > > >> > > > >>>> > >
> > > > > > >> > > > >>>> > >
> > > > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> > > > > > >> > > cmccabe@apache.org
> > > > > > >> > > > >
> > > > > > >> > > > >>>> > wrote:
> > > > > > >> > > > >>>> > >
> > > > > > >> > > > >>>> > > > I don't have a strong opinion. But I think we
> > > should
> > > > > > >> > probably
> > > > > > >> > > be
> > > > > > >> > > > >>>> > > > consistent with how segment.ms works, and
> just
> > > use
> > > > > 0.
> > > > > > >> > > > >>>> > > >
> > > > > > >> > > > >>>> > > > best,
> > > > > > >> > > > >>>> > > > Colin
> > > > > > >> > > > >>>> > > >
> > > > > > >> > > > >>>> > > >
> > > > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann
> wrote:
> > > > > > >> > > > >>>> > > > > OK thanks for that clarification. I see why
> > > you're
> > > > > > >> > > > uncomfortable
> > > > > > >> > > > >>>> > with 0
> > > > > > >> > > > >>>> > > > now.
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer
> > > consistency in
> > > > > > >> > > > >>>> configuration
> > > > > > >> > > > >>>> > > > options.
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > > Personally I lean towards treating 0 and 1
> > > > > similarly
> > > > > > in
> > > > > > >> > that
> > > > > > >> > > > >>>> > scenario,
> > > > > > >> > > > >>>> > > > > because it favours the person thinking
> about
> > > > > setting
> > > > > > >> the
> > > > > > >> > > > >>>> > configurations,
> > > > > > >> > > > >>>> > > > > and a person doesn't care about a 1ms edge
> case
> > > > > > >> especially
> > > > > > >> > > > when
> > > > > > >> > > > >>>> the
> > > > > > >> > > > >>>> > > > context
> > > > > > >> > > > >>>> > > > > is the true minimum is tied to the log
> cleaner
> > > > > > cadence.
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because
> there
> > > is
> > > > > > some
> > > > > > >> > > > >>>> uniquness in
> > > > > > >> > > > >>>> > > > > segment.ms not being able to be set to 0,
> > > reduces
> > > > > > >> > > > configuration
> > > > > > >> > > > >>>> > > > consistency
> > > > > > >> > > > >>>> > > > > in favour of capturing a MS gap in an edge
> case
> > > > > that
> > > > > > >> > nobody
> > > > > > >> > > > >>>> would
> > > > > > >> > > > >>>> > ever
> > > > > > >> > > > >>>> > > > > notice. For someone to understand why
> > > everywhere
> > > > > else
> > > > > > >> -1
> > > > > > >> > is
> > > > > > >> > > > >>>> used to
> > > > > > >> > > > >>>> > > > > disable, but here 0 is used, they would
> need to
> > > > > learn
> > > > > > >> > about
> > > > > > >> > > > >>>> > segment.ms
> > > > > > >> > > > >>>> > > > > having a 1ms minimum and then after
> learning
> > > would
> > > > > > >> think
> > > > > > >> > > "who
> > > > > > >> > > > >>>> cares
> > > > > > >> > > > >>>> > about
> > > > > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to majority.
> > > Curious
> > > > > > >> which
> > > > > > >> > > way
> > > > > > >> > > > >>>> Colin
> > > > > > >> > > > >>>> > falls
> > > > > > >> > > > >>>> > > > > now.
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > > Don't want to spend more time on this
> though,
> > > It's
> > > > > > well
> > > > > > >> > into
> > > > > > >> > > > >>>> > > > bikeshedding
> > > > > > >> > > > >>>> > > > > territory now. :)
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > > > > > >> > > > xiongqiwu@gmail.com>
> > > > > > >> > > > >>>> > wrote:
> > > > > > >> > > > >>>> > > > >
> > > > > > >> > > > >>>> > > > > > I want to honor the minimum value of
> > > segment.ms
> > > > > > >> (which
> > > > > > >> > is
> > > > > > >> > > > >>>> 1ms) to
> > > > > > >> > > > >>>> > > > force
> > > > > > >> > > > >>>> > > > > > roll an active segment.
> > > > > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any
> > > value >
> > > > > > 0,
> > > > > > >> the
> > > > > > >> > > > >>>> minimum of
> > > > > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms
> will be
> > > > > used
> > > > > > to
> > > > > > >> > seal
> > > > > > >> > > > an
> > > > > > >> > > > >>>> > active
> > > > > > >> > > > >>>> > > > > > segment.
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0,
> the
> > > > > current
> > > > > > >> > > > >>>> implementation
> > > > > > >> > > > >>>> > will
> > > > > > >> > > > >>>> > > > > > treat it as disabled.
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > > It is a little bit weird to treat
> > > > > > >> max.compaction.lag=0
> > > > > > >> > the
> > > > > > >> > > > >>>> same as
> > > > > > >> > > > >>>> > > > > > max.compaction.lag=1.
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > > There might be a reason why we set the
> > > minimum of
> > > > > > >> > > > segment.ms
> > > > > > >> > > > >>>> to 1,
> > > > > > >> > > > >>>> > > > and I
> > > > > > >> > > > >>>> > > > > > don't want to break this assumption.
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > > > > > >> > > > >>>> > <br...@zendesk.com.invalid>
> > > > > > >> > > > >>>> > > > > > wrote:
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > > > > > You're rolling a new segment if the
> > > condition
> > > > > is
> > > > > > >> met
> > > > > > >> > > > right?
> > > > > > >> > > > >>>> So
> > > > > > >> > > > >>>> > I'm
> > > > > > >> > > > >>>> > > > > > > struggling to understand the relevance
> of
> > > > > > >> segment.ms
> > > > > > >> > > > here.
> > > > > > >> > > > >>>> > Maybe an
> > > > > > >> > > > >>>> > > > > > > example
> > > > > > >> > > > >>>> > > > > > > would help my understanding:
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in,
> after
> > > 1ms
> > > > > the
> > > > > > >> topic
> > > > > > >> > > > >>>> should be
> > > > > > >> > > > >>>> > > > eligible
> > > > > > >> > > > >>>> > > > > > > for compaction when the log compaction
> > > thread
> > > > > > gets
> > > > > > >> > > around
> > > > > > >> > > > to
> > > > > > >> > > > >>>> > > > evaluating
> > > > > > >> > > > >>>> > > > > > the
> > > > > > >> > > > >>>> > > > > > > topic.
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > > if we have
> > > > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > > When a duplicate message comes in,
> after
> > > 0ms
> > > > > the
> > > > > > >> topic
> > > > > > >> > > > >>>> should be
> > > > > > >> > > > >>>> > > > eligible
> > > > > > >> > > > >>>> > > > > > > for compaction when the log compaction
> > > thread
> > > > > > gets
> > > > > > >> > > around
> > > > > > >> > > > to
> > > > > > >> > > > >>>> > > > evaluating
> > > > > > >> > > > >>>> > > > > > the
> > > > > > >> > > > >>>> > > > > > > topic.
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > > In both of those cases the change would
> > > mean a
> > > > > > new
> > > > > > >> > > segment
> > > > > > >> > > > >>>> is
> > > > > > >> > > > >>>> > rolled
> > > > > > >> > > > >>>> > > > so
> > > > > > >> > > > >>>> > > > > > the
> > > > > > >> > > > >>>> > > > > > > new message would be part of the
> compaction
> > > > > > task. 0
> > > > > > >> > and
> > > > > > >> > > 1
> > > > > > >> > > > >>>> are
> > > > > > >> > > > >>>> > > > practically
> > > > > > >> > > > >>>> > > > > > > the same meaning since neither is
> > > providing an
> > > > > > >> actual
> > > > > > >> > > > >>>> guarantee
> > > > > > >> > > > >>>> > at
> > > > > > >> > > > >>>> > > > such
> > > > > > >> > > > >>>> > > > > > low
> > > > > > >> > > > >>>> > > > > > > MS settings, but effectively tying it
> to
> > > both
> > > > > the
> > > > > > >> > > > frequency
> > > > > > >> > > > >>>> of
> > > > > > >> > > > >>>> > the
> > > > > > >> > > > >>>> > > > log
> > > > > > >> > > > >>>> > > > > > > cleaner running and the priority of the
> > > given
> > > > > > topic
> > > > > > >> > > being
> > > > > > >> > > > >>>> the
> > > > > > >> > > > >>>> > highest
> > > > > > >> > > > >>>> > > > > > > priority of all topics that are
> evaluated
> > > for
> > > > > > >> cleaning
> > > > > > >> > > on
> > > > > > >> > > > >>>> the
> > > > > > >> > > > >>>> > next
> > > > > > >> > > > >>>> > > > cycle.
> > > > > > >> > > > >>>> > > > > > > You've captured that nuance with
> careful
> > > > > > "skipped"
> > > > > > >> > > wording
> > > > > > >> > > > >>>> in
> > > > > > >> > > > >>>> > the KIP
> > > > > > >> > > > >>>> > > > > > > here "controls
> > > > > > >> > > > >>>> > > > > > > the max time interval a message/segment
> > > can be
> > > > > > >> skipped
> > > > > > >> > > for
> > > > > > >> > > > >>>> log
> > > > > > >> > > > >>>> > > > > > compaction".
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > > How is 0 different to 1, practically?
> And
> > > how
> > > > > is
> > > > > > it
> > > > > > >> > > > >>>> relating to
> > > > > > >> > > > >>>> > > > > > segment.ms
> > > > > > >> > > > >>>> > > > > > > ?
> > > > > > >> > > > >>>> > > > > > > Is it that you're proposing to have 0
> mean
> > > "use
> > > > > > >> > > > segment.ms
> > > > > > >> > > > >>>> > > > instead?" as
> > > > > > >> > > > >>>> > > > > > a
> > > > > > >> > > > >>>> > > > > > > kind of third option?
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM
> xiongqi wu
> > > <
> > > > > > >> > > > >>>> xiongqiwu@gmail.com>
> > > > > > >> > > > >>>> > > > wrote:
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > > > > To make it clear,
> > > > > > >> > > > >>>> > > > > > > > I don't against using -1 as disabled,
> > > but we
> > > > > > >> need to
> > > > > > >> > > > come
> > > > > > >> > > > >>>> up
> > > > > > >> > > > >>>> > with
> > > > > > >> > > > >>>> > > > the
> > > > > > >> > > > >>>> > > > > > > > meaning of "0".
> > > > > > >> > > > >>>> > > > > > > > If "0" means immediate compaction,
> but
> > > the
> > > > > > actual
> > > > > > >> > > > >>>> compaction
> > > > > > >> > > > >>>> > lag
> > > > > > >> > > > >>>> > > > will
> > > > > > >> > > > >>>> > > > > > be
> > > > > > >> > > > >>>> > > > > > > > segment.ms.
> > > > > > >> > > > >>>> > > > > > > > It has longer lag than setting the
> value
> > > to
> > > > > be
> > > > > > >> half
> > > > > > >> > of
> > > > > > >> > > > >>>> > segment.ms.
> > > > > > >> > > > >>>> > > > > > > > We cannot provide "0" as max
> compaction
> > > lag.
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > Here are two options.
> > > > > > >> > > > >>>> > > > > > > > Option 1:
> > > > > > >> > > > >>>> > > > > > > > Keep 0 as disabled
> > > > > > >> > > > >>>> > > > > > > > Option 2:
> > > > > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag
> =
> > > > > > >> segment.ms),
> > > > > > >> > > and
> > > > > > >> > > > >>>> > others.
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett
> Rann
> > > > > > >> > > > >>>> > > > <br...@zendesk.com.invalid>
> > > > > > >> > > > >>>> > > > > > > > wrote:
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special" with
> > > these
> > > > > > >> settings
> > > > > > >> > > for
> > > > > > >> > > > >>>> > example:
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > log.retention.bytes
> > > > > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > > > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > > > > >> > > > >>>> > > > > > > > > queued.max.request.bytes
> > > > > > >> > > > >>>> > > > > > > > > retention.bytes
> > > > > > >> > > > >>>> > > > > > > > > retention.ms
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > and acks.
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS
> > > > > defaults,
> > > > > > >> max
> > > > > > >> > > > (acks),
> > > > > > >> > > > >>>> > etc. I
> > > > > > >> > > > >>>> > > > don't
> > > > > > >> > > > >>>> > > > > > > see
> > > > > > >> > > > >>>> > > > > > > > > much convention of 0 meaning those
> > > things.
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > There are some NULLs but it seems
> > > convetion
> > > > > > >> there
> > > > > > >> > is
> > > > > > >> > > > >>>> NULL is
> > > > > > >> > > > >>>> > used
> > > > > > >> > > > >>>> > > > > > where
> > > > > > >> > > > >>>> > > > > > > > > there's another setting in the
> > > hierarchy
> > > > > that
> > > > > > >> > would
> > > > > > >> > > be
> > > > > > >> > > > >>>> used
> > > > > > >> > > > >>>> > > > instead.
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM
> Brett
> > > Rann
> > > > > <
> > > > > > >> > > > >>>> > brann@zendesk.com>
> > > > > > >> > > > >>>> > > > > > wrote:
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0,
> > > then
> > > > > > we're
> > > > > > >> not
> > > > > > >> > > > being
> > > > > > >> > > > >>>> > > > consistent
> > > > > > >> > > > >>>> > > > > > > > > > by using 0 for this new setting?
> I
> > > throw
> > > > > > out
> > > > > > >> -1
> > > > > > >> > > for
> > > > > > >> > > > >>>> > > > consideration
> > > > > > >> > > > >>>> > > > > > > > > > again :)
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM
> > > xiongqi
> > > > > wu
> > > > > > <
> > > > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > > > >> > > > >>>> > > > > > > > wrote:
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document after
> PR is
> > > > > > merged.
> > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum
> of "
> > > > > > >> segment.ms"
> > > > > > >> > > to
> > > > > > >> > > > >>>> 1, we
> > > > > > >> > > > >>>> > > > cannot
> > > > > > >> > > > >>>> > > > > > set
> > > > > > >> > > > >>>> > > > > > > "
> > > > > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > > >> I also updated the title of this
> > > KIP.
> > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM
> Brett
> > > > > Rann
> > > > > > >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1
> > > since i'm
> > > > > > in
> > > > > > >> the
> > > > > > >> > > > >>>> minority.
> > > > > > >> > > > >>>> > :)
> > > > > > >> > > > >>>> > > > Can
> > > > > > >> > > > >>>> > > > > > we
> > > > > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as
> > > meaning
> > > > > > >> > disabled
> > > > > > >> > > > >>>> here:
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>>
> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >> > > > >>>> > <
> > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >> > > > >>>> > <
> > > https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >> > > > >>>> > <
> > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >> > > > >>>> > <
> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > >> > > > >>>>
> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >> > > > >>>> > <
> > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >> > > > >>>> > <
> > > https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >> > > > >>>> > <
> > > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > > >> > > > >>>> > <
> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >>>>
> > > > > > ?
> > > > > > >> > > > >>>> > > > > > > > > >> > And while there it would be
> good
> > > if
> > > > > > >> > segment.ms
> > > > > > >> > > > is
> > > > > > >> > > > >>>> > > > documented
> > > > > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too.
> (there's
> > > some
> > > > > > >> > hierarchy
> > > > > > >> > > > of
> > > > > > >> > > > >>>> > configs
> > > > > > >> > > > >>>> > > > for
> > > > > > >> > > > >>>> > > > > > > that
> > > > > > >> > > > >>>> > > > > > > > > too
> > > > > > >> > > > >>>> > > > > > > > > >> > if its not set and null for
> others
> > > > > means
> > > > > > >> > > > disabled!)
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM
> > > xiongqi
> > > > > > wu
> > > > > > >> <
> > > > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > > > >> > > > >>>> > > > > > > > > wrote:
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate
> > > immediate
> > > > > > >> > compaction,
> > > > > > >> > > > the
> > > > > > >> > > > >>>> > > > compaction
> > > > > > >> > > > >>>> > > > > > lag
> > > > > > >> > > > >>>> > > > > > > > is
> > > > > > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in
> > > worst
> > > > > > >> case. If
> > > > > > >> > > > >>>> segment.ms
> > > > > > >> > > > >>>> > is
> > > > > > >> > > > >>>> > > > 24
> > > > > > >> > > > >>>> > > > > > > > hours,
> > > > > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a
> > > weaker
> > > > > > >> > guarantee
> > > > > > >> > > > than
> > > > > > >> > > > >>>> > setting
> > > > > > >> > > > >>>> > > > any
> > > > > > >> > > > >>>> > > > > > > > value
> > > > > > >> > > > >>>> > > > > > > > > >> less
> > > > > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the
> > > definition of
> > > > > > "max
> > > > > > >> > > > >>>> compaction
> > > > > > >> > > > >>>> > lag",
> > > > > > >> > > > >>>> > > > we
> > > > > > >> > > > >>>> > > > > > > > cannot
> > > > > > >> > > > >>>> > > > > > > > > >> have
> > > > > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to
> indicate
> > > > > > >> "disable".
> > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34
> AM
> > > Colin
> > > > > > >> > McCabe <
> > > > > > >> > > > >>>> > > > > > cmccabe@apache.org
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at
> 22:11,
> > > > > Brett
> > > > > > >> Rann
> > > > > > >> > > > >>>> wrote:
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point.
> We
> > > should
> > > > > > >> make
> > > > > > >> > 0 =
> > > > > > >> > > > >>>> > disable, to
> > > > > > >> > > > >>>> > > > be
> > > > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for
> > > disable
> > > > > > and
> > > > > > >> > when
> > > > > > >> > > > >>>> seeing
> > > > > > >> > > > >>>> > it
> > > > > > >> > > > >>>> > > > in a
> > > > > > >> > > > >>>> > > > > > > > config
> > > > > > >> > > > >>>> > > > > > > > > >> it's
> > > > > > >> > > > >>>> > > > > > > > > >> > > > clear
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special
> > > meaning. 0
> > > > > > >> doesn't
> > > > > > >> > > have
> > > > > > >> > > > >>>> to
> > > > > > >> > > > >>>> > mean
> > > > > > >> > > > >>>> > > > > > instant,
> > > > > > >> > > > >>>> > > > > > > > it
> > > > > > >> > > > >>>> > > > > > > > > >> just
> > > > > > >> > > > >>>> > > > > > > > > >> > > > means
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible.
> I
> > > don't
> > > > > > >> think 0
> > > > > > >> > > is
> > > > > > >> > > > >>>> > intuitive
> > > > > > >> > > > >>>> > > > for
> > > > > > >> > > > >>>> > > > > > > > > disabled
> > > > > > >> > > > >>>> > > > > > > > > >> and
> > > > > > >> > > > >>>> > > > > > > > > >> > > it
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I
> wasn't
> > > > > aware
> > > > > > >> > > > segment.ms=0
> > > > > > >> > > > >>>> ==
> > > > > > >> > > > >>>> > > > > > disabled,
> > > > > > >> > > > >>>> > > > > > > > > but I
> > > > > > >> > > > >>>> > > > > > > > > >> > > think
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > that is also
> unintuitive.
> > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an
> argument
> > > for
> > > > > > >> keeping
> > > > > > >> > > > these
> > > > > > >> > > > >>>> two
> > > > > > >> > > > >>>> > > > > > > > configurations
> > > > > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they
> are so
> > > > > > >> similar. I
> > > > > > >> > > > agree
> > > > > > >> > > > >>>> that
> > > > > > >> > > > >>>> > 0
> > > > > > >> > > > >>>> > > > was an
> > > > > > >> > > > >>>> > > > > > > > > >> > unfortunate
> > > > > > >> > > > >>>> > > > > > > > > >> > > > choice.,
> > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > best,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > Colin
> > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at
> > > 11:38 AM
> > > > > > >> Colin
> > > > > > >> > > > McCabe
> > > > > > >> > > > >>>> <
> > > > > > >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > > > > > >> > > > >>>> > > > > > > > > >> > > wrote:
> > > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018,
> at
> > > 17:47,
> > > > > > >> > xiongqi
> > > > > > >> > > wu
> > > > > > >> > > > >>>> wrote:
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for
> comments.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply
> > > below.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018
> at
> > > 5:24
> > > > > PM
> > > > > > >> > Colin
> > > > > > >> > > > >>>> McCabe <
> > > > > > >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > > > > >> > > > >>>> > > > > > > > > >> > > > wrote:
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this
> KIP.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a
> bit
> > > > > > >> ambiguous.
> > > > > > >> > Our
> > > > > > >> > > > >>>> > compaction
> > > > > > >> > > > >>>> > > > > > > policies
> > > > > > >> > > > >>>> > > > > > > > > are
> > > > > > >> > > > >>>> > > > > > > > > >> > > already
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after
> > > all. It
> > > > > > >> seems
> > > > > > >> > > like
> > > > > > >> > > > >>>> this
> > > > > > >> > > > >>>> > > > change is
> > > > > > >> > > > >>>> > > > > > > > > focused
> > > > > > >> > > > >>>> > > > > > > > > >> > > around
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > adding
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “
> > > max.compaction.lag.ms
> > > > > ."
> > > > > > >> > Perhaps
> > > > > > >> > > > >>>> the KIP
> > > > > > >> > > > >>>> > > > title
> > > > > > >> > > > >>>> > > > > > > should
> > > > > > >> > > > >>>> > > > > > > > > be
> > > > > > >> > > > >>>> > > > > > > > > >> > > > something
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum
> > > > > compaction
> > > > > > >> lag
> > > > > > >> > > > time"?
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure.
> I
> > > will
> > > > > > >> change
> > > > > > >> > the
> > > > > > >> > > > >>>> title.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active
> segment is
> > > > > forced
> > > > > > >> to
> > > > > > >> > > roll
> > > > > > >> > > > >>>> when
> > > > > > >> > > > >>>> > > > either "
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms
> "
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms"
> (
> > > > > > >> log.roll.ms
> > > > > > >> > and
> > > > > > >> > > > >>>> > > > log.roll.hours)
> > > > > > >> > > > >>>> > > > > > > has
> > > > > > >> > > > >>>> > > > > > > > > >> > reached.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> > > > > > max.compaction.lag.ms
> > > > > > >> is
> > > > > > >> > > low,
> > > > > > >> > > > >>>> it
> > > > > > >> > > > >>>> > seems
> > > > > > >> > > > >>>> > > > like
> > > > > > >> > > > >>>> > > > > > > > > segments
> > > > > > >> > > > >>>> > > > > > > > > >> > will
> > > > > > >> > > > >>>> > > > > > > > > >> > > > be
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very
> > > frequently.
> > > > > This
> > > > > > >> can
> > > > > > >> > > be a
> > > > > > >> > > > >>>> > source of
> > > > > > >> > > > >>>> > > > > > > problems
> > > > > > >> > > > >>>> > > > > > > > > in
> > > > > > >> > > > >>>> > > > > > > > > >> the
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating
> many
> > > > > > different
> > > > > > >> > small
> > > > > > >> > > > log
> > > > > > >> > > > >>>> > segments
> > > > > > >> > > > >>>> > > > > > > > consumes
> > > > > > >> > > > >>>> > > > > > > > > a
> > > > > > >> > > > >>>> > > > > > > > > >> > huge
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > amount of
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources.
> > > > > > Therefore,
> > > > > > >> I
> > > > > > >> > > would
> > > > > > >> > > > >>>> > suggest
> > > > > > >> > > > >>>> > > > > > adding a
> > > > > > >> > > > >>>> > > > > > > > > >> > > broker-level
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration
> which
> > > allows
> > > > > > us
> > > > > > >> to
> > > > > > >> > > set
> > > > > > >> > > > a
> > > > > > >> > > > >>>> > minimum
> > > > > > >> > > > >>>> > > > value
> > > > > > >> > > > >>>> > > > > > > for
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> max.compaction.lag.ms.
> > > If
> > > > > > we
> > > > > > >> let
> > > > > > >> > > > >>>> users set
> > > > > > >> > > > >>>> > it
> > > > > > >> > > > >>>> > > > on a
> > > > > > >> > > > >>>> > > > > > > > > >> per-topic
> > > > > > >> > > > >>>> > > > > > > > > >> > > > basis,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set
> a
> > > value
> > > > > of
> > > > > > >> 1 ms
> > > > > > >> > > or
> > > > > > >> > > > >>>> > something,
> > > > > > >> > > > >>>> > > > and
> > > > > > >> > > > >>>> > > > > > > > cause
> > > > > > >> > > > >>>> > > > > > > > > >> > chaos.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this
> > > applies to
> > > > > > >> > > > segment.ms
> > > > > > >> > > > >>>> as
> > > > > > >> > > > >>>> > well.
> > > > > > >> > > > >>>> > > > > > Today
> > > > > > >> > > > >>>> > > > > > > > > users
> > > > > > >> > > > >>>> > > > > > > > > >> can
> > > > > > >> > > > >>>> > > > > > > > > >> > > > set "
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a
> very
> > > low
> > > > > > >> value,
> > > > > > >> > and
> > > > > > >> > > > >>>> cause a
> > > > > > >> > > > >>>> > > > frequent
> > > > > > >> > > > >>>> > > > > > > > > >> rolling of
> > > > > > >> > > > >>>> > > > > > > > > >> > > > active
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is
> an
> > > > > existing
> > > > > > >> > > problem
> > > > > > >> > > > >>>> with
> > > > > > >> > > > >>>> > > > > > segment.ms.
> > > > > > >> > > > >>>> > > > > > > > > >> However,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > that
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we
> > > shouldn't
> > > > > > fix
> > > > > > >> > it.
> > > > > > >> > > As
> > > > > > >> > > > >>>> you
> > > > > > >> > > > >>>> > noted,
> > > > > > >> > > > >>>> > > > > > there
> > > > > > >> > > > >>>> > > > > > > > > will
> > > > > > >> > > > >>>> > > > > > > > > >> be
> > > > > > >> > > > >>>> > > > > > > > > >> > > more
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these
> > > topic-level
> > > > > > >> > retention
> > > > > > >> > > > >>>> > settings as
> > > > > > >> > > > >>>> > > > a
> > > > > > >> > > > >>>> > > > > > > result
> > > > > > >> > > > >>>> > > > > > > > > of
> > > > > > >> > > > >>>> > > > > > > > > >> > GDPR.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > It
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that
> > > pre-existing
> > > > > > >> > problems
> > > > > > >> > > > >>>> will
> > > > > > >> > > > >>>> > cause
> > > > > > >> > > > >>>> > > > more
> > > > > > >> > > > >>>> > > > > > > > > trouble.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems
> relatively
> > > > > > >> > > straightforward
> > > > > > >> > > > >>>> here --
> > > > > > >> > > > >>>> > > > add a
> > > > > > >> > > > >>>> > > > > > > > > >> broker-level
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms
> that
> > > > > > overrides
> > > > > > >> > > > >>>> per-topic
> > > > > > >> > > > >>>> > > > minimums.
> > > > > > >> > > > >>>> > > > > > We
> > > > > > >> > > > >>>> > > > > > > > can
> > > > > > >> > > > >>>> > > > > > > > > >> also
> > > > > > >> > > > >>>> > > > > > > > > >> > > fail
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error
> > > message
> > > > > > when
> > > > > > >> > > someone
> > > > > > >> > > > >>>> > attempts
> > > > > > >> > > > >>>> > > > to
> > > > > > >> > > > >>>> > > > > > set
> > > > > > >> > > > >>>> > > > > > > an
> > > > > > >> > > > >>>> > > > > > > > > >> > invalid
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the
> > > minimum
> > > > > of "
> > > > > > >> > > > >>>> > > > max.compaction.lag.ms"
> > > > > > >> > > > >>>> > > > > > > > should
> > > > > > >> > > > >>>> > > > > > > > > >> be
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the
> minimum of
> > > "
> > > > > > >> > segment.ms
> > > > > > >> > > ".
> > > > > > >> > > > >>>> Since
> > > > > > >> > > > >>>> > > > today the
> > > > > > >> > > > >>>> > > > > > > > > >> minimum of
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "
> > > > > max.compaction.lag.ms"
> > > > > > >> also
> > > > > > >> > > > >>>> starts
> > > > > > >> > > > >>>> > with
> > > > > > >> > > > >>>> > > > 1. "0"
> > > > > > >> > > > >>>> > > > > > > > means
> > > > > > >> > > > >>>> > > > > > > > > >> > > > disable. I
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as
> disable,
> > > but
> > > > > it
> > > > > > is
> > > > > > >> > hard
> > > > > > >> > > > to
> > > > > > >> > > > >>>> > define
> > > > > > >> > > > >>>> > > > the
> > > > > > >> > > > >>>> > > > > > > > meaning
> > > > > > >> > > > >>>> > > > > > > > > >> of 0
> > > > > > >> > > > >>>> > > > > > > > > >> > > > because
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > we
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the
> > > active
> > > > > > >> segment
> > > > > > >> > > > >>>> > immediately.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point.
> We
> > > should
> > > > > > >> make
> > > > > > >> > 0 =
> > > > > > >> > > > >>>> > disable, to
> > > > > > >> > > > >>>> > > > be
> > > > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > best,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > Colin
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an
> > > > > alternative
> > > > > > >> > > > >>>> configuration
> > > > > > >> > > > >>>> > is to
> > > > > > >> > > > >>>> > > > use
> > > > > > >> > > > >>>> > > > > > -1
> > > > > > >> > > > >>>> > > > > > > > as
> > > > > > >> > > > >>>> > > > > > > > > >> > > > "disabled"
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > and
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate
> > > > > compaction".
> > > > > > >> > > Because
> > > > > > >> > > > >>>> > compaction
> > > > > > >> > > > >>>> > > > lag
> > > > > > >> > > > >>>> > > > > > is
> > > > > > >> > > > >>>> > > > > > > > > still
> > > > > > >> > > > >>>> > > > > > > > > >> > > > determined
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> > > > > > min.compaction.lag
> > > > > > >> and
> > > > > > >> > > how
> > > > > > >> > > > >>>> long
> > > > > > >> > > > >>>> > to
> > > > > > >> > > > >>>> > > > roll
> > > > > > >> > > > >>>> > > > > > an
> > > > > > >> > > > >>>> > > > > > > > > active
> > > > > > >> > > > >>>> > > > > > > > > >> > > > segment,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > the
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for
> > > > > compaction
> > > > > > is
> > > > > > >> > > > >>>> undetermined
> > > > > > >> > > > >>>> > if
> > > > > > >> > > > >>>> > > > we
> > > > > > >> > > > >>>> > > > > > use
> > > > > > >> > > > >>>> > > > > > > > "0".
> > > > > > >> > > > >>>> > > > > > > > > >> On
> > > > > > >> > > > >>>> > > > > > > > > >> > the
> > > > > > >> > > > >>>> > > > > > > > > >> > > > other
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can
> > > already set
> > > > > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > > > > >> > > > >>>> > > > > > > > to
> > > > > > >> > > > >>>> > > > > > > > > >> > achieve
> > > > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > same
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we
> > > choose
> > > > > > "0"
> > > > > > >> as
> > > > > > >> > > > >>>> "disabled".
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1
> to
> > > be
> > > > > the
> > > > > > >> > invalid
> > > > > > >> > > > >>>> setting.
> > > > > > >> > > > >>>> > > > > > Treating
> > > > > > >> > > > >>>> > > > > > > 0
> > > > > > >> > > > >>>> > > > > > > > > >> > > differently
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > than
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange
> to me.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my
> previous
> > > > > > >> comment, I
> > > > > > >> > > am
> > > > > > >> > > > >>>> not
> > > > > > >> > > > >>>> > > > strongly
> > > > > > >> > > > >>>> > > > > > > > against,
> > > > > > >> > > > >>>> > > > > > > > > >> but
> > > > > > >> > > > >>>> > > > > > > > > >> > 0
> > > > > > >> > > > >>>> > > > > > > > > >> > > is
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > not a
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration
> in
> > > my
> > > > > > >> option.
> > > > > > >> > So
> > > > > > >> > > I
> > > > > > >> > > > >>>> use
> > > > > > >> > > > >>>> > "0" as
> > > > > > >> > > > >>>> > > > > > > disabled
> > > > > > >> > > > >>>> > > > > > > > > >> state.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > best,
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4,
> 2018,
> > > at
> > > > > > 15:04,
> > > > > > >> > > > xiongqi
> > > > > > >> > > > >>>> wu
> > > > > > >> > > > >>>> > wrote:
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for
> this
> > > KIP.
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> >>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > >> > > > >>>> >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> >>>>
> > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > >> > > > >>>> > > >
> > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> >>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > >> > > > >>>> >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> >>>>>
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> >>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > >> > > > >>>> >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> >>>>
> > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > >> > > > >>>> > > >
> > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> >>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > >> > > > >>>> >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > <
> > > > > > >> > > > >>>>
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >
> > > > > > >> > > > >>>> > > > > > <
> > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > > > >>>> > <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > > >> > >>>>>>
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > >> > > > %3A+Time-based+log+compaction+policy
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > >> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > > > >> > > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > > >> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > > >> > > > >>>> > > > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > > >> > > > >>>> > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi
> (Wesley) Wu
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> > --
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> > Brett Rann
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne
> VIC
> > > 3000
> > > > > > >> > > Australia
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > > >> > > > >>>> > > > > > > > > >>
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > > --
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > > Brett Rann
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC
> > > 3000
> > > > > > >> Australia
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > > > >> > > > >>>> > > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > --
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > Brett Rann
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC
> 3000
> > > > > > >> Australia
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > > > >> > > > >>>> > > > > > > > >
> > > > > > >> > > > >>>> > > > > > > >
> > > > > > >> > > > >>>> > > > > > >
> > > > > > >> > > > >>>> > > > > >
> > > > > > >> > > > >>>> > > >
> > > > > > >> > > > >>>> >
> > > > > > >> > > > >>>>
> > > > > > >> > > > >>>
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by Colin McCabe <cm...@apache.org>.
Thanks, Xiongqi Wu. +1 (binding)
regards,
Colin
On Tue, Dec 4, 2018, at 20:58, xiongqi (wesley) wu wrote:
> Colin,
>
> Thanks for comments.
> Out of ordered message timestamp is a very good point.
> We can combine max.compaction.lag.ms with
> log.message.timestamp.difference.max.ms to achieve what we want in an
> environment that message timestamp can be shifted a lot.
>
> There are similar discussions regarding log.retention.ms and
> log.message.timestamp.difference.max.ms in KAFKA-4340.
>
> I agree that we can always use first message timestamp not the maxTimestamp
> of the previous log segment to make it simple.
>
> I have updated the KIP.
>
> Xiongqi (wesley) Wu
>
>
> On Tue, Dec 4, 2018 at 5:13 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Xiongqi,
> >
> > Thinking about this a little bit more, it seems like we don't have any
> > guarantees just by looking at the timestamp of the first message in a log
> > segment. Similarly, we don't have any guarantees just by looking at the
> > maxTimestamp of the previous log segment. Old data could appear anywhere--
> > you could put data that was years old in the middle of a segment from 2018.
> >
> > However, if log.message.timestamp.difference.max.ms is set, then we can
> > make some actual guarantees that old data gets purged-- which is what the
> > GDPR requires, of course.
> >
> > Overall, maybe we can make KIP-354 simpler by just always looking at the
> > timestamp of the first log message. I don't think looking at the
> > maxTimestamp of the previous segment is any more accurate. Aside from
> > that, looks good, since we can get what we need with the combination of
> > this and log.message.timestamp.difference.max.ms.
> >
> > best,
> > Colin
> >
> >
> > On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote:
> > > Thanks for binding and non-binding votes.
> > > Can I get one more binding vote?
> > >
> > > Thanks in advance!
> > >
> > > Xiongqi (Wesley) Wu
> > >
> > >
> > > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <ma...@frmr.me> wrote:
> > >
> > > > I'm a +1 (non-binding) — This looks like it would have saved us a lot
> > of
> > > > pain in an issue we had to debug recently. I can't go into details, but
> > > > figuring out how to achieve this effect gave me quite a headache. :)
> > > >
> > > > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xi...@gmail.com>
> > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Can I have one more vote on this KIP?
> > > > > Any comment is appreciated.
> > > > >
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
> > > > >
> > > > >
> > > > > Xiongqi (Wesley) Wu
> > > > >
> > > > >
> > > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xi...@gmail.com>
> > wrote:
> > > > >
> > > > > > Thanks Dong.
> > > > > > I have updated the KIP.
> > > > > >
> > > > > > Xiongqi (Wesley) Wu
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com>
> > wrote:
> > > > > >
> > > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> > > > > >>
> > > > > >> One minor comment: it may be a bit better to clarify in the public
> > > > > >> interface section that the value of the newly added metric is
> > > > determined
> > > > > >> based by applying that formula across all compactable segments.
> > For
> > > > > >> example:
> > > > > >>
> > > > > >> The maximum value of Math.max(now -
> > > > > >> earliest_timestamp_in_ms_of_uncompacted_segment -
> > > > max.compaction.lag.ms
> > > > > ,
> > > > > >> 0)/1000 across all compactable partitions, where the
> > > > > >> max.compaction.lag.ms
> > > > > >> can be overridden on per-topic basis.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com>
> > > > wrote:
> > > > > >>
> > > > > >> > Thanks Joel.
> > > > > >> > Tracking the delay at second granularity makes sense
> > > > > >> > I have updated KIP.
> > > > > >> >
> > > > > >> > Xiongqi (Wesley) Wu
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com>
> > > > > wrote:
> > > > > >> >
> > > > > >> > > +1 with one suggestion on the proposed metric. You should
> > probably
> > > > > >> > include
> > > > > >> > > the unit. So for e.g., max-compaction-delay-secs.
> > > > > >> > >
> > > > > >> > > Joel
> > > > > >> > >
> > > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <
> > xiongqiwu@gmail.com>
> > > > > >> wrote:
> > > > > >> > >
> > > > > >> > > > bump
> > > > > >> > > > Xiongqi (Wesley) Wu
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <
> > xiongqiwu@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> > > >
> > > > > >> > > > >
> > > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for
> > > > > >> feedback.
> > > > > >> > > > > Can I have more feedback or VOTE on this KIP?
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > Xiongqi (Wesley) Wu
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> > > > > xiongqiwu@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > >> Any other votes or comments?
> > > > > >> > > > >>
> > > > > >> > > > >> Xiongqi (Wesley) Wu
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> > > > > xiongqiwu@gmail.com
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > >>
> > > > > >> > > > >>> Yes, more votes and code review.
> > > > > >> > > > >>>
> > > > > >> > > > >>> Xiongqi (Wesley) Wu
> > > > > >> > > > >>>
> > > > > >> > > > >>>
> > > > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > > > > >> > > <brann@zendesk.com.invalid
> > > > > >> > > > >
> > > > > >> > > > >>> wrote:
> > > > > >> > > > >>>
> > > > > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> Where do we go from here? More votes?
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> > > > > >> cmccabe@apache.org>
> > > > > >> > > > >>>> wrote:
> > > > > >> > > > >>>>
> > > > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > > > > >> > > > >>>> > > Thank you for comments. I will use '0' for now.
> > > > > >> > > > >>>> > >
> > > > > >> > > > >>>> > > If we create topics through admin client in the
> > future,
> > > > > we
> > > > > >> > might
> > > > > >> > > > >>>> perform
> > > > > >> > > > >>>> > > some useful checks. (but the assumption is all
> > brokers
> > > > in
> > > > > >> the
> > > > > >> > > same
> > > > > >> > > > >>>> > cluster
> > > > > >> > > > >>>> > > have the same default configurations value,
> > > > otherwise,it
> > > > > >> might
> > > > > >> > > > >>>> still be
> > > > > >> > > > >>>> > > tricky to do such cross validation check.)
> > > > > >> > > > >>>> >
> > > > > >> > > > >>>> > This isn't something that we might do in the future--
> > > > this
> > > > > is
> > > > > >> > > > >>>> something we
> > > > > >> > > > >>>> > are doing now. We already have Create Topic policies
> > > > which
> > > > > >> are
> > > > > >> > > > >>>> enforced by
> > > > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details.
> > This
> > > > is
> > > > > >> one
> > > > > >> > of
> > > > > >> > > > the
> > > > > >> > > > >>>> > motivations for getting rid of direct ZK access--
> > making
> > > > > sure
> > > > > >> > that
> > > > > >> > > > >>>> these
> > > > > >> > > > >>>> > policies are applied.
> > > > > >> > > > >>>> >
> > > > > >> > > > >>>> > I agree that having different configurations on
> > different
> > > > > >> > brokers
> > > > > >> > > > can
> > > > > >> > > > >>>> be
> > > > > >> > > > >>>> > confusing and frustrating . That's why more
> > > > configurations
> > > > > >> are
> > > > > >> > > being
> > > > > >> > > > >>>> made
> > > > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are
> > stored
> > > > > >> > centrally
> > > > > >> > > > in
> > > > > >> > > > >>>> ZK,
> > > > > >> > > > >>>> > so they are the same on all brokers (modulo
> > propagation
> > > > > >> delays).
> > > > > >> > > In
> > > > > >> > > > >>>> any
> > > > > >> > > > >>>> > case, this is a general issue, not specific to
> > "create
> > > > > >> topics".
> > > > > >> > > > >>>> >
> > > > > >> > > > >>>> > cheers,
> > > > > >> > > > >>>> > Colin
> > > > > >> > > > >>>> >
> > > > > >> > > > >>>> >
> > > > > >> > > > >>>> > >
> > > > > >> > > > >>>> > >
> > > > > >> > > > >>>> > > Xiongqi (Wesley) Wu
> > > > > >> > > > >>>> > >
> > > > > >> > > > >>>> > >
> > > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> > > > > >> > > cmccabe@apache.org
> > > > > >> > > > >
> > > > > >> > > > >>>> > wrote:
> > > > > >> > > > >>>> > >
> > > > > >> > > > >>>> > > > I don't have a strong opinion. But I think we
> > should
> > > > > >> > probably
> > > > > >> > > be
> > > > > >> > > > >>>> > > > consistent with how segment.ms works, and just
> > use
> > > > 0.
> > > > > >> > > > >>>> > > >
> > > > > >> > > > >>>> > > > best,
> > > > > >> > > > >>>> > > > Colin
> > > > > >> > > > >>>> > > >
> > > > > >> > > > >>>> > > >
> > > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> > > > > >> > > > >>>> > > > > OK thanks for that clarification. I see why
> > you're
> > > > > >> > > > uncomfortable
> > > > > >> > > > >>>> > with 0
> > > > > >> > > > >>>> > > > now.
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer
> > consistency in
> > > > > >> > > > >>>> configuration
> > > > > >> > > > >>>> > > > options.
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > > Personally I lean towards treating 0 and 1
> > > > similarly
> > > > > in
> > > > > >> > that
> > > > > >> > > > >>>> > scenario,
> > > > > >> > > > >>>> > > > > because it favours the person thinking about
> > > > setting
> > > > > >> the
> > > > > >> > > > >>>> > configurations,
> > > > > >> > > > >>>> > > > > and a person doesn't care about a 1ms edge case
> > > > > >> especially
> > > > > >> > > > when
> > > > > >> > > > >>>> the
> > > > > >> > > > >>>> > > > context
> > > > > >> > > > >>>> > > > > is the true minimum is tied to the log cleaner
> > > > > cadence.
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because there
> > is
> > > > > some
> > > > > >> > > > >>>> uniquness in
> > > > > >> > > > >>>> > > > > segment.ms not being able to be set to 0,
> > reduces
> > > > > >> > > > configuration
> > > > > >> > > > >>>> > > > consistency
> > > > > >> > > > >>>> > > > > in favour of capturing a MS gap in an edge case
> > > > that
> > > > > >> > nobody
> > > > > >> > > > >>>> would
> > > > > >> > > > >>>> > ever
> > > > > >> > > > >>>> > > > > notice. For someone to understand why
> > everywhere
> > > > else
> > > > > >> -1
> > > > > >> > is
> > > > > >> > > > >>>> used to
> > > > > >> > > > >>>> > > > > disable, but here 0 is used, they would need to
> > > > learn
> > > > > >> > about
> > > > > >> > > > >>>> > segment.ms
> > > > > >> > > > >>>> > > > > having a 1ms minimum and then after learning
> > would
> > > > > >> think
> > > > > >> > > "who
> > > > > >> > > > >>>> cares
> > > > > >> > > > >>>> > about
> > > > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to majority.
> > Curious
> > > > > >> which
> > > > > >> > > way
> > > > > >> > > > >>>> Colin
> > > > > >> > > > >>>> > falls
> > > > > >> > > > >>>> > > > > now.
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > > Don't want to spend more time on this though,
> > It's
> > > > > well
> > > > > >> > into
> > > > > >> > > > >>>> > > > bikeshedding
> > > > > >> > > > >>>> > > > > territory now. :)
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > > > > >> > > > xiongqiwu@gmail.com>
> > > > > >> > > > >>>> > wrote:
> > > > > >> > > > >>>> > > > >
> > > > > >> > > > >>>> > > > > > I want to honor the minimum value of
> > segment.ms
> > > > > >> (which
> > > > > >> > is
> > > > > >> > > > >>>> 1ms) to
> > > > > >> > > > >>>> > > > force
> > > > > >> > > > >>>> > > > > > roll an active segment.
> > > > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any
> > value >
> > > > > 0,
> > > > > >> the
> > > > > >> > > > >>>> minimum of
> > > > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be
> > > > used
> > > > > to
> > > > > >> > seal
> > > > > >> > > > an
> > > > > >> > > > >>>> > active
> > > > > >> > > > >>>> > > > > > segment.
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the
> > > > current
> > > > > >> > > > >>>> implementation
> > > > > >> > > > >>>> > will
> > > > > >> > > > >>>> > > > > > treat it as disabled.
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > > It is a little bit weird to treat
> > > > > >> max.compaction.lag=0
> > > > > >> > the
> > > > > >> > > > >>>> same as
> > > > > >> > > > >>>> > > > > > max.compaction.lag=1.
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > > There might be a reason why we set the
> > minimum of
> > > > > >> > > > segment.ms
> > > > > >> > > > >>>> to 1,
> > > > > >> > > > >>>> > > > and I
> > > > > >> > > > >>>> > > > > > don't want to break this assumption.
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > > > > >> > > > >>>> > <br...@zendesk.com.invalid>
> > > > > >> > > > >>>> > > > > > wrote:
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > > > > > You're rolling a new segment if the
> > condition
> > > > is
> > > > > >> met
> > > > > >> > > > right?
> > > > > >> > > > >>>> So
> > > > > >> > > > >>>> > I'm
> > > > > >> > > > >>>> > > > > > > struggling to understand the relevance of
> > > > > >> segment.ms
> > > > > >> > > > here.
> > > > > >> > > > >>>> > Maybe an
> > > > > >> > > > >>>> > > > > > > example
> > > > > >> > > > >>>> > > > > > > would help my understanding:
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after
> > 1ms
> > > > the
> > > > > >> topic
> > > > > >> > > > >>>> should be
> > > > > >> > > > >>>> > > > eligible
> > > > > >> > > > >>>> > > > > > > for compaction when the log compaction
> > thread
> > > > > gets
> > > > > >> > > around
> > > > > >> > > > to
> > > > > >> > > > >>>> > > > evaluating
> > > > > >> > > > >>>> > > > > > the
> > > > > >> > > > >>>> > > > > > > topic.
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > > if we have
> > > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after
> > 0ms
> > > > the
> > > > > >> topic
> > > > > >> > > > >>>> should be
> > > > > >> > > > >>>> > > > eligible
> > > > > >> > > > >>>> > > > > > > for compaction when the log compaction
> > thread
> > > > > gets
> > > > > >> > > around
> > > > > >> > > > to
> > > > > >> > > > >>>> > > > evaluating
> > > > > >> > > > >>>> > > > > > the
> > > > > >> > > > >>>> > > > > > > topic.
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > > In both of those cases the change would
> > mean a
> > > > > new
> > > > > >> > > segment
> > > > > >> > > > >>>> is
> > > > > >> > > > >>>> > rolled
> > > > > >> > > > >>>> > > > so
> > > > > >> > > > >>>> > > > > > the
> > > > > >> > > > >>>> > > > > > > new message would be part of the compaction
> > > > > task. 0
> > > > > >> > and
> > > > > >> > > 1
> > > > > >> > > > >>>> are
> > > > > >> > > > >>>> > > > practically
> > > > > >> > > > >>>> > > > > > > the same meaning since neither is
> > providing an
> > > > > >> actual
> > > > > >> > > > >>>> guarantee
> > > > > >> > > > >>>> > at
> > > > > >> > > > >>>> > > > such
> > > > > >> > > > >>>> > > > > > low
> > > > > >> > > > >>>> > > > > > > MS settings, but effectively tying it to
> > both
> > > > the
> > > > > >> > > > frequency
> > > > > >> > > > >>>> of
> > > > > >> > > > >>>> > the
> > > > > >> > > > >>>> > > > log
> > > > > >> > > > >>>> > > > > > > cleaner running and the priority of the
> > given
> > > > > topic
> > > > > >> > > being
> > > > > >> > > > >>>> the
> > > > > >> > > > >>>> > highest
> > > > > >> > > > >>>> > > > > > > priority of all topics that are evaluated
> > for
> > > > > >> cleaning
> > > > > >> > > on
> > > > > >> > > > >>>> the
> > > > > >> > > > >>>> > next
> > > > > >> > > > >>>> > > > cycle.
> > > > > >> > > > >>>> > > > > > > You've captured that nuance with careful
> > > > > "skipped"
> > > > > >> > > wording
> > > > > >> > > > >>>> in
> > > > > >> > > > >>>> > the KIP
> > > > > >> > > > >>>> > > > > > > here "controls
> > > > > >> > > > >>>> > > > > > > the max time interval a message/segment
> > can be
> > > > > >> skipped
> > > > > >> > > for
> > > > > >> > > > >>>> log
> > > > > >> > > > >>>> > > > > > compaction".
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > > How is 0 different to 1, practically? And
> > how
> > > > is
> > > > > it
> > > > > >> > > > >>>> relating to
> > > > > >> > > > >>>> > > > > > segment.ms
> > > > > >> > > > >>>> > > > > > > ?
> > > > > >> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean
> > "use
> > > > > >> > > > segment.ms
> > > > > >> > > > >>>> > > > instead?" as
> > > > > >> > > > >>>> > > > > > a
> > > > > >> > > > >>>> > > > > > > kind of third option?
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu
> > <
> > > > > >> > > > >>>> xiongqiwu@gmail.com>
> > > > > >> > > > >>>> > > > wrote:
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > > > > To make it clear,
> > > > > >> > > > >>>> > > > > > > > I don't against using -1 as disabled,
> > but we
> > > > > >> need to
> > > > > >> > > > come
> > > > > >> > > > >>>> up
> > > > > >> > > > >>>> > with
> > > > > >> > > > >>>> > > > the
> > > > > >> > > > >>>> > > > > > > > meaning of "0".
> > > > > >> > > > >>>> > > > > > > > If "0" means immediate compaction, but
> > the
> > > > > actual
> > > > > >> > > > >>>> compaction
> > > > > >> > > > >>>> > lag
> > > > > >> > > > >>>> > > > will
> > > > > >> > > > >>>> > > > > > be
> > > > > >> > > > >>>> > > > > > > > segment.ms.
> > > > > >> > > > >>>> > > > > > > > It has longer lag than setting the value
> > to
> > > > be
> > > > > >> half
> > > > > >> > of
> > > > > >> > > > >>>> > segment.ms.
> > > > > >> > > > >>>> > > > > > > > We cannot provide "0" as max compaction
> > lag.
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > Here are two options.
> > > > > >> > > > >>>> > > > > > > > Option 1:
> > > > > >> > > > >>>> > > > > > > > Keep 0 as disabled
> > > > > >> > > > >>>> > > > > > > > Option 2:
> > > > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag =
> > > > > >> segment.ms),
> > > > > >> > > and
> > > > > >> > > > >>>> > others.
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> > > > > >> > > > >>>> > > > <br...@zendesk.com.invalid>
> > > > > >> > > > >>>> > > > > > > > wrote:
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special" with
> > these
> > > > > >> settings
> > > > > >> > > for
> > > > > >> > > > >>>> > example:
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > log.retention.bytes
> > > > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > > > >> > > > >>>> > > > > > > > > queued.max.request.bytes
> > > > > >> > > > >>>> > > > > > > > > retention.bytes
> > > > > >> > > > >>>> > > > > > > > > retention.ms
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > and acks.
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS
> > > > defaults,
> > > > > >> max
> > > > > >> > > > (acks),
> > > > > >> > > > >>>> > etc. I
> > > > > >> > > > >>>> > > > don't
> > > > > >> > > > >>>> > > > > > > see
> > > > > >> > > > >>>> > > > > > > > > much convention of 0 meaning those
> > things.
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > There are some NULLs but it seems
> > convetion
> > > > > >> there
> > > > > >> > is
> > > > > >> > > > >>>> NULL is
> > > > > >> > > > >>>> > used
> > > > > >> > > > >>>> > > > > > where
> > > > > >> > > > >>>> > > > > > > > > there's another setting in the
> > hierarchy
> > > > that
> > > > > >> > would
> > > > > >> > > be
> > > > > >> > > > >>>> used
> > > > > >> > > > >>>> > > > instead.
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett
> > Rann
> > > > <
> > > > > >> > > > >>>> > brann@zendesk.com>
> > > > > >> > > > >>>> > > > > > wrote:
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0,
> > then
> > > > > we're
> > > > > >> not
> > > > > >> > > > being
> > > > > >> > > > >>>> > > > consistent
> > > > > >> > > > >>>> > > > > > > > > > by using 0 for this new setting? I
> > throw
> > > > > out
> > > > > >> -1
> > > > > >> > > for
> > > > > >> > > > >>>> > > > consideration
> > > > > >> > > > >>>> > > > > > > > > > again :)
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM
> > xiongqi
> > > > wu
> > > > > <
> > > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > > >> > > > >>>> > > > > > > > wrote:
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is
> > > > > merged.
> > > > > >> > > > >>>> > > > > > > > > >>
> > > > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "
> > > > > >> segment.ms"
> > > > > >> > > to
> > > > > >> > > > >>>> 1, we
> > > > > >> > > > >>>> > > > cannot
> > > > > >> > > > >>>> > > > > > set
> > > > > >> > > > >>>> > > > > > > "
> > > > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > > > >> > > > >>>> > > > > > > > > >>
> > > > > >> > > > >>>> > > > > > > > > >> I also updated the title of this
> > KIP.
> > > > > >> > > > >>>> > > > > > > > > >>
> > > > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > > > >> > > > >>>> > > > > > > > > >>
> > > > > >> > > > >>>> > > > > > > > > >>
> > > > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett
> > > > Rann
> > > > > >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > > >> > > > >>>> > > > > > > > > >>
> > > > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1
> > since i'm
> > > > > in
> > > > > >> the
> > > > > >> > > > >>>> minority.
> > > > > >> > > > >>>> > :)
> > > > > >> > > > >>>> > > > Can
> > > > > >> > > > >>>> > > > > > we
> > > > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as
> > meaning
> > > > > >> > disabled
> > > > > >> > > > >>>> here:
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >> > > > >>>> > <
> > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > >> > > > >>>> > > > > > <
> > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >> > > > >>>> > <
> > https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >> > > > >>>> > <
> > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > >> > > > >>>> > > > > > <
> > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >> > > > >>>> > <
> > https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >> > > > >>>> > <
> > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > >> > > > >>>> > > > > > <
> > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >> > > > >>>> > <
> > https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >> > > > >>>> > <
> > https://kafka.apache.org/documentation/#brokerconfigs>
> > > > > >> > > > >>>> > > > > > <
> > > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > > >> > > > >>>> > <
> > https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>>
> > > > > ?
> > > > > >> > > > >>>> > > > > > > > > >> > And while there it would be good
> > if
> > > > > >> > segment.ms
> > > > > >> > > > is
> > > > > >> > > > >>>> > > > documented
> > > > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's
> > some
> > > > > >> > hierarchy
> > > > > >> > > > of
> > > > > >> > > > >>>> > configs
> > > > > >> > > > >>>> > > > for
> > > > > >> > > > >>>> > > > > > > that
> > > > > >> > > > >>>> > > > > > > > > too
> > > > > >> > > > >>>> > > > > > > > > >> > if its not set and null for others
> > > > means
> > > > > >> > > > disabled!)
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM
> > xiongqi
> > > > > wu
> > > > > >> <
> > > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > > >> > > > >>>> > > > > > > > > wrote:
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate
> > immediate
> > > > > >> > compaction,
> > > > > >> > > > the
> > > > > >> > > > >>>> > > > compaction
> > > > > >> > > > >>>> > > > > > lag
> > > > > >> > > > >>>> > > > > > > > is
> > > > > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in
> > worst
> > > > > >> case. If
> > > > > >> > > > >>>> segment.ms
> > > > > >> > > > >>>> > is
> > > > > >> > > > >>>> > > > 24
> > > > > >> > > > >>>> > > > > > > > hours,
> > > > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a
> > weaker
> > > > > >> > guarantee
> > > > > >> > > > than
> > > > > >> > > > >>>> > setting
> > > > > >> > > > >>>> > > > any
> > > > > >> > > > >>>> > > > > > > > value
> > > > > >> > > > >>>> > > > > > > > > >> less
> > > > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the
> > definition of
> > > > > "max
> > > > > >> > > > >>>> compaction
> > > > > >> > > > >>>> > lag",
> > > > > >> > > > >>>> > > > we
> > > > > >> > > > >>>> > > > > > > > cannot
> > > > > >> > > > >>>> > > > > > > > > >> have
> > > > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate
> > > > > >> "disable".
> > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM
> > Colin
> > > > > >> > McCabe <
> > > > > >> > > > >>>> > > > > > cmccabe@apache.org
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11,
> > > > Brett
> > > > > >> Rann
> > > > > >> > > > >>>> wrote:
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We
> > should
> > > > > >> make
> > > > > >> > 0 =
> > > > > >> > > > >>>> > disable, to
> > > > > >> > > > >>>> > > > be
> > > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > > >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for
> > disable
> > > > > and
> > > > > >> > when
> > > > > >> > > > >>>> seeing
> > > > > >> > > > >>>> > it
> > > > > >> > > > >>>> > > > in a
> > > > > >> > > > >>>> > > > > > > > config
> > > > > >> > > > >>>> > > > > > > > > >> it's
> > > > > >> > > > >>>> > > > > > > > > >> > > > clear
> > > > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special
> > meaning. 0
> > > > > >> doesn't
> > > > > >> > > have
> > > > > >> > > > >>>> to
> > > > > >> > > > >>>> > mean
> > > > > >> > > > >>>> > > > > > instant,
> > > > > >> > > > >>>> > > > > > > > it
> > > > > >> > > > >>>> > > > > > > > > >> just
> > > > > >> > > > >>>> > > > > > > > > >> > > > means
> > > > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I
> > don't
> > > > > >> think 0
> > > > > >> > > is
> > > > > >> > > > >>>> > intuitive
> > > > > >> > > > >>>> > > > for
> > > > > >> > > > >>>> > > > > > > > > disabled
> > > > > >> > > > >>>> > > > > > > > > >> and
> > > > > >> > > > >>>> > > > > > > > > >> > > it
> > > > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't
> > > > aware
> > > > > >> > > > segment.ms=0
> > > > > >> > > > >>>> ==
> > > > > >> > > > >>>> > > > > > disabled,
> > > > > >> > > > >>>> > > > > > > > > but I
> > > > > >> > > > >>>> > > > > > > > > >> > > think
> > > > > >> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an argument
> > for
> > > > > >> keeping
> > > > > >> > > > these
> > > > > >> > > > >>>> two
> > > > > >> > > > >>>> > > > > > > > configurations
> > > > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they are so
> > > > > >> similar. I
> > > > > >> > > > agree
> > > > > >> > > > >>>> that
> > > > > >> > > > >>>> > 0
> > > > > >> > > > >>>> > > > was an
> > > > > >> > > > >>>> > > > > > > > > >> > unfortunate
> > > > > >> > > > >>>> > > > > > > > > >> > > > choice.,
> > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > best,
> > > > > >> > > > >>>> > > > > > > > > >> > > > Colin
> > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at
> > 11:38 AM
> > > > > >> Colin
> > > > > >> > > > McCabe
> > > > > >> > > > >>>> <
> > > > > >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > > > > >> > > > >>>> > > > > > > > > >> > > wrote:
> > > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at
> > 17:47,
> > > > > >> > xiongqi
> > > > > >> > > wu
> > > > > >> > > > >>>> wrote:
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply
> > below.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at
> > 5:24
> > > > PM
> > > > > >> > Colin
> > > > > >> > > > >>>> McCabe <
> > > > > >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > > > >> > > > >>>> > > > > > > > > >> > > > wrote:
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit
> > > > > >> ambiguous.
> > > > > >> > Our
> > > > > >> > > > >>>> > compaction
> > > > > >> > > > >>>> > > > > > > policies
> > > > > >> > > > >>>> > > > > > > > > are
> > > > > >> > > > >>>> > > > > > > > > >> > > already
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after
> > all. It
> > > > > >> seems
> > > > > >> > > like
> > > > > >> > > > >>>> this
> > > > > >> > > > >>>> > > > change is
> > > > > >> > > > >>>> > > > > > > > > focused
> > > > > >> > > > >>>> > > > > > > > > >> > > around
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > adding
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “
> > max.compaction.lag.ms
> > > > ."
> > > > > >> > Perhaps
> > > > > >> > > > >>>> the KIP
> > > > > >> > > > >>>> > > > title
> > > > > >> > > > >>>> > > > > > > should
> > > > > >> > > > >>>> > > > > > > > > be
> > > > > >> > > > >>>> > > > > > > > > >> > > > something
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum
> > > > compaction
> > > > > >> lag
> > > > > >> > > > time"?
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I
> > will
> > > > > >> change
> > > > > >> > the
> > > > > >> > > > >>>> title.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is
> > > > forced
> > > > > >> to
> > > > > >> > > roll
> > > > > >> > > > >>>> when
> > > > > >> > > > >>>> > > > either "
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (
> > > > > >> log.roll.ms
> > > > > >> > and
> > > > > >> > > > >>>> > > > log.roll.hours)
> > > > > >> > > > >>>> > > > > > > has
> > > > > >> > > > >>>> > > > > > > > > >> > reached.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> > > > > max.compaction.lag.ms
> > > > > >> is
> > > > > >> > > low,
> > > > > >> > > > >>>> it
> > > > > >> > > > >>>> > seems
> > > > > >> > > > >>>> > > > like
> > > > > >> > > > >>>> > > > > > > > > segments
> > > > > >> > > > >>>> > > > > > > > > >> > will
> > > > > >> > > > >>>> > > > > > > > > >> > > > be
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very
> > frequently.
> > > > This
> > > > > >> can
> > > > > >> > > be a
> > > > > >> > > > >>>> > source of
> > > > > >> > > > >>>> > > > > > > problems
> > > > > >> > > > >>>> > > > > > > > > in
> > > > > >> > > > >>>> > > > > > > > > >> the
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating many
> > > > > different
> > > > > >> > small
> > > > > >> > > > log
> > > > > >> > > > >>>> > segments
> > > > > >> > > > >>>> > > > > > > > consumes
> > > > > >> > > > >>>> > > > > > > > > a
> > > > > >> > > > >>>> > > > > > > > > >> > huge
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > amount of
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources.
> > > > > Therefore,
> > > > > >> I
> > > > > >> > > would
> > > > > >> > > > >>>> > suggest
> > > > > >> > > > >>>> > > > > > adding a
> > > > > >> > > > >>>> > > > > > > > > >> > > broker-level
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration which
> > allows
> > > > > us
> > > > > >> to
> > > > > >> > > set
> > > > > >> > > > a
> > > > > >> > > > >>>> > minimum
> > > > > >> > > > >>>> > > > value
> > > > > >> > > > >>>> > > > > > > for
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms.
> > If
> > > > > we
> > > > > >> let
> > > > > >> > > > >>>> users set
> > > > > >> > > > >>>> > it
> > > > > >> > > > >>>> > > > on a
> > > > > >> > > > >>>> > > > > > > > > >> per-topic
> > > > > >> > > > >>>> > > > > > > > > >> > > > basis,
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a
> > value
> > > > of
> > > > > >> 1 ms
> > > > > >> > > or
> > > > > >> > > > >>>> > something,
> > > > > >> > > > >>>> > > > and
> > > > > >> > > > >>>> > > > > > > > cause
> > > > > >> > > > >>>> > > > > > > > > >> > chaos.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this
> > applies to
> > > > > >> > > > segment.ms
> > > > > >> > > > >>>> as
> > > > > >> > > > >>>> > well.
> > > > > >> > > > >>>> > > > > > Today
> > > > > >> > > > >>>> > > > > > > > > users
> > > > > >> > > > >>>> > > > > > > > > >> can
> > > > > >> > > > >>>> > > > > > > > > >> > > > set "
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very
> > low
> > > > > >> value,
> > > > > >> > and
> > > > > >> > > > >>>> cause a
> > > > > >> > > > >>>> > > > frequent
> > > > > >> > > > >>>> > > > > > > > > >> rolling of
> > > > > >> > > > >>>> > > > > > > > > >> > > > active
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an
> > > > existing
> > > > > >> > > problem
> > > > > >> > > > >>>> with
> > > > > >> > > > >>>> > > > > > segment.ms.
> > > > > >> > > > >>>> > > > > > > > > >> However,
> > > > > >> > > > >>>> > > > > > > > > >> > > > that
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we
> > shouldn't
> > > > > fix
> > > > > >> > it.
> > > > > >> > > As
> > > > > >> > > > >>>> you
> > > > > >> > > > >>>> > noted,
> > > > > >> > > > >>>> > > > > > there
> > > > > >> > > > >>>> > > > > > > > > will
> > > > > >> > > > >>>> > > > > > > > > >> be
> > > > > >> > > > >>>> > > > > > > > > >> > > more
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these
> > topic-level
> > > > > >> > retention
> > > > > >> > > > >>>> > settings as
> > > > > >> > > > >>>> > > > a
> > > > > >> > > > >>>> > > > > > > result
> > > > > >> > > > >>>> > > > > > > > > of
> > > > > >> > > > >>>> > > > > > > > > >> > GDPR.
> > > > > >> > > > >>>> > > > > > > > > >> > > > It
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that
> > pre-existing
> > > > > >> > problems
> > > > > >> > > > >>>> will
> > > > > >> > > > >>>> > cause
> > > > > >> > > > >>>> > > > more
> > > > > >> > > > >>>> > > > > > > > > trouble.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> > > > > >> > > straightforward
> > > > > >> > > > >>>> here --
> > > > > >> > > > >>>> > > > add a
> > > > > >> > > > >>>> > > > > > > > > >> broker-level
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that
> > > > > overrides
> > > > > >> > > > >>>> per-topic
> > > > > >> > > > >>>> > > > minimums.
> > > > > >> > > > >>>> > > > > > We
> > > > > >> > > > >>>> > > > > > > > can
> > > > > >> > > > >>>> > > > > > > > > >> also
> > > > > >> > > > >>>> > > > > > > > > >> > > fail
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error
> > message
> > > > > when
> > > > > >> > > someone
> > > > > >> > > > >>>> > attempts
> > > > > >> > > > >>>> > > > to
> > > > > >> > > > >>>> > > > > > set
> > > > > >> > > > >>>> > > > > > > an
> > > > > >> > > > >>>> > > > > > > > > >> > invalid
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the
> > minimum
> > > > of "
> > > > > >> > > > >>>> > > > max.compaction.lag.ms"
> > > > > >> > > > >>>> > > > > > > > should
> > > > > >> > > > >>>> > > > > > > > > >> be
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of
> > "
> > > > > >> > segment.ms
> > > > > >> > > ".
> > > > > >> > > > >>>> Since
> > > > > >> > > > >>>> > > > today the
> > > > > >> > > > >>>> > > > > > > > > >> minimum of
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "
> > > > max.compaction.lag.ms"
> > > > > >> also
> > > > > >> > > > >>>> starts
> > > > > >> > > > >>>> > with
> > > > > >> > > > >>>> > > > 1. "0"
> > > > > >> > > > >>>> > > > > > > > means
> > > > > >> > > > >>>> > > > > > > > > >> > > > disable. I
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable,
> > but
> > > > it
> > > > > is
> > > > > >> > hard
> > > > > >> > > > to
> > > > > >> > > > >>>> > define
> > > > > >> > > > >>>> > > > the
> > > > > >> > > > >>>> > > > > > > > meaning
> > > > > >> > > > >>>> > > > > > > > > >> of 0
> > > > > >> > > > >>>> > > > > > > > > >> > > > because
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > we
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the
> > active
> > > > > >> segment
> > > > > >> > > > >>>> > immediately.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We
> > should
> > > > > >> make
> > > > > >> > 0 =
> > > > > >> > > > >>>> > disable, to
> > > > > >> > > > >>>> > > > be
> > > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > best,
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > Colin
> > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an
> > > > alternative
> > > > > >> > > > >>>> configuration
> > > > > >> > > > >>>> > is to
> > > > > >> > > > >>>> > > > use
> > > > > >> > > > >>>> > > > > > -1
> > > > > >> > > > >>>> > > > > > > > as
> > > > > >> > > > >>>> > > > > > > > > >> > > > "disabled"
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > and
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate
> > > > compaction".
> > > > > >> > > Because
> > > > > >> > > > >>>> > compaction
> > > > > >> > > > >>>> > > > lag
> > > > > >> > > > >>>> > > > > > is
> > > > > >> > > > >>>> > > > > > > > > still
> > > > > >> > > > >>>> > > > > > > > > >> > > > determined
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> > > > > min.compaction.lag
> > > > > >> and
> > > > > >> > > how
> > > > > >> > > > >>>> long
> > > > > >> > > > >>>> > to
> > > > > >> > > > >>>> > > > roll
> > > > > >> > > > >>>> > > > > > an
> > > > > >> > > > >>>> > > > > > > > > active
> > > > > >> > > > >>>> > > > > > > > > >> > > > segment,
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > the
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for
> > > > compaction
> > > > > is
> > > > > >> > > > >>>> undetermined
> > > > > >> > > > >>>> > if
> > > > > >> > > > >>>> > > > we
> > > > > >> > > > >>>> > > > > > use
> > > > > >> > > > >>>> > > > > > > > "0".
> > > > > >> > > > >>>> > > > > > > > > >> On
> > > > > >> > > > >>>> > > > > > > > > >> > the
> > > > > >> > > > >>>> > > > > > > > > >> > > > other
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can
> > already set
> > > > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > > > >> > > > >>>> > > > > > > > to
> > > > > >> > > > >>>> > > > > > > > > >> > achieve
> > > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > same
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we
> > choose
> > > > > "0"
> > > > > >> as
> > > > > >> > > > >>>> "disabled".
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to
> > be
> > > > the
> > > > > >> > invalid
> > > > > >> > > > >>>> setting.
> > > > > >> > > > >>>> > > > > > Treating
> > > > > >> > > > >>>> > > > > > > 0
> > > > > >> > > > >>>> > > > > > > > > >> > > differently
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > than
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous
> > > > > >> comment, I
> > > > > >> > > am
> > > > > >> > > > >>>> not
> > > > > >> > > > >>>> > > > strongly
> > > > > >> > > > >>>> > > > > > > > against,
> > > > > >> > > > >>>> > > > > > > > > >> but
> > > > > >> > > > >>>> > > > > > > > > >> > 0
> > > > > >> > > > >>>> > > > > > > > > >> > > is
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > not a
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in
> > my
> > > > > >> option.
> > > > > >> > So
> > > > > >> > > I
> > > > > >> > > > >>>> use
> > > > > >> > > > >>>> > "0" as
> > > > > >> > > > >>>> > > > > > > disabled
> > > > > >> > > > >>>> > > > > > > > > >> state.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > best,
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018,
> > at
> > > > > 15:04,
> > > > > >> > > > xiongqi
> > > > > >> > > > >>>> wu
> > > > > >> > > > >>>> > wrote:
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this
> > KIP.
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> >>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > >> > > > >>>> >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> >>>>
> > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > >> > > > >>>> > > >
> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> >>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > >> > > > >>>> >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> >>>>>
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> >>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > >> > > > >>>> >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> >>>>
> > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > >> > > > >>>> > > >
> > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> >>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > >> > > > >>>> >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >>
> > > > > >> > > > >>>> > > > > > > > <
> > > > > >> > > > >>>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >
> > > > > >> > > > >>>> > > > > > <
> > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > > > >>>> > <
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > > >> > >>>>>>
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > >> > > > %3A+Time-based+log+compaction+policy
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > >> > > > >>>> > > > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > >> > > > >>>> > > > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > >> https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > >> > > > >>>> > > > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > >> > > > >>>> > > > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > > >> > > > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > >> > > > >>>> > > > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > >> > > > >>>> > > > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > > >> https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > >> > > > >>>> > > > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > > >> > > > >>>> > > > > > > > > >> <
> > > > > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > > >> > > > >>>> > > > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> > > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > > >> > > > >>>> > > > > > > > > >> > >
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> > --
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> > Brett Rann
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC
> > 3000
> > > > > >> > > Australia
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > > > > >> > > > >>>> > > > > > > > > >> >
> > > > > >> > > > >>>> > > > > > > > > >>
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > > --
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > > Brett Rann
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC
> > 3000
> > > > > >> Australia
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > > >> > > > >>>> > > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > --
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > Brett Rann
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> > > > > >> Australia
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > > >> > > > >>>> > > > > > > > >
> > > > > >> > > > >>>> > > > > > > >
> > > > > >> > > > >>>> > > > > > >
> > > > > >> > > > >>>> > > > > >
> > > > > >> > > > >>>> > > >
> > > > > >> > > > >>>> >
> > > > > >> > > > >>>>
> > > > > >> > > > >>>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> >
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by "xiongqi (wesley) wu" <sn...@gmail.com>.
Colin,
Thanks for comments.
Out of ordered message timestamp is a very good point.
We can combine max.compaction.lag.ms with
log.message.timestamp.difference.max.ms to achieve what we want in an
environment that message timestamp can be shifted a lot.
There are similar discussions regarding log.retention.ms and
log.message.timestamp.difference.max.ms in KAFKA-4340.
I agree that we can always use first message timestamp not the maxTimestamp
of the previous log segment to make it simple.
I have updated the KIP.
Xiongqi (wesley) Wu
On Tue, Dec 4, 2018 at 5:13 PM Colin McCabe <cm...@apache.org> wrote:
> Hi Xiongqi,
>
> Thinking about this a little bit more, it seems like we don't have any
> guarantees just by looking at the timestamp of the first message in a log
> segment. Similarly, we don't have any guarantees just by looking at the
> maxTimestamp of the previous log segment. Old data could appear anywhere--
> you could put data that was years old in the middle of a segment from 2018.
>
> However, if log.message.timestamp.difference.max.ms is set, then we can
> make some actual guarantees that old data gets purged-- which is what the
> GDPR requires, of course.
>
> Overall, maybe we can make KIP-354 simpler by just always looking at the
> timestamp of the first log message. I don't think looking at the
> maxTimestamp of the previous segment is any more accurate. Aside from
> that, looks good, since we can get what we need with the combination of
> this and log.message.timestamp.difference.max.ms.
>
> best,
> Colin
>
>
> On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote:
> > Thanks for binding and non-binding votes.
> > Can I get one more binding vote?
> >
> > Thanks in advance!
> >
> > Xiongqi (Wesley) Wu
> >
> >
> > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <ma...@frmr.me> wrote:
> >
> > > I'm a +1 (non-binding) — This looks like it would have saved us a lot
> of
> > > pain in an issue we had to debug recently. I can't go into details, but
> > > figuring out how to achieve this effect gave me quite a headache. :)
> > >
> > > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Can I have one more vote on this KIP?
> > > > Any comment is appreciated.
> > > >
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
> > > >
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > > >
> > > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xi...@gmail.com>
> wrote:
> > > >
> > > > > Thanks Dong.
> > > > > I have updated the KIP.
> > > > >
> > > > > Xiongqi (Wesley) Wu
> > > > >
> > > > >
> > > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com>
> wrote:
> > > > >
> > > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> > > > >>
> > > > >> One minor comment: it may be a bit better to clarify in the public
> > > > >> interface section that the value of the newly added metric is
> > > determined
> > > > >> based by applying that formula across all compactable segments.
> For
> > > > >> example:
> > > > >>
> > > > >> The maximum value of Math.max(now -
> > > > >> earliest_timestamp_in_ms_of_uncompacted_segment -
> > > max.compaction.lag.ms
> > > > ,
> > > > >> 0)/1000 across all compactable partitions, where the
> > > > >> max.compaction.lag.ms
> > > > >> can be overridden on per-topic basis.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > Thanks Joel.
> > > > >> > Tracking the delay at second granularity makes sense
> > > > >> > I have updated KIP.
> > > > >> >
> > > > >> > Xiongqi (Wesley) Wu
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > +1 with one suggestion on the proposed metric. You should
> probably
> > > > >> > include
> > > > >> > > the unit. So for e.g., max-compaction-delay-secs.
> > > > >> > >
> > > > >> > > Joel
> > > > >> > >
> > > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <
> xiongqiwu@gmail.com>
> > > > >> wrote:
> > > > >> > >
> > > > >> > > > bump
> > > > >> > > > Xiongqi (Wesley) Wu
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <
> xiongqiwu@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for
> > > > >> feedback.
> > > > >> > > > > Can I have more feedback or VOTE on this KIP?
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> > > > xiongqiwu@gmail.com>
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > >> Any other votes or comments?
> > > > >> > > > >>
> > > > >> > > > >> Xiongqi (Wesley) Wu
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> > > > xiongqiwu@gmail.com
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > >>
> > > > >> > > > >>> Yes, more votes and code review.
> > > > >> > > > >>>
> > > > >> > > > >>> Xiongqi (Wesley) Wu
> > > > >> > > > >>>
> > > > >> > > > >>>
> > > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > > > >> > > <brann@zendesk.com.invalid
> > > > >> > > > >
> > > > >> > > > >>> wrote:
> > > > >> > > > >>>
> > > > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > > > >> > > > >>>>
> > > > >> > > > >>>> Where do we go from here? More votes?
> > > > >> > > > >>>>
> > > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> > > > >> cmccabe@apache.org>
> > > > >> > > > >>>> wrote:
> > > > >> > > > >>>>
> > > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > > > >> > > > >>>> > > Thank you for comments. I will use '0' for now.
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > > If we create topics through admin client in the
> future,
> > > > we
> > > > >> > might
> > > > >> > > > >>>> perform
> > > > >> > > > >>>> > > some useful checks. (but the assumption is all
> brokers
> > > in
> > > > >> the
> > > > >> > > same
> > > > >> > > > >>>> > cluster
> > > > >> > > > >>>> > > have the same default configurations value,
> > > otherwise,it
> > > > >> might
> > > > >> > > > >>>> still be
> > > > >> > > > >>>> > > tricky to do such cross validation check.)
> > > > >> > > > >>>> >
> > > > >> > > > >>>> > This isn't something that we might do in the future--
> > > this
> > > > is
> > > > >> > > > >>>> something we
> > > > >> > > > >>>> > are doing now. We already have Create Topic policies
> > > which
> > > > >> are
> > > > >> > > > >>>> enforced by
> > > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details.
> This
> > > is
> > > > >> one
> > > > >> > of
> > > > >> > > > the
> > > > >> > > > >>>> > motivations for getting rid of direct ZK access--
> making
> > > > sure
> > > > >> > that
> > > > >> > > > >>>> these
> > > > >> > > > >>>> > policies are applied.
> > > > >> > > > >>>> >
> > > > >> > > > >>>> > I agree that having different configurations on
> different
> > > > >> > brokers
> > > > >> > > > can
> > > > >> > > > >>>> be
> > > > >> > > > >>>> > confusing and frustrating . That's why more
> > > configurations
> > > > >> are
> > > > >> > > being
> > > > >> > > > >>>> made
> > > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are
> stored
> > > > >> > centrally
> > > > >> > > > in
> > > > >> > > > >>>> ZK,
> > > > >> > > > >>>> > so they are the same on all brokers (modulo
> propagation
> > > > >> delays).
> > > > >> > > In
> > > > >> > > > >>>> any
> > > > >> > > > >>>> > case, this is a general issue, not specific to
> "create
> > > > >> topics".
> > > > >> > > > >>>> >
> > > > >> > > > >>>> > cheers,
> > > > >> > > > >>>> > Colin
> > > > >> > > > >>>> >
> > > > >> > > > >>>> >
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> > > > >> > > cmccabe@apache.org
> > > > >> > > > >
> > > > >> > > > >>>> > wrote:
> > > > >> > > > >>>> > >
> > > > >> > > > >>>> > > > I don't have a strong opinion. But I think we
> should
> > > > >> > probably
> > > > >> > > be
> > > > >> > > > >>>> > > > consistent with how segment.ms works, and just
> use
> > > 0.
> > > > >> > > > >>>> > > >
> > > > >> > > > >>>> > > > best,
> > > > >> > > > >>>> > > > Colin
> > > > >> > > > >>>> > > >
> > > > >> > > > >>>> > > >
> > > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> > > > >> > > > >>>> > > > > OK thanks for that clarification. I see why
> you're
> > > > >> > > > uncomfortable
> > > > >> > > > >>>> > with 0
> > > > >> > > > >>>> > > > now.
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > I'm not really fussed. I just prefer
> consistency in
> > > > >> > > > >>>> configuration
> > > > >> > > > >>>> > > > options.
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > Personally I lean towards treating 0 and 1
> > > similarly
> > > > in
> > > > >> > that
> > > > >> > > > >>>> > scenario,
> > > > >> > > > >>>> > > > > because it favours the person thinking about
> > > setting
> > > > >> the
> > > > >> > > > >>>> > configurations,
> > > > >> > > > >>>> > > > > and a person doesn't care about a 1ms edge case
> > > > >> especially
> > > > >> > > > when
> > > > >> > > > >>>> the
> > > > >> > > > >>>> > > > context
> > > > >> > > > >>>> > > > > is the true minimum is tied to the log cleaner
> > > > cadence.
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because there
> is
> > > > some
> > > > >> > > > >>>> uniquness in
> > > > >> > > > >>>> > > > > segment.ms not being able to be set to 0,
> reduces
> > > > >> > > > configuration
> > > > >> > > > >>>> > > > consistency
> > > > >> > > > >>>> > > > > in favour of capturing a MS gap in an edge case
> > > that
> > > > >> > nobody
> > > > >> > > > >>>> would
> > > > >> > > > >>>> > ever
> > > > >> > > > >>>> > > > > notice. For someone to understand why
> everywhere
> > > else
> > > > >> -1
> > > > >> > is
> > > > >> > > > >>>> used to
> > > > >> > > > >>>> > > > > disable, but here 0 is used, they would need to
> > > learn
> > > > >> > about
> > > > >> > > > >>>> > segment.ms
> > > > >> > > > >>>> > > > > having a 1ms minimum and then after learning
> would
> > > > >> think
> > > > >> > > "who
> > > > >> > > > >>>> cares
> > > > >> > > > >>>> > about
> > > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > my 2c anyway. Will again defer to majority.
> Curious
> > > > >> which
> > > > >> > > way
> > > > >> > > > >>>> Colin
> > > > >> > > > >>>> > falls
> > > > >> > > > >>>> > > > > now.
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > Don't want to spend more time on this though,
> It's
> > > > well
> > > > >> > into
> > > > >> > > > >>>> > > > bikeshedding
> > > > >> > > > >>>> > > > > territory now. :)
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > > > >> > > > xiongqiwu@gmail.com>
> > > > >> > > > >>>> > wrote:
> > > > >> > > > >>>> > > > >
> > > > >> > > > >>>> > > > > > I want to honor the minimum value of
> segment.ms
> > > > >> (which
> > > > >> > is
> > > > >> > > > >>>> 1ms) to
> > > > >> > > > >>>> > > > force
> > > > >> > > > >>>> > > > > > roll an active segment.
> > > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any
> value >
> > > > 0,
> > > > >> the
> > > > >> > > > >>>> minimum of
> > > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be
> > > used
> > > > to
> > > > >> > seal
> > > > >> > > > an
> > > > >> > > > >>>> > active
> > > > >> > > > >>>> > > > > > segment.
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the
> > > current
> > > > >> > > > >>>> implementation
> > > > >> > > > >>>> > will
> > > > >> > > > >>>> > > > > > treat it as disabled.
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > It is a little bit weird to treat
> > > > >> max.compaction.lag=0
> > > > >> > the
> > > > >> > > > >>>> same as
> > > > >> > > > >>>> > > > > > max.compaction.lag=1.
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > There might be a reason why we set the
> minimum of
> > > > >> > > > segment.ms
> > > > >> > > > >>>> to 1,
> > > > >> > > > >>>> > > > and I
> > > > >> > > > >>>> > > > > > don't want to break this assumption.
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > > > >> > > > >>>> > <br...@zendesk.com.invalid>
> > > > >> > > > >>>> > > > > > wrote:
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > > > > > You're rolling a new segment if the
> condition
> > > is
> > > > >> met
> > > > >> > > > right?
> > > > >> > > > >>>> So
> > > > >> > > > >>>> > I'm
> > > > >> > > > >>>> > > > > > > struggling to understand the relevance of
> > > > >> segment.ms
> > > > >> > > > here.
> > > > >> > > > >>>> > Maybe an
> > > > >> > > > >>>> > > > > > > example
> > > > >> > > > >>>> > > > > > > would help my understanding:
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after
> 1ms
> > > the
> > > > >> topic
> > > > >> > > > >>>> should be
> > > > >> > > > >>>> > > > eligible
> > > > >> > > > >>>> > > > > > > for compaction when the log compaction
> thread
> > > > gets
> > > > >> > > around
> > > > >> > > > to
> > > > >> > > > >>>> > > > evaluating
> > > > >> > > > >>>> > > > > > the
> > > > >> > > > >>>> > > > > > > topic.
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > if we have
> > > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > When a duplicate message comes in, after
> 0ms
> > > the
> > > > >> topic
> > > > >> > > > >>>> should be
> > > > >> > > > >>>> > > > eligible
> > > > >> > > > >>>> > > > > > > for compaction when the log compaction
> thread
> > > > gets
> > > > >> > > around
> > > > >> > > > to
> > > > >> > > > >>>> > > > evaluating
> > > > >> > > > >>>> > > > > > the
> > > > >> > > > >>>> > > > > > > topic.
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > In both of those cases the change would
> mean a
> > > > new
> > > > >> > > segment
> > > > >> > > > >>>> is
> > > > >> > > > >>>> > rolled
> > > > >> > > > >>>> > > > so
> > > > >> > > > >>>> > > > > > the
> > > > >> > > > >>>> > > > > > > new message would be part of the compaction
> > > > task. 0
> > > > >> > and
> > > > >> > > 1
> > > > >> > > > >>>> are
> > > > >> > > > >>>> > > > practically
> > > > >> > > > >>>> > > > > > > the same meaning since neither is
> providing an
> > > > >> actual
> > > > >> > > > >>>> guarantee
> > > > >> > > > >>>> > at
> > > > >> > > > >>>> > > > such
> > > > >> > > > >>>> > > > > > low
> > > > >> > > > >>>> > > > > > > MS settings, but effectively tying it to
> both
> > > the
> > > > >> > > > frequency
> > > > >> > > > >>>> of
> > > > >> > > > >>>> > the
> > > > >> > > > >>>> > > > log
> > > > >> > > > >>>> > > > > > > cleaner running and the priority of the
> given
> > > > topic
> > > > >> > > being
> > > > >> > > > >>>> the
> > > > >> > > > >>>> > highest
> > > > >> > > > >>>> > > > > > > priority of all topics that are evaluated
> for
> > > > >> cleaning
> > > > >> > > on
> > > > >> > > > >>>> the
> > > > >> > > > >>>> > next
> > > > >> > > > >>>> > > > cycle.
> > > > >> > > > >>>> > > > > > > You've captured that nuance with careful
> > > > "skipped"
> > > > >> > > wording
> > > > >> > > > >>>> in
> > > > >> > > > >>>> > the KIP
> > > > >> > > > >>>> > > > > > > here "controls
> > > > >> > > > >>>> > > > > > > the max time interval a message/segment
> can be
> > > > >> skipped
> > > > >> > > for
> > > > >> > > > >>>> log
> > > > >> > > > >>>> > > > > > compaction".
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > How is 0 different to 1, practically? And
> how
> > > is
> > > > it
> > > > >> > > > >>>> relating to
> > > > >> > > > >>>> > > > > > segment.ms
> > > > >> > > > >>>> > > > > > > ?
> > > > >> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean
> "use
> > > > >> > > > segment.ms
> > > > >> > > > >>>> > > > instead?" as
> > > > >> > > > >>>> > > > > > a
> > > > >> > > > >>>> > > > > > > kind of third option?
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu
> <
> > > > >> > > > >>>> xiongqiwu@gmail.com>
> > > > >> > > > >>>> > > > wrote:
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > > > > To make it clear,
> > > > >> > > > >>>> > > > > > > > I don't against using -1 as disabled,
> but we
> > > > >> need to
> > > > >> > > > come
> > > > >> > > > >>>> up
> > > > >> > > > >>>> > with
> > > > >> > > > >>>> > > > the
> > > > >> > > > >>>> > > > > > > > meaning of "0".
> > > > >> > > > >>>> > > > > > > > If "0" means immediate compaction, but
> the
> > > > actual
> > > > >> > > > >>>> compaction
> > > > >> > > > >>>> > lag
> > > > >> > > > >>>> > > > will
> > > > >> > > > >>>> > > > > > be
> > > > >> > > > >>>> > > > > > > > segment.ms.
> > > > >> > > > >>>> > > > > > > > It has longer lag than setting the value
> to
> > > be
> > > > >> half
> > > > >> > of
> > > > >> > > > >>>> > segment.ms.
> > > > >> > > > >>>> > > > > > > > We cannot provide "0" as max compaction
> lag.
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > Here are two options.
> > > > >> > > > >>>> > > > > > > > Option 1:
> > > > >> > > > >>>> > > > > > > > Keep 0 as disabled
> > > > >> > > > >>>> > > > > > > > Option 2:
> > > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag =
> > > > >> segment.ms),
> > > > >> > > and
> > > > >> > > > >>>> > others.
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> > > > >> > > > >>>> > > > <br...@zendesk.com.invalid>
> > > > >> > > > >>>> > > > > > > > wrote:
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > -1 is consistent as "special" with
> these
> > > > >> settings
> > > > >> > > for
> > > > >> > > > >>>> > example:
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > log.retention.bytes
> > > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > > >> > > > >>>> > > > > > > > > queued.max.request.bytes
> > > > >> > > > >>>> > > > > > > > > retention.bytes
> > > > >> > > > >>>> > > > > > > > > retention.ms
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > and acks.
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS
> > > defaults,
> > > > >> max
> > > > >> > > > (acks),
> > > > >> > > > >>>> > etc. I
> > > > >> > > > >>>> > > > don't
> > > > >> > > > >>>> > > > > > > see
> > > > >> > > > >>>> > > > > > > > > much convention of 0 meaning those
> things.
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > There are some NULLs but it seems
> convetion
> > > > >> there
> > > > >> > is
> > > > >> > > > >>>> NULL is
> > > > >> > > > >>>> > used
> > > > >> > > > >>>> > > > > > where
> > > > >> > > > >>>> > > > > > > > > there's another setting in the
> hierarchy
> > > that
> > > > >> > would
> > > > >> > > be
> > > > >> > > > >>>> used
> > > > >> > > > >>>> > > > instead.
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett
> Rann
> > > <
> > > > >> > > > >>>> > brann@zendesk.com>
> > > > >> > > > >>>> > > > > > wrote:
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0,
> then
> > > > we're
> > > > >> not
> > > > >> > > > being
> > > > >> > > > >>>> > > > consistent
> > > > >> > > > >>>> > > > > > > > > > by using 0 for this new setting? I
> throw
> > > > out
> > > > >> -1
> > > > >> > > for
> > > > >> > > > >>>> > > > consideration
> > > > >> > > > >>>> > > > > > > > > > again :)
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM
> xiongqi
> > > wu
> > > > <
> > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > >> > > > >>>> > > > > > > > wrote:
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is
> > > > merged.
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "
> > > > >> segment.ms"
> > > > >> > > to
> > > > >> > > > >>>> 1, we
> > > > >> > > > >>>> > > > cannot
> > > > >> > > > >>>> > > > > > set
> > > > >> > > > >>>> > > > > > > "
> > > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> I also updated the title of this
> KIP.
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett
> > > Rann
> > > > >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1
> since i'm
> > > > in
> > > > >> the
> > > > >> > > > >>>> minority.
> > > > >> > > > >>>> > :)
> > > > >> > > > >>>> > > > Can
> > > > >> > > > >>>> > > > > > we
> > > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as
> meaning
> > > > >> > disabled
> > > > >> > > > >>>> here:
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >> > > > >>>> > > > > > <
> > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >> > > > >>>> > > > > > <
> > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >> > > > >>>> > > > > > <
> > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >> > > > >>>> > > > > > <
> > > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >> > > > >>>> > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>>
> > > > ?
> > > > >> > > > >>>> > > > > > > > > >> > And while there it would be good
> if
> > > > >> > segment.ms
> > > > >> > > > is
> > > > >> > > > >>>> > > > documented
> > > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's
> some
> > > > >> > hierarchy
> > > > >> > > > of
> > > > >> > > > >>>> > configs
> > > > >> > > > >>>> > > > for
> > > > >> > > > >>>> > > > > > > that
> > > > >> > > > >>>> > > > > > > > > too
> > > > >> > > > >>>> > > > > > > > > >> > if its not set and null for others
> > > means
> > > > >> > > > disabled!)
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM
> xiongqi
> > > > wu
> > > > >> <
> > > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > >> > > > >>>> > > > > > > > > wrote:
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate
> immediate
> > > > >> > compaction,
> > > > >> > > > the
> > > > >> > > > >>>> > > > compaction
> > > > >> > > > >>>> > > > > > lag
> > > > >> > > > >>>> > > > > > > > is
> > > > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in
> worst
> > > > >> case. If
> > > > >> > > > >>>> segment.ms
> > > > >> > > > >>>> > is
> > > > >> > > > >>>> > > > 24
> > > > >> > > > >>>> > > > > > > > hours,
> > > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a
> weaker
> > > > >> > guarantee
> > > > >> > > > than
> > > > >> > > > >>>> > setting
> > > > >> > > > >>>> > > > any
> > > > >> > > > >>>> > > > > > > > value
> > > > >> > > > >>>> > > > > > > > > >> less
> > > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the
> definition of
> > > > "max
> > > > >> > > > >>>> compaction
> > > > >> > > > >>>> > lag",
> > > > >> > > > >>>> > > > we
> > > > >> > > > >>>> > > > > > > > cannot
> > > > >> > > > >>>> > > > > > > > > >> have
> > > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate
> > > > >> "disable".
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM
> Colin
> > > > >> > McCabe <
> > > > >> > > > >>>> > > > > > cmccabe@apache.org
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> wrote:
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11,
> > > Brett
> > > > >> Rann
> > > > >> > > > >>>> wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We
> should
> > > > >> make
> > > > >> > 0 =
> > > > >> > > > >>>> > disable, to
> > > > >> > > > >>>> > > > be
> > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for
> disable
> > > > and
> > > > >> > when
> > > > >> > > > >>>> seeing
> > > > >> > > > >>>> > it
> > > > >> > > > >>>> > > > in a
> > > > >> > > > >>>> > > > > > > > config
> > > > >> > > > >>>> > > > > > > > > >> it's
> > > > >> > > > >>>> > > > > > > > > >> > > > clear
> > > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special
> meaning. 0
> > > > >> doesn't
> > > > >> > > have
> > > > >> > > > >>>> to
> > > > >> > > > >>>> > mean
> > > > >> > > > >>>> > > > > > instant,
> > > > >> > > > >>>> > > > > > > > it
> > > > >> > > > >>>> > > > > > > > > >> just
> > > > >> > > > >>>> > > > > > > > > >> > > > means
> > > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I
> don't
> > > > >> think 0
> > > > >> > > is
> > > > >> > > > >>>> > intuitive
> > > > >> > > > >>>> > > > for
> > > > >> > > > >>>> > > > > > > > > disabled
> > > > >> > > > >>>> > > > > > > > > >> and
> > > > >> > > > >>>> > > > > > > > > >> > > it
> > > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't
> > > aware
> > > > >> > > > segment.ms=0
> > > > >> > > > >>>> ==
> > > > >> > > > >>>> > > > > > disabled,
> > > > >> > > > >>>> > > > > > > > > but I
> > > > >> > > > >>>> > > > > > > > > >> > > think
> > > > >> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > I think there is an argument
> for
> > > > >> keeping
> > > > >> > > > these
> > > > >> > > > >>>> two
> > > > >> > > > >>>> > > > > > > > configurations
> > > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they are so
> > > > >> similar. I
> > > > >> > > > agree
> > > > >> > > > >>>> that
> > > > >> > > > >>>> > 0
> > > > >> > > > >>>> > > > was an
> > > > >> > > > >>>> > > > > > > > > >> > unfortunate
> > > > >> > > > >>>> > > > > > > > > >> > > > choice.,
> > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > best,
> > > > >> > > > >>>> > > > > > > > > >> > > > Colin
> > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at
> 11:38 AM
> > > > >> Colin
> > > > >> > > > McCabe
> > > > >> > > > >>>> <
> > > > >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > > > >> > > > >>>> > > > > > > > > >> > > wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at
> 17:47,
> > > > >> > xiongqi
> > > > >> > > wu
> > > > >> > > > >>>> wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply
> below.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at
> 5:24
> > > PM
> > > > >> > Colin
> > > > >> > > > >>>> McCabe <
> > > > >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > > >> > > > >>>> > > > > > > > > >> > > > wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit
> > > > >> ambiguous.
> > > > >> > Our
> > > > >> > > > >>>> > compaction
> > > > >> > > > >>>> > > > > > > policies
> > > > >> > > > >>>> > > > > > > > > are
> > > > >> > > > >>>> > > > > > > > > >> > > already
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after
> all. It
> > > > >> seems
> > > > >> > > like
> > > > >> > > > >>>> this
> > > > >> > > > >>>> > > > change is
> > > > >> > > > >>>> > > > > > > > > focused
> > > > >> > > > >>>> > > > > > > > > >> > > around
> > > > >> > > > >>>> > > > > > > > > >> > > > > > adding
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “
> max.compaction.lag.ms
> > > ."
> > > > >> > Perhaps
> > > > >> > > > >>>> the KIP
> > > > >> > > > >>>> > > > title
> > > > >> > > > >>>> > > > > > > should
> > > > >> > > > >>>> > > > > > > > > be
> > > > >> > > > >>>> > > > > > > > > >> > > > something
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum
> > > compaction
> > > > >> lag
> > > > >> > > > time"?
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I
> will
> > > > >> change
> > > > >> > the
> > > > >> > > > >>>> title.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is
> > > forced
> > > > >> to
> > > > >> > > roll
> > > > >> > > > >>>> when
> > > > >> > > > >>>> > > > either "
> > > > >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (
> > > > >> log.roll.ms
> > > > >> > and
> > > > >> > > > >>>> > > > log.roll.hours)
> > > > >> > > > >>>> > > > > > > has
> > > > >> > > > >>>> > > > > > > > > >> > reached.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> > > > max.compaction.lag.ms
> > > > >> is
> > > > >> > > low,
> > > > >> > > > >>>> it
> > > > >> > > > >>>> > seems
> > > > >> > > > >>>> > > > like
> > > > >> > > > >>>> > > > > > > > > segments
> > > > >> > > > >>>> > > > > > > > > >> > will
> > > > >> > > > >>>> > > > > > > > > >> > > > be
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very
> frequently.
> > > This
> > > > >> can
> > > > >> > > be a
> > > > >> > > > >>>> > source of
> > > > >> > > > >>>> > > > > > > problems
> > > > >> > > > >>>> > > > > > > > > in
> > > > >> > > > >>>> > > > > > > > > >> the
> > > > >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating many
> > > > different
> > > > >> > small
> > > > >> > > > log
> > > > >> > > > >>>> > segments
> > > > >> > > > >>>> > > > > > > > consumes
> > > > >> > > > >>>> > > > > > > > > a
> > > > >> > > > >>>> > > > > > > > > >> > huge
> > > > >> > > > >>>> > > > > > > > > >> > > > > > amount of
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources.
> > > > Therefore,
> > > > >> I
> > > > >> > > would
> > > > >> > > > >>>> > suggest
> > > > >> > > > >>>> > > > > > adding a
> > > > >> > > > >>>> > > > > > > > > >> > > broker-level
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration which
> allows
> > > > us
> > > > >> to
> > > > >> > > set
> > > > >> > > > a
> > > > >> > > > >>>> > minimum
> > > > >> > > > >>>> > > > value
> > > > >> > > > >>>> > > > > > > for
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms.
> If
> > > > we
> > > > >> let
> > > > >> > > > >>>> users set
> > > > >> > > > >>>> > it
> > > > >> > > > >>>> > > > on a
> > > > >> > > > >>>> > > > > > > > > >> per-topic
> > > > >> > > > >>>> > > > > > > > > >> > > > basis,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a
> value
> > > of
> > > > >> 1 ms
> > > > >> > > or
> > > > >> > > > >>>> > something,
> > > > >> > > > >>>> > > > and
> > > > >> > > > >>>> > > > > > > > cause
> > > > >> > > > >>>> > > > > > > > > >> > chaos.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this
> applies to
> > > > >> > > > segment.ms
> > > > >> > > > >>>> as
> > > > >> > > > >>>> > well.
> > > > >> > > > >>>> > > > > > Today
> > > > >> > > > >>>> > > > > > > > > users
> > > > >> > > > >>>> > > > > > > > > >> can
> > > > >> > > > >>>> > > > > > > > > >> > > > set "
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very
> low
> > > > >> value,
> > > > >> > and
> > > > >> > > > >>>> cause a
> > > > >> > > > >>>> > > > frequent
> > > > >> > > > >>>> > > > > > > > > >> rolling of
> > > > >> > > > >>>> > > > > > > > > >> > > > active
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an
> > > existing
> > > > >> > > problem
> > > > >> > > > >>>> with
> > > > >> > > > >>>> > > > > > segment.ms.
> > > > >> > > > >>>> > > > > > > > > >> However,
> > > > >> > > > >>>> > > > > > > > > >> > > > that
> > > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we
> shouldn't
> > > > fix
> > > > >> > it.
> > > > >> > > As
> > > > >> > > > >>>> you
> > > > >> > > > >>>> > noted,
> > > > >> > > > >>>> > > > > > there
> > > > >> > > > >>>> > > > > > > > > will
> > > > >> > > > >>>> > > > > > > > > >> be
> > > > >> > > > >>>> > > > > > > > > >> > > more
> > > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these
> topic-level
> > > > >> > retention
> > > > >> > > > >>>> > settings as
> > > > >> > > > >>>> > > > a
> > > > >> > > > >>>> > > > > > > result
> > > > >> > > > >>>> > > > > > > > > of
> > > > >> > > > >>>> > > > > > > > > >> > GDPR.
> > > > >> > > > >>>> > > > > > > > > >> > > > It
> > > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that
> pre-existing
> > > > >> > problems
> > > > >> > > > >>>> will
> > > > >> > > > >>>> > cause
> > > > >> > > > >>>> > > > more
> > > > >> > > > >>>> > > > > > > > > trouble.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> > > > >> > > straightforward
> > > > >> > > > >>>> here --
> > > > >> > > > >>>> > > > add a
> > > > >> > > > >>>> > > > > > > > > >> broker-level
> > > > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that
> > > > overrides
> > > > >> > > > >>>> per-topic
> > > > >> > > > >>>> > > > minimums.
> > > > >> > > > >>>> > > > > > We
> > > > >> > > > >>>> > > > > > > > can
> > > > >> > > > >>>> > > > > > > > > >> also
> > > > >> > > > >>>> > > > > > > > > >> > > fail
> > > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error
> message
> > > > when
> > > > >> > > someone
> > > > >> > > > >>>> > attempts
> > > > >> > > > >>>> > > > to
> > > > >> > > > >>>> > > > > > set
> > > > >> > > > >>>> > > > > > > an
> > > > >> > > > >>>> > > > > > > > > >> > invalid
> > > > >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the
> minimum
> > > of "
> > > > >> > > > >>>> > > > max.compaction.lag.ms"
> > > > >> > > > >>>> > > > > > > > should
> > > > >> > > > >>>> > > > > > > > > >> be
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of
> "
> > > > >> > segment.ms
> > > > >> > > ".
> > > > >> > > > >>>> Since
> > > > >> > > > >>>> > > > today the
> > > > >> > > > >>>> > > > > > > > > >> minimum of
> > > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "
> > > max.compaction.lag.ms"
> > > > >> also
> > > > >> > > > >>>> starts
> > > > >> > > > >>>> > with
> > > > >> > > > >>>> > > > 1. "0"
> > > > >> > > > >>>> > > > > > > > means
> > > > >> > > > >>>> > > > > > > > > >> > > > disable. I
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable,
> but
> > > it
> > > > is
> > > > >> > hard
> > > > >> > > > to
> > > > >> > > > >>>> > define
> > > > >> > > > >>>> > > > the
> > > > >> > > > >>>> > > > > > > > meaning
> > > > >> > > > >>>> > > > > > > > > >> of 0
> > > > >> > > > >>>> > > > > > > > > >> > > > because
> > > > >> > > > >>>> > > > > > > > > >> > > > > > we
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the
> active
> > > > >> segment
> > > > >> > > > >>>> > immediately.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We
> should
> > > > >> make
> > > > >> > 0 =
> > > > >> > > > >>>> > disable, to
> > > > >> > > > >>>> > > > be
> > > > >> > > > >>>> > > > > > > > > >> consistent
> > > > >> > > > >>>> > > > > > > > > >> > > with
> > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > best,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > Colin
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an
> > > alternative
> > > > >> > > > >>>> configuration
> > > > >> > > > >>>> > is to
> > > > >> > > > >>>> > > > use
> > > > >> > > > >>>> > > > > > -1
> > > > >> > > > >>>> > > > > > > > as
> > > > >> > > > >>>> > > > > > > > > >> > > > "disabled"
> > > > >> > > > >>>> > > > > > > > > >> > > > > > and
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate
> > > compaction".
> > > > >> > > Because
> > > > >> > > > >>>> > compaction
> > > > >> > > > >>>> > > > lag
> > > > >> > > > >>>> > > > > > is
> > > > >> > > > >>>> > > > > > > > > still
> > > > >> > > > >>>> > > > > > > > > >> > > > determined
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> > > > min.compaction.lag
> > > > >> and
> > > > >> > > how
> > > > >> > > > >>>> long
> > > > >> > > > >>>> > to
> > > > >> > > > >>>> > > > roll
> > > > >> > > > >>>> > > > > > an
> > > > >> > > > >>>> > > > > > > > > active
> > > > >> > > > >>>> > > > > > > > > >> > > > segment,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > the
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for
> > > compaction
> > > > is
> > > > >> > > > >>>> undetermined
> > > > >> > > > >>>> > if
> > > > >> > > > >>>> > > > we
> > > > >> > > > >>>> > > > > > use
> > > > >> > > > >>>> > > > > > > > "0".
> > > > >> > > > >>>> > > > > > > > > >> On
> > > > >> > > > >>>> > > > > > > > > >> > the
> > > > >> > > > >>>> > > > > > > > > >> > > > other
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can
> already set
> > > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > > >> > > > >>>> > > > > > > > to
> > > > >> > > > >>>> > > > > > > > > >> > achieve
> > > > >> > > > >>>> > > > > > > > > >> > > > the
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > same
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we
> choose
> > > > "0"
> > > > >> as
> > > > >> > > > >>>> "disabled".
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to
> be
> > > the
> > > > >> > invalid
> > > > >> > > > >>>> setting.
> > > > >> > > > >>>> > > > > > Treating
> > > > >> > > > >>>> > > > > > > 0
> > > > >> > > > >>>> > > > > > > > > >> > > differently
> > > > >> > > > >>>> > > > > > > > > >> > > > > > than
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous
> > > > >> comment, I
> > > > >> > > am
> > > > >> > > > >>>> not
> > > > >> > > > >>>> > > > strongly
> > > > >> > > > >>>> > > > > > > > against,
> > > > >> > > > >>>> > > > > > > > > >> but
> > > > >> > > > >>>> > > > > > > > > >> > 0
> > > > >> > > > >>>> > > > > > > > > >> > > is
> > > > >> > > > >>>> > > > > > > > > >> > > > > > not a
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in
> my
> > > > >> option.
> > > > >> > So
> > > > >> > > I
> > > > >> > > > >>>> use
> > > > >> > > > >>>> > "0" as
> > > > >> > > > >>>> > > > > > > disabled
> > > > >> > > > >>>> > > > > > > > > >> state.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > best,
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018,
> at
> > > > 15:04,
> > > > >> > > > xiongqi
> > > > >> > > > >>>> wu
> > > > >> > > > >>>> > wrote:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this
> KIP.
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>>
> > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > >> > > > >>>> > > >
> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > >> > > > >>>> > > > > > >
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>>
> > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > >> > > > >>>> > > >
> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> >>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > >> > > > >>>> >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>
> > > > >> > > > >>>> > > > > > > > <
> > > > >> > > > >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >
> > > > >> > > > >>>> > > > > > <
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > > > >>>> > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >> > >>>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > %3A+Time-based+log+compaction+policy
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > >> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > > >> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > <
> > > > >> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >> > > > >>>> > > > > > > > > >> <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >> > > > >>>> > > > > > > > <
> https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> > > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > > >> > > > >>>> > > > > > > > > >> > > >
> > > > >> > > > >>>> > > > > > > > > >> > >
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > --
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > Brett Rann
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC
> 3000
> > > > >> > > Australia
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > > > >> > > > >>>> > > > > > > > > >> >
> > > > >> > > > >>>> > > > > > > > > >>
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > --
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > Brett Rann
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC
> 3000
> > > > >> Australia
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > >> > > > >>>> > > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > --
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Brett Rann
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Zendesk International Ltd
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> > > > >> Australia
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > >> > > > >>>> > > > > > > > >
> > > > >> > > > >>>> > > > > > > >
> > > > >> > > > >>>> > > > > > >
> > > > >> > > > >>>> > > > > >
> > > > >> > > > >>>> > > >
> > > > >> > > > >>>> >
> > > > >> > > > >>>>
> > > > >> > > > >>>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by Colin McCabe <cm...@apache.org>.
Hi Xiongqi,
Thinking about this a little bit more, it seems like we don't have any guarantees just by looking at the timestamp of the first message in a log segment. Similarly, we don't have any guarantees just by looking at the maxTimestamp of the previous log segment. Old data could appear anywhere-- you could put data that was years old in the middle of a segment from 2018.
However, if log.message.timestamp.difference.max.ms is set, then we can make some actual guarantees that old data gets purged-- which is what the GDPR requires, of course.
Overall, maybe we can make KIP-354 simpler by just always looking at the timestamp of the first log message. I don't think looking at the maxTimestamp of the previous segment is any more accurate. Aside from that, looks good, since we can get what we need with the combination of this and log.message.timestamp.difference.max.ms.
best,
Colin
On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote:
> Thanks for binding and non-binding votes.
> Can I get one more binding vote?
>
> Thanks in advance!
>
> Xiongqi (Wesley) Wu
>
>
> On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <ma...@frmr.me> wrote:
>
> > I'm a +1 (non-binding) — This looks like it would have saved us a lot of
> > pain in an issue we had to debug recently. I can't go into details, but
> > figuring out how to achieve this effect gave me quite a headache. :)
> >
> > On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xi...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Can I have one more vote on this KIP?
> > > Any comment is appreciated.
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
> > >
> > >
> > > Xiongqi (Wesley) Wu
> > >
> > >
> > > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xi...@gmail.com> wrote:
> > >
> > > > Thanks Dong.
> > > > I have updated the KIP.
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > > >
> > > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com> wrote:
> > > >
> > > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> > > >>
> > > >> One minor comment: it may be a bit better to clarify in the public
> > > >> interface section that the value of the newly added metric is
> > determined
> > > >> based by applying that formula across all compactable segments. For
> > > >> example:
> > > >>
> > > >> The maximum value of Math.max(now -
> > > >> earliest_timestamp_in_ms_of_uncompacted_segment -
> > max.compaction.lag.ms
> > > ,
> > > >> 0)/1000 across all compactable partitions, where the
> > > >> max.compaction.lag.ms
> > > >> can be overridden on per-topic basis.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com>
> > wrote:
> > > >>
> > > >> > Thanks Joel.
> > > >> > Tracking the delay at second granularity makes sense
> > > >> > I have updated KIP.
> > > >> >
> > > >> > Xiongqi (Wesley) Wu
> > > >> >
> > > >> >
> > > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > +1 with one suggestion on the proposed metric. You should probably
> > > >> > include
> > > >> > > the unit. So for e.g., max-compaction-delay-secs.
> > > >> > >
> > > >> > > Joel
> > > >> > >
> > > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xi...@gmail.com>
> > > >> wrote:
> > > >> > >
> > > >> > > > bump
> > > >> > > > Xiongqi (Wesley) Wu
> > > >> > > >
> > > >> > > >
> > > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xiongqiwu@gmail.com
> > >
> > > >> > wrote:
> > > >> > > >
> > > >> > > > >
> > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for
> > > >> feedback.
> > > >> > > > > Can I have more feedback or VOTE on this KIP?
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Xiongqi (Wesley) Wu
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> > > xiongqiwu@gmail.com>
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > >> Any other votes or comments?
> > > >> > > > >>
> > > >> > > > >> Xiongqi (Wesley) Wu
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> > > xiongqiwu@gmail.com
> > > >> >
> > > >> > > > wrote:
> > > >> > > > >>
> > > >> > > > >>> Yes, more votes and code review.
> > > >> > > > >>>
> > > >> > > > >>> Xiongqi (Wesley) Wu
> > > >> > > > >>>
> > > >> > > > >>>
> > > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > > >> > > <brann@zendesk.com.invalid
> > > >> > > > >
> > > >> > > > >>> wrote:
> > > >> > > > >>>
> > > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > > >> > > > >>>>
> > > >> > > > >>>> Where do we go from here? More votes?
> > > >> > > > >>>>
> > > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> > > >> cmccabe@apache.org>
> > > >> > > > >>>> wrote:
> > > >> > > > >>>>
> > > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > > >> > > > >>>> > > Thank you for comments. I will use '0' for now.
> > > >> > > > >>>> > >
> > > >> > > > >>>> > > If we create topics through admin client in the future,
> > > we
> > > >> > might
> > > >> > > > >>>> perform
> > > >> > > > >>>> > > some useful checks. (but the assumption is all brokers
> > in
> > > >> the
> > > >> > > same
> > > >> > > > >>>> > cluster
> > > >> > > > >>>> > > have the same default configurations value,
> > otherwise,it
> > > >> might
> > > >> > > > >>>> still be
> > > >> > > > >>>> > > tricky to do such cross validation check.)
> > > >> > > > >>>> >
> > > >> > > > >>>> > This isn't something that we might do in the future--
> > this
> > > is
> > > >> > > > >>>> something we
> > > >> > > > >>>> > are doing now. We already have Create Topic policies
> > which
> > > >> are
> > > >> > > > >>>> enforced by
> > > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details. This
> > is
> > > >> one
> > > >> > of
> > > >> > > > the
> > > >> > > > >>>> > motivations for getting rid of direct ZK access-- making
> > > sure
> > > >> > that
> > > >> > > > >>>> these
> > > >> > > > >>>> > policies are applied.
> > > >> > > > >>>> >
> > > >> > > > >>>> > I agree that having different configurations on different
> > > >> > brokers
> > > >> > > > can
> > > >> > > > >>>> be
> > > >> > > > >>>> > confusing and frustrating . That's why more
> > configurations
> > > >> are
> > > >> > > being
> > > >> > > > >>>> made
> > > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are stored
> > > >> > centrally
> > > >> > > > in
> > > >> > > > >>>> ZK,
> > > >> > > > >>>> > so they are the same on all brokers (modulo propagation
> > > >> delays).
> > > >> > > In
> > > >> > > > >>>> any
> > > >> > > > >>>> > case, this is a general issue, not specific to "create
> > > >> topics".
> > > >> > > > >>>> >
> > > >> > > > >>>> > cheers,
> > > >> > > > >>>> > Colin
> > > >> > > > >>>> >
> > > >> > > > >>>> >
> > > >> > > > >>>> > >
> > > >> > > > >>>> > >
> > > >> > > > >>>> > > Xiongqi (Wesley) Wu
> > > >> > > > >>>> > >
> > > >> > > > >>>> > >
> > > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> > > >> > > cmccabe@apache.org
> > > >> > > > >
> > > >> > > > >>>> > wrote:
> > > >> > > > >>>> > >
> > > >> > > > >>>> > > > I don't have a strong opinion. But I think we should
> > > >> > probably
> > > >> > > be
> > > >> > > > >>>> > > > consistent with how segment.ms works, and just use
> > 0.
> > > >> > > > >>>> > > >
> > > >> > > > >>>> > > > best,
> > > >> > > > >>>> > > > Colin
> > > >> > > > >>>> > > >
> > > >> > > > >>>> > > >
> > > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> > > >> > > > >>>> > > > > OK thanks for that clarification. I see why you're
> > > >> > > > uncomfortable
> > > >> > > > >>>> > with 0
> > > >> > > > >>>> > > > now.
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > > I'm not really fussed. I just prefer consistency in
> > > >> > > > >>>> configuration
> > > >> > > > >>>> > > > options.
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > > Personally I lean towards treating 0 and 1
> > similarly
> > > in
> > > >> > that
> > > >> > > > >>>> > scenario,
> > > >> > > > >>>> > > > > because it favours the person thinking about
> > setting
> > > >> the
> > > >> > > > >>>> > configurations,
> > > >> > > > >>>> > > > > and a person doesn't care about a 1ms edge case
> > > >> especially
> > > >> > > > when
> > > >> > > > >>>> the
> > > >> > > > >>>> > > > context
> > > >> > > > >>>> > > > > is the true minimum is tied to the log cleaner
> > > cadence.
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because there is
> > > some
> > > >> > > > >>>> uniquness in
> > > >> > > > >>>> > > > > segment.ms not being able to be set to 0, reduces
> > > >> > > > configuration
> > > >> > > > >>>> > > > consistency
> > > >> > > > >>>> > > > > in favour of capturing a MS gap in an edge case
> > that
> > > >> > nobody
> > > >> > > > >>>> would
> > > >> > > > >>>> > ever
> > > >> > > > >>>> > > > > notice. For someone to understand why everywhere
> > else
> > > >> -1
> > > >> > is
> > > >> > > > >>>> used to
> > > >> > > > >>>> > > > > disable, but here 0 is used, they would need to
> > learn
> > > >> > about
> > > >> > > > >>>> > segment.ms
> > > >> > > > >>>> > > > > having a 1ms minimum and then after learning would
> > > >> think
> > > >> > > "who
> > > >> > > > >>>> cares
> > > >> > > > >>>> > about
> > > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > > my 2c anyway. Will again defer to majority. Curious
> > > >> which
> > > >> > > way
> > > >> > > > >>>> Colin
> > > >> > > > >>>> > falls
> > > >> > > > >>>> > > > > now.
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > > Don't want to spend more time on this though, It's
> > > well
> > > >> > into
> > > >> > > > >>>> > > > bikeshedding
> > > >> > > > >>>> > > > > territory now. :)
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > > >> > > > xiongqiwu@gmail.com>
> > > >> > > > >>>> > wrote:
> > > >> > > > >>>> > > > >
> > > >> > > > >>>> > > > > > I want to honor the minimum value of segment.ms
> > > >> (which
> > > >> > is
> > > >> > > > >>>> 1ms) to
> > > >> > > > >>>> > > > force
> > > >> > > > >>>> > > > > > roll an active segment.
> > > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any value >
> > > 0,
> > > >> the
> > > >> > > > >>>> minimum of
> > > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be
> > used
> > > to
> > > >> > seal
> > > >> > > > an
> > > >> > > > >>>> > active
> > > >> > > > >>>> > > > > > segment.
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the
> > current
> > > >> > > > >>>> implementation
> > > >> > > > >>>> > will
> > > >> > > > >>>> > > > > > treat it as disabled.
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > > It is a little bit weird to treat
> > > >> max.compaction.lag=0
> > > >> > the
> > > >> > > > >>>> same as
> > > >> > > > >>>> > > > > > max.compaction.lag=1.
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > > There might be a reason why we set the minimum of
> > > >> > > > segment.ms
> > > >> > > > >>>> to 1,
> > > >> > > > >>>> > > > and I
> > > >> > > > >>>> > > > > > don't want to break this assumption.
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > > >> > > > >>>> > <br...@zendesk.com.invalid>
> > > >> > > > >>>> > > > > > wrote:
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > > > > > You're rolling a new segment if the condition
> > is
> > > >> met
> > > >> > > > right?
> > > >> > > > >>>> So
> > > >> > > > >>>> > I'm
> > > >> > > > >>>> > > > > > > struggling to understand the relevance of
> > > >> segment.ms
> > > >> > > > here.
> > > >> > > > >>>> > Maybe an
> > > >> > > > >>>> > > > > > > example
> > > >> > > > >>>> > > > > > > would help my understanding:
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > > When a duplicate message comes in, after 1ms
> > the
> > > >> topic
> > > >> > > > >>>> should be
> > > >> > > > >>>> > > > eligible
> > > >> > > > >>>> > > > > > > for compaction when the log compaction thread
> > > gets
> > > >> > > around
> > > >> > > > to
> > > >> > > > >>>> > > > evaluating
> > > >> > > > >>>> > > > > > the
> > > >> > > > >>>> > > > > > > topic.
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > > if we have
> > > >> > > > >>>> > > > > > > segment.ms=9999999
> > > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > > When a duplicate message comes in, after 0ms
> > the
> > > >> topic
> > > >> > > > >>>> should be
> > > >> > > > >>>> > > > eligible
> > > >> > > > >>>> > > > > > > for compaction when the log compaction thread
> > > gets
> > > >> > > around
> > > >> > > > to
> > > >> > > > >>>> > > > evaluating
> > > >> > > > >>>> > > > > > the
> > > >> > > > >>>> > > > > > > topic.
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > > In both of those cases the change would mean a
> > > new
> > > >> > > segment
> > > >> > > > >>>> is
> > > >> > > > >>>> > rolled
> > > >> > > > >>>> > > > so
> > > >> > > > >>>> > > > > > the
> > > >> > > > >>>> > > > > > > new message would be part of the compaction
> > > task. 0
> > > >> > and
> > > >> > > 1
> > > >> > > > >>>> are
> > > >> > > > >>>> > > > practically
> > > >> > > > >>>> > > > > > > the same meaning since neither is providing an
> > > >> actual
> > > >> > > > >>>> guarantee
> > > >> > > > >>>> > at
> > > >> > > > >>>> > > > such
> > > >> > > > >>>> > > > > > low
> > > >> > > > >>>> > > > > > > MS settings, but effectively tying it to both
> > the
> > > >> > > > frequency
> > > >> > > > >>>> of
> > > >> > > > >>>> > the
> > > >> > > > >>>> > > > log
> > > >> > > > >>>> > > > > > > cleaner running and the priority of the given
> > > topic
> > > >> > > being
> > > >> > > > >>>> the
> > > >> > > > >>>> > highest
> > > >> > > > >>>> > > > > > > priority of all topics that are evaluated for
> > > >> cleaning
> > > >> > > on
> > > >> > > > >>>> the
> > > >> > > > >>>> > next
> > > >> > > > >>>> > > > cycle.
> > > >> > > > >>>> > > > > > > You've captured that nuance with careful
> > > "skipped"
> > > >> > > wording
> > > >> > > > >>>> in
> > > >> > > > >>>> > the KIP
> > > >> > > > >>>> > > > > > > here "controls
> > > >> > > > >>>> > > > > > > the max time interval a message/segment can be
> > > >> skipped
> > > >> > > for
> > > >> > > > >>>> log
> > > >> > > > >>>> > > > > > compaction".
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > > How is 0 different to 1, practically? And how
> > is
> > > it
> > > >> > > > >>>> relating to
> > > >> > > > >>>> > > > > > segment.ms
> > > >> > > > >>>> > > > > > > ?
> > > >> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean "use
> > > >> > > > segment.ms
> > > >> > > > >>>> > > > instead?" as
> > > >> > > > >>>> > > > > > a
> > > >> > > > >>>> > > > > > > kind of third option?
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
> > > >> > > > >>>> xiongqiwu@gmail.com>
> > > >> > > > >>>> > > > wrote:
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > > > > To make it clear,
> > > >> > > > >>>> > > > > > > > I don't against using -1 as disabled, but we
> > > >> need to
> > > >> > > > come
> > > >> > > > >>>> up
> > > >> > > > >>>> > with
> > > >> > > > >>>> > > > the
> > > >> > > > >>>> > > > > > > > meaning of "0".
> > > >> > > > >>>> > > > > > > > If "0" means immediate compaction, but the
> > > actual
> > > >> > > > >>>> compaction
> > > >> > > > >>>> > lag
> > > >> > > > >>>> > > > will
> > > >> > > > >>>> > > > > > be
> > > >> > > > >>>> > > > > > > > segment.ms.
> > > >> > > > >>>> > > > > > > > It has longer lag than setting the value to
> > be
> > > >> half
> > > >> > of
> > > >> > > > >>>> > segment.ms.
> > > >> > > > >>>> > > > > > > > We cannot provide "0" as max compaction lag.
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > > Here are two options.
> > > >> > > > >>>> > > > > > > > Option 1:
> > > >> > > > >>>> > > > > > > > Keep 0 as disabled
> > > >> > > > >>>> > > > > > > > Option 2:
> > > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag =
> > > >> segment.ms),
> > > >> > > and
> > > >> > > > >>>> > others.
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> > > >> > > > >>>> > > > <br...@zendesk.com.invalid>
> > > >> > > > >>>> > > > > > > > wrote:
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > > > -1 is consistent as "special" with these
> > > >> settings
> > > >> > > for
> > > >> > > > >>>> > example:
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > log.retention.bytes
> > > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > >> > > > >>>> > > > > > > > > queued.max.request.bytes
> > > >> > > > >>>> > > > > > > > > retention.bytes
> > > >> > > > >>>> > > > > > > > > retention.ms
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > and acks.
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS
> > defaults,
> > > >> max
> > > >> > > > (acks),
> > > >> > > > >>>> > etc. I
> > > >> > > > >>>> > > > don't
> > > >> > > > >>>> > > > > > > see
> > > >> > > > >>>> > > > > > > > > much convention of 0 meaning those things.
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > There are some NULLs but it seems convetion
> > > >> there
> > > >> > is
> > > >> > > > >>>> NULL is
> > > >> > > > >>>> > used
> > > >> > > > >>>> > > > > > where
> > > >> > > > >>>> > > > > > > > > there's another setting in the hierarchy
> > that
> > > >> > would
> > > >> > > be
> > > >> > > > >>>> used
> > > >> > > > >>>> > > > instead.
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann
> > <
> > > >> > > > >>>> > brann@zendesk.com>
> > > >> > > > >>>> > > > > > wrote:
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0, then
> > > we're
> > > >> not
> > > >> > > > being
> > > >> > > > >>>> > > > consistent
> > > >> > > > >>>> > > > > > > > > > by using 0 for this new setting? I throw
> > > out
> > > >> -1
> > > >> > > for
> > > >> > > > >>>> > > > consideration
> > > >> > > > >>>> > > > > > > > > > again :)
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi
> > wu
> > > <
> > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > >> > > > >>>> > > > > > > > wrote:
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is
> > > merged.
> > > >> > > > >>>> > > > > > > > > >>
> > > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "
> > > >> segment.ms"
> > > >> > > to
> > > >> > > > >>>> 1, we
> > > >> > > > >>>> > > > cannot
> > > >> > > > >>>> > > > > > set
> > > >> > > > >>>> > > > > > > "
> > > >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > >> > > > >>>> > > > > > > > > >>
> > > >> > > > >>>> > > > > > > > > >> I also updated the title of this KIP.
> > > >> > > > >>>> > > > > > > > > >>
> > > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > >> > > > >>>> > > > > > > > > >>
> > > >> > > > >>>> > > > > > > > > >>
> > > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett
> > Rann
> > > >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> wrote:
> > > >> > > > >>>> > > > > > > > > >>
> > > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm
> > > in
> > > >> the
> > > >> > > > >>>> minority.
> > > >> > > > >>>> > :)
> > > >> > > > >>>> > > > Can
> > > >> > > > >>>> > > > > > we
> > > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning
> > > >> > disabled
> > > >> > > > >>>> here:
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > >> > > > >>>> > > > > > <
> > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > >> > > > >>>> > > > > > <
> > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs
> > >>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > >> > > > >>>> > > > > > <
> > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > >> > > > >>>> > > > > > <
> > > >> https://kafka.apache.org/documentation/#brokerconfigs
> > > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs
> > >>>>
> > > ?
> > > >> > > > >>>> > > > > > > > > >> > And while there it would be good if
> > > >> > segment.ms
> > > >> > > > is
> > > >> > > > >>>> > > > documented
> > > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some
> > > >> > hierarchy
> > > >> > > > of
> > > >> > > > >>>> > configs
> > > >> > > > >>>> > > > for
> > > >> > > > >>>> > > > > > > that
> > > >> > > > >>>> > > > > > > > > too
> > > >> > > > >>>> > > > > > > > > >> > if its not set and null for others
> > means
> > > >> > > > disabled!)
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi
> > > wu
> > > >> <
> > > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > > >> > > > >>>> > > > > > > > > wrote:
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate
> > > >> > compaction,
> > > >> > > > the
> > > >> > > > >>>> > > > compaction
> > > >> > > > >>>> > > > > > lag
> > > >> > > > >>>> > > > > > > > is
> > > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in worst
> > > >> case. If
> > > >> > > > >>>> segment.ms
> > > >> > > > >>>> > is
> > > >> > > > >>>> > > > 24
> > > >> > > > >>>> > > > > > > > hours,
> > > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker
> > > >> > guarantee
> > > >> > > > than
> > > >> > > > >>>> > setting
> > > >> > > > >>>> > > > any
> > > >> > > > >>>> > > > > > > > value
> > > >> > > > >>>> > > > > > > > > >> less
> > > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the definition of
> > > "max
> > > >> > > > >>>> compaction
> > > >> > > > >>>> > lag",
> > > >> > > > >>>> > > > we
> > > >> > > > >>>> > > > > > > > cannot
> > > >> > > > >>>> > > > > > > > > >> have
> > > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate
> > > >> "disable".
> > > >> > > > >>>> > > > > > > > > >> > >
> > > >> > > > >>>> > > > > > > > > >> > >
> > > >> > > > >>>> > > > > > > > > >> > >
> > > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > >> > > > >>>> > > > > > > > > >> > >
> > > >> > > > >>>> > > > > > > > > >> > >
> > > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin
> > > >> > McCabe <
> > > >> > > > >>>> > > > > > cmccabe@apache.org
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> wrote:
> > > >> > > > >>>> > > > > > > > > >> > >
> > > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11,
> > Brett
> > > >> Rann
> > > >> > > > >>>> wrote:
> > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should
> > > >> make
> > > >> > 0 =
> > > >> > > > >>>> > disable, to
> > > >> > > > >>>> > > > be
> > > >> > > > >>>> > > > > > > > > >> consistent
> > > >> > > > >>>> > > > > > > > > >> > > with
> > > >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > > >> > > > >>>> > > > > > > > > >> > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable
> > > and
> > > >> > when
> > > >> > > > >>>> seeing
> > > >> > > > >>>> > it
> > > >> > > > >>>> > > > in a
> > > >> > > > >>>> > > > > > > > config
> > > >> > > > >>>> > > > > > > > > >> it's
> > > >> > > > >>>> > > > > > > > > >> > > > clear
> > > >> > > > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0
> > > >> doesn't
> > > >> > > have
> > > >> > > > >>>> to
> > > >> > > > >>>> > mean
> > > >> > > > >>>> > > > > > instant,
> > > >> > > > >>>> > > > > > > > it
> > > >> > > > >>>> > > > > > > > > >> just
> > > >> > > > >>>> > > > > > > > > >> > > > means
> > > >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't
> > > >> think 0
> > > >> > > is
> > > >> > > > >>>> > intuitive
> > > >> > > > >>>> > > > for
> > > >> > > > >>>> > > > > > > > > disabled
> > > >> > > > >>>> > > > > > > > > >> and
> > > >> > > > >>>> > > > > > > > > >> > > it
> > > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't
> > aware
> > > >> > > > segment.ms=0
> > > >> > > > >>>> ==
> > > >> > > > >>>> > > > > > disabled,
> > > >> > > > >>>> > > > > > > > > but I
> > > >> > > > >>>> > > > > > > > > >> > > think
> > > >> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> > > >> > > > >>>> > > > > > > > > >> > > >
> > > >> > > > >>>> > > > > > > > > >> > > > I think there is an argument for
> > > >> keeping
> > > >> > > > these
> > > >> > > > >>>> two
> > > >> > > > >>>> > > > > > > > configurations
> > > >> > > > >>>> > > > > > > > > >> > > > consistent, since they are so
> > > >> similar. I
> > > >> > > > agree
> > > >> > > > >>>> that
> > > >> > > > >>>> > 0
> > > >> > > > >>>> > > > was an
> > > >> > > > >>>> > > > > > > > > >> > unfortunate
> > > >> > > > >>>> > > > > > > > > >> > > > choice.,
> > > >> > > > >>>> > > > > > > > > >> > > >
> > > >> > > > >>>> > > > > > > > > >> > > > best,
> > > >> > > > >>>> > > > > > > > > >> > > > Colin
> > > >> > > > >>>> > > > > > > > > >> > > >
> > > >> > > > >>>> > > > > > > > > >> > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM
> > > >> Colin
> > > >> > > > McCabe
> > > >> > > > >>>> <
> > > >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > > >> > > > >>>> > > > > > > > > >> > > wrote:
> > > >> > > > >>>> > > > > > > > > >> > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47,
> > > >> > xiongqi
> > > >> > > wu
> > > >> > > > >>>> wrote:
> > > >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> > > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply below.
> > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24
> > PM
> > > >> > Colin
> > > >> > > > >>>> McCabe <
> > > >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > >> > > > >>>> > > > > > > > > >> > > > wrote:
> > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit
> > > >> ambiguous.
> > > >> > Our
> > > >> > > > >>>> > compaction
> > > >> > > > >>>> > > > > > > policies
> > > >> > > > >>>> > > > > > > > > are
> > > >> > > > >>>> > > > > > > > > >> > > already
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It
> > > >> seems
> > > >> > > like
> > > >> > > > >>>> this
> > > >> > > > >>>> > > > change is
> > > >> > > > >>>> > > > > > > > > focused
> > > >> > > > >>>> > > > > > > > > >> > > around
> > > >> > > > >>>> > > > > > > > > >> > > > > > adding
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms
> > ."
> > > >> > Perhaps
> > > >> > > > >>>> the KIP
> > > >> > > > >>>> > > > title
> > > >> > > > >>>> > > > > > > should
> > > >> > > > >>>> > > > > > > > > be
> > > >> > > > >>>> > > > > > > > > >> > > > something
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum
> > compaction
> > > >> lag
> > > >> > > > time"?
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will
> > > >> change
> > > >> > the
> > > >> > > > >>>> title.
> > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is
> > forced
> > > >> to
> > > >> > > roll
> > > >> > > > >>>> when
> > > >> > > > >>>> > > > either "
> > > >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (
> > > >> log.roll.ms
> > > >> > and
> > > >> > > > >>>> > > > log.roll.hours)
> > > >> > > > >>>> > > > > > > has
> > > >> > > > >>>> > > > > > > > > >> > reached.
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> > > max.compaction.lag.ms
> > > >> is
> > > >> > > low,
> > > >> > > > >>>> it
> > > >> > > > >>>> > seems
> > > >> > > > >>>> > > > like
> > > >> > > > >>>> > > > > > > > > segments
> > > >> > > > >>>> > > > > > > > > >> > will
> > > >> > > > >>>> > > > > > > > > >> > > > be
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very frequently.
> > This
> > > >> can
> > > >> > > be a
> > > >> > > > >>>> > source of
> > > >> > > > >>>> > > > > > > problems
> > > >> > > > >>>> > > > > > > > > in
> > > >> > > > >>>> > > > > > > > > >> the
> > > >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating many
> > > different
> > > >> > small
> > > >> > > > log
> > > >> > > > >>>> > segments
> > > >> > > > >>>> > > > > > > > consumes
> > > >> > > > >>>> > > > > > > > > a
> > > >> > > > >>>> > > > > > > > > >> > huge
> > > >> > > > >>>> > > > > > > > > >> > > > > > amount of
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources.
> > > Therefore,
> > > >> I
> > > >> > > would
> > > >> > > > >>>> > suggest
> > > >> > > > >>>> > > > > > adding a
> > > >> > > > >>>> > > > > > > > > >> > > broker-level
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration which allows
> > > us
> > > >> to
> > > >> > > set
> > > >> > > > a
> > > >> > > > >>>> > minimum
> > > >> > > > >>>> > > > value
> > > >> > > > >>>> > > > > > > for
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If
> > > we
> > > >> let
> > > >> > > > >>>> users set
> > > >> > > > >>>> > it
> > > >> > > > >>>> > > > on a
> > > >> > > > >>>> > > > > > > > > >> per-topic
> > > >> > > > >>>> > > > > > > > > >> > > > basis,
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a value
> > of
> > > >> 1 ms
> > > >> > > or
> > > >> > > > >>>> > something,
> > > >> > > > >>>> > > > and
> > > >> > > > >>>> > > > > > > > cause
> > > >> > > > >>>> > > > > > > > > >> > chaos.
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this applies to
> > > >> > > > segment.ms
> > > >> > > > >>>> as
> > > >> > > > >>>> > well.
> > > >> > > > >>>> > > > > > Today
> > > >> > > > >>>> > > > > > > > > users
> > > >> > > > >>>> > > > > > > > > >> can
> > > >> > > > >>>> > > > > > > > > >> > > > set "
> > > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low
> > > >> value,
> > > >> > and
> > > >> > > > >>>> cause a
> > > >> > > > >>>> > > > frequent
> > > >> > > > >>>> > > > > > > > > >> rolling of
> > > >> > > > >>>> > > > > > > > > >> > > > active
> > > >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an
> > existing
> > > >> > > problem
> > > >> > > > >>>> with
> > > >> > > > >>>> > > > > > segment.ms.
> > > >> > > > >>>> > > > > > > > > >> However,
> > > >> > > > >>>> > > > > > > > > >> > > > that
> > > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't
> > > fix
> > > >> > it.
> > > >> > > As
> > > >> > > > >>>> you
> > > >> > > > >>>> > noted,
> > > >> > > > >>>> > > > > > there
> > > >> > > > >>>> > > > > > > > > will
> > > >> > > > >>>> > > > > > > > > >> be
> > > >> > > > >>>> > > > > > > > > >> > > more
> > > >> > > > >>>> > > > > > > > > >> > > > > > interest in these topic-level
> > > >> > retention
> > > >> > > > >>>> > settings as
> > > >> > > > >>>> > > > a
> > > >> > > > >>>> > > > > > > result
> > > >> > > > >>>> > > > > > > > > of
> > > >> > > > >>>> > > > > > > > > >> > GDPR.
> > > >> > > > >>>> > > > > > > > > >> > > > It
> > > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing
> > > >> > problems
> > > >> > > > >>>> will
> > > >> > > > >>>> > cause
> > > >> > > > >>>> > > > more
> > > >> > > > >>>> > > > > > > > > trouble.
> > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> > > >> > > straightforward
> > > >> > > > >>>> here --
> > > >> > > > >>>> > > > add a
> > > >> > > > >>>> > > > > > > > > >> broker-level
> > > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that
> > > overrides
> > > >> > > > >>>> per-topic
> > > >> > > > >>>> > > > minimums.
> > > >> > > > >>>> > > > > > We
> > > >> > > > >>>> > > > > > > > can
> > > >> > > > >>>> > > > > > > > > >> also
> > > >> > > > >>>> > > > > > > > > >> > > fail
> > > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error message
> > > when
> > > >> > > someone
> > > >> > > > >>>> > attempts
> > > >> > > > >>>> > > > to
> > > >> > > > >>>> > > > > > set
> > > >> > > > >>>> > > > > > > an
> > > >> > > > >>>> > > > > > > > > >> > invalid
> > > >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the minimum
> > of "
> > > >> > > > >>>> > > > max.compaction.lag.ms"
> > > >> > > > >>>> > > > > > > > should
> > > >> > > > >>>> > > > > > > > > >> be
> > > >> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of "
> > > >> > segment.ms
> > > >> > > ".
> > > >> > > > >>>> Since
> > > >> > > > >>>> > > > today the
> > > >> > > > >>>> > > > > > > > > >> minimum of
> > > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "
> > max.compaction.lag.ms"
> > > >> also
> > > >> > > > >>>> starts
> > > >> > > > >>>> > with
> > > >> > > > >>>> > > > 1. "0"
> > > >> > > > >>>> > > > > > > > means
> > > >> > > > >>>> > > > > > > > > >> > > > disable. I
> > > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but
> > it
> > > is
> > > >> > hard
> > > >> > > > to
> > > >> > > > >>>> > define
> > > >> > > > >>>> > > > the
> > > >> > > > >>>> > > > > > > > meaning
> > > >> > > > >>>> > > > > > > > > >> of 0
> > > >> > > > >>>> > > > > > > > > >> > > > because
> > > >> > > > >>>> > > > > > > > > >> > > > > > we
> > > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the active
> > > >> segment
> > > >> > > > >>>> > immediately.
> > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should
> > > >> make
> > > >> > 0 =
> > > >> > > > >>>> > disable, to
> > > >> > > > >>>> > > > be
> > > >> > > > >>>> > > > > > > > > >> consistent
> > > >> > > > >>>> > > > > > > > > >> > > with
> > > >> > > > >>>> > > > > > > > > >> > > > the
> > > >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > best,
> > > >> > > > >>>> > > > > > > > > >> > > > > > Colin
> > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an
> > alternative
> > > >> > > > >>>> configuration
> > > >> > > > >>>> > is to
> > > >> > > > >>>> > > > use
> > > >> > > > >>>> > > > > > -1
> > > >> > > > >>>> > > > > > > > as
> > > >> > > > >>>> > > > > > > > > >> > > > "disabled"
> > > >> > > > >>>> > > > > > > > > >> > > > > > and
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate
> > compaction".
> > > >> > > Because
> > > >> > > > >>>> > compaction
> > > >> > > > >>>> > > > lag
> > > >> > > > >>>> > > > > > is
> > > >> > > > >>>> > > > > > > > > still
> > > >> > > > >>>> > > > > > > > > >> > > > determined
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> > > min.compaction.lag
> > > >> and
> > > >> > > how
> > > >> > > > >>>> long
> > > >> > > > >>>> > to
> > > >> > > > >>>> > > > roll
> > > >> > > > >>>> > > > > > an
> > > >> > > > >>>> > > > > > > > > active
> > > >> > > > >>>> > > > > > > > > >> > > > segment,
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > the
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for
> > compaction
> > > is
> > > >> > > > >>>> undetermined
> > > >> > > > >>>> > if
> > > >> > > > >>>> > > > we
> > > >> > > > >>>> > > > > > use
> > > >> > > > >>>> > > > > > > > "0".
> > > >> > > > >>>> > > > > > > > > >> On
> > > >> > > > >>>> > > > > > > > > >> > the
> > > >> > > > >>>> > > > > > > > > >> > > > other
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set
> > > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > >> > > > >>>> > > > > > > > to
> > > >> > > > >>>> > > > > > > > > >> > achieve
> > > >> > > > >>>> > > > > > > > > >> > > > the
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > same
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose
> > > "0"
> > > >> as
> > > >> > > > >>>> "disabled".
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be
> > the
> > > >> > invalid
> > > >> > > > >>>> setting.
> > > >> > > > >>>> > > > > > Treating
> > > >> > > > >>>> > > > > > > 0
> > > >> > > > >>>> > > > > > > > > >> > > differently
> > > >> > > > >>>> > > > > > > > > >> > > > > > than
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous
> > > >> comment, I
> > > >> > > am
> > > >> > > > >>>> not
> > > >> > > > >>>> > > > strongly
> > > >> > > > >>>> > > > > > > > against,
> > > >> > > > >>>> > > > > > > > > >> but
> > > >> > > > >>>> > > > > > > > > >> > 0
> > > >> > > > >>>> > > > > > > > > >> > > is
> > > >> > > > >>>> > > > > > > > > >> > > > > > not a
> > > >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in my
> > > >> option.
> > > >> > So
> > > >> > > I
> > > >> > > > >>>> use
> > > >> > > > >>>> > "0" as
> > > >> > > > >>>> > > > > > > disabled
> > > >> > > > >>>> > > > > > > > > >> state.
> > > >> > > > >>>> > > > > > > > > >> > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > best,
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at
> > > 15:04,
> > > >> > > > xiongqi
> > > >> > > > >>>> wu
> > > >> > > > >>>> > wrote:
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> >>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > >> > > > >>>> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> >>>>
> > > >> > > > >>>> > > > > > > > > >> > > <
> > > >> > > > >>>> > > >
> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> >>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > >> > > > >>>> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> >>>>>
> > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > >> > > > >>>> > > > > > >
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> >>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > >> > > > >>>> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> >>>>
> > > >> > > > >>>> > > > > > > > > >> > > <
> > > >> > > > >>>> > > >
> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> >>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > >> > > > >>>> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>
> > > >> > > > >>>> > > > > > > > <
> > > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >
> > > >> > > > >>>> > > > > > <
> > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > > > >>>> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >> > >>>>>>
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > >> > > > %3A+Time-based+log+compaction+policy
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > >> > > > >>>> https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > >> > > > >>>> > > > > > > > > >> > > <
> > > >> https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > > >> > > > >>>> > > > > > > > > >> > > > > > <
> > > >> > > > https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > >> > > > >>>> > > > > > > > > >> > > <
> > > >> https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > >> > > > >>>> > > > > > > > > >> <
> > > https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> > > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > > > >
> > > >> > > > >>>> > > > > > > > > >> > > >
> > > >> > > > >>>> > > > > > > > > >> > >
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> > --
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> > Brett Rann
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000
> > > >> > > Australia
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > > >> > > > >>>> > > > > > > > > >> >
> > > >> > > > >>>> > > > > > > > > >>
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > > --
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > > Brett Rann
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> > > >> Australia
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > >> > > > >>>> > > > > > > > > >
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > --
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > Brett Rann
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > Zendesk International Ltd
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> > > >> Australia
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > >> > > > >>>> > > > > > > > >
> > > >> > > > >>>> > > > > > > >
> > > >> > > > >>>> > > > > > >
> > > >> > > > >>>> > > > > >
> > > >> > > > >>>> > > >
> > > >> > > > >>>> >
> > > >> > > > >>>>
> > > >> > > > >>>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by xiongqi wu <xi...@gmail.com>.
Thanks for binding and non-binding votes.
Can I get one more binding vote?
Thanks in advance!
Xiongqi (Wesley) Wu
On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer <ma...@frmr.me> wrote:
> I'm a +1 (non-binding) — This looks like it would have saved us a lot of
> pain in an issue we had to debug recently. I can't go into details, but
> figuring out how to achieve this effect gave me quite a headache. :)
>
> On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xi...@gmail.com> wrote:
>
> > Hi all,
> >
> > Can I have one more vote on this KIP?
> > Any comment is appreciated.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
> >
> >
> > Xiongqi (Wesley) Wu
> >
> >
> > On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xi...@gmail.com> wrote:
> >
> > > Thanks Dong.
> > > I have updated the KIP.
> > >
> > > Xiongqi (Wesley) Wu
> > >
> > >
> > > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com> wrote:
> > >
> > >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> > >>
> > >> One minor comment: it may be a bit better to clarify in the public
> > >> interface section that the value of the newly added metric is
> determined
> > >> based by applying that formula across all compactable segments. For
> > >> example:
> > >>
> > >> The maximum value of Math.max(now -
> > >> earliest_timestamp_in_ms_of_uncompacted_segment -
> max.compaction.lag.ms
> > ,
> > >> 0)/1000 across all compactable partitions, where the
> > >> max.compaction.lag.ms
> > >> can be overridden on per-topic basis.
> > >>
> > >>
> > >>
> > >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com>
> wrote:
> > >>
> > >> > Thanks Joel.
> > >> > Tracking the delay at second granularity makes sense
> > >> > I have updated KIP.
> > >> >
> > >> > Xiongqi (Wesley) Wu
> > >> >
> > >> >
> > >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com>
> > wrote:
> > >> >
> > >> > > +1 with one suggestion on the proposed metric. You should probably
> > >> > include
> > >> > > the unit. So for e.g., max-compaction-delay-secs.
> > >> > >
> > >> > > Joel
> > >> > >
> > >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xi...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > > > bump
> > >> > > > Xiongqi (Wesley) Wu
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xiongqiwu@gmail.com
> >
> > >> > wrote:
> > >> > > >
> > >> > > > >
> > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for
> > >> feedback.
> > >> > > > > Can I have more feedback or VOTE on this KIP?
> > >> > > > >
> > >> > > > >
> > >> > > > > Xiongqi (Wesley) Wu
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> > xiongqiwu@gmail.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > >> Any other votes or comments?
> > >> > > > >>
> > >> > > > >> Xiongqi (Wesley) Wu
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> > xiongqiwu@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > > >>
> > >> > > > >>> Yes, more votes and code review.
> > >> > > > >>>
> > >> > > > >>> Xiongqi (Wesley) Wu
> > >> > > > >>>
> > >> > > > >>>
> > >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > >> > > <brann@zendesk.com.invalid
> > >> > > > >
> > >> > > > >>> wrote:
> > >> > > > >>>
> > >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > >> > > > >>>>
> > >> > > > >>>> Where do we go from here? More votes?
> > >> > > > >>>>
> > >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> > >> cmccabe@apache.org>
> > >> > > > >>>> wrote:
> > >> > > > >>>>
> > >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > >> > > > >>>> > > Thank you for comments. I will use '0' for now.
> > >> > > > >>>> > >
> > >> > > > >>>> > > If we create topics through admin client in the future,
> > we
> > >> > might
> > >> > > > >>>> perform
> > >> > > > >>>> > > some useful checks. (but the assumption is all brokers
> in
> > >> the
> > >> > > same
> > >> > > > >>>> > cluster
> > >> > > > >>>> > > have the same default configurations value,
> otherwise,it
> > >> might
> > >> > > > >>>> still be
> > >> > > > >>>> > > tricky to do such cross validation check.)
> > >> > > > >>>> >
> > >> > > > >>>> > This isn't something that we might do in the future--
> this
> > is
> > >> > > > >>>> something we
> > >> > > > >>>> > are doing now. We already have Create Topic policies
> which
> > >> are
> > >> > > > >>>> enforced by
> > >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details. This
> is
> > >> one
> > >> > of
> > >> > > > the
> > >> > > > >>>> > motivations for getting rid of direct ZK access-- making
> > sure
> > >> > that
> > >> > > > >>>> these
> > >> > > > >>>> > policies are applied.
> > >> > > > >>>> >
> > >> > > > >>>> > I agree that having different configurations on different
> > >> > brokers
> > >> > > > can
> > >> > > > >>>> be
> > >> > > > >>>> > confusing and frustrating . That's why more
> configurations
> > >> are
> > >> > > being
> > >> > > > >>>> made
> > >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are stored
> > >> > centrally
> > >> > > > in
> > >> > > > >>>> ZK,
> > >> > > > >>>> > so they are the same on all brokers (modulo propagation
> > >> delays).
> > >> > > In
> > >> > > > >>>> any
> > >> > > > >>>> > case, this is a general issue, not specific to "create
> > >> topics".
> > >> > > > >>>> >
> > >> > > > >>>> > cheers,
> > >> > > > >>>> > Colin
> > >> > > > >>>> >
> > >> > > > >>>> >
> > >> > > > >>>> > >
> > >> > > > >>>> > >
> > >> > > > >>>> > > Xiongqi (Wesley) Wu
> > >> > > > >>>> > >
> > >> > > > >>>> > >
> > >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> > >> > > cmccabe@apache.org
> > >> > > > >
> > >> > > > >>>> > wrote:
> > >> > > > >>>> > >
> > >> > > > >>>> > > > I don't have a strong opinion. But I think we should
> > >> > probably
> > >> > > be
> > >> > > > >>>> > > > consistent with how segment.ms works, and just use
> 0.
> > >> > > > >>>> > > >
> > >> > > > >>>> > > > best,
> > >> > > > >>>> > > > Colin
> > >> > > > >>>> > > >
> > >> > > > >>>> > > >
> > >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> > >> > > > >>>> > > > > OK thanks for that clarification. I see why you're
> > >> > > > uncomfortable
> > >> > > > >>>> > with 0
> > >> > > > >>>> > > > now.
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > > I'm not really fussed. I just prefer consistency in
> > >> > > > >>>> configuration
> > >> > > > >>>> > > > options.
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > > Personally I lean towards treating 0 and 1
> similarly
> > in
> > >> > that
> > >> > > > >>>> > scenario,
> > >> > > > >>>> > > > > because it favours the person thinking about
> setting
> > >> the
> > >> > > > >>>> > configurations,
> > >> > > > >>>> > > > > and a person doesn't care about a 1ms edge case
> > >> especially
> > >> > > > when
> > >> > > > >>>> the
> > >> > > > >>>> > > > context
> > >> > > > >>>> > > > > is the true minimum is tied to the log cleaner
> > cadence.
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > > Introducing 0 to mean "disabled" because there is
> > some
> > >> > > > >>>> uniquness in
> > >> > > > >>>> > > > > segment.ms not being able to be set to 0, reduces
> > >> > > > configuration
> > >> > > > >>>> > > > consistency
> > >> > > > >>>> > > > > in favour of capturing a MS gap in an edge case
> that
> > >> > nobody
> > >> > > > >>>> would
> > >> > > > >>>> > ever
> > >> > > > >>>> > > > > notice. For someone to understand why everywhere
> else
> > >> -1
> > >> > is
> > >> > > > >>>> used to
> > >> > > > >>>> > > > > disable, but here 0 is used, they would need to
> learn
> > >> > about
> > >> > > > >>>> > segment.ms
> > >> > > > >>>> > > > > having a 1ms minimum and then after learning would
> > >> think
> > >> > > "who
> > >> > > > >>>> cares
> > >> > > > >>>> > about
> > >> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > > my 2c anyway. Will again defer to majority. Curious
> > >> which
> > >> > > way
> > >> > > > >>>> Colin
> > >> > > > >>>> > falls
> > >> > > > >>>> > > > > now.
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > > Don't want to spend more time on this though, It's
> > well
> > >> > into
> > >> > > > >>>> > > > bikeshedding
> > >> > > > >>>> > > > > territory now. :)
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > >> > > > xiongqiwu@gmail.com>
> > >> > > > >>>> > wrote:
> > >> > > > >>>> > > > >
> > >> > > > >>>> > > > > > I want to honor the minimum value of segment.ms
> > >> (which
> > >> > is
> > >> > > > >>>> 1ms) to
> > >> > > > >>>> > > > force
> > >> > > > >>>> > > > > > roll an active segment.
> > >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any value >
> > 0,
> > >> the
> > >> > > > >>>> minimum of
> > >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be
> used
> > to
> > >> > seal
> > >> > > > an
> > >> > > > >>>> > active
> > >> > > > >>>> > > > > > segment.
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the
> current
> > >> > > > >>>> implementation
> > >> > > > >>>> > will
> > >> > > > >>>> > > > > > treat it as disabled.
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > > It is a little bit weird to treat
> > >> max.compaction.lag=0
> > >> > the
> > >> > > > >>>> same as
> > >> > > > >>>> > > > > > max.compaction.lag=1.
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > > There might be a reason why we set the minimum of
> > >> > > > segment.ms
> > >> > > > >>>> to 1,
> > >> > > > >>>> > > > and I
> > >> > > > >>>> > > > > > don't want to break this assumption.
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > >> > > > >>>> > <br...@zendesk.com.invalid>
> > >> > > > >>>> > > > > > wrote:
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > > > > > You're rolling a new segment if the condition
> is
> > >> met
> > >> > > > right?
> > >> > > > >>>> So
> > >> > > > >>>> > I'm
> > >> > > > >>>> > > > > > > struggling to understand the relevance of
> > >> segment.ms
> > >> > > > here.
> > >> > > > >>>> > Maybe an
> > >> > > > >>>> > > > > > > example
> > >> > > > >>>> > > > > > > would help my understanding:
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > > segment.ms=9999999
> > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > > When a duplicate message comes in, after 1ms
> the
> > >> topic
> > >> > > > >>>> should be
> > >> > > > >>>> > > > eligible
> > >> > > > >>>> > > > > > > for compaction when the log compaction thread
> > gets
> > >> > > around
> > >> > > > to
> > >> > > > >>>> > > > evaluating
> > >> > > > >>>> > > > > > the
> > >> > > > >>>> > > > > > > topic.
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > > if we have
> > >> > > > >>>> > > > > > > segment.ms=9999999
> > >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > > When a duplicate message comes in, after 0ms
> the
> > >> topic
> > >> > > > >>>> should be
> > >> > > > >>>> > > > eligible
> > >> > > > >>>> > > > > > > for compaction when the log compaction thread
> > gets
> > >> > > around
> > >> > > > to
> > >> > > > >>>> > > > evaluating
> > >> > > > >>>> > > > > > the
> > >> > > > >>>> > > > > > > topic.
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > > In both of those cases the change would mean a
> > new
> > >> > > segment
> > >> > > > >>>> is
> > >> > > > >>>> > rolled
> > >> > > > >>>> > > > so
> > >> > > > >>>> > > > > > the
> > >> > > > >>>> > > > > > > new message would be part of the compaction
> > task. 0
> > >> > and
> > >> > > 1
> > >> > > > >>>> are
> > >> > > > >>>> > > > practically
> > >> > > > >>>> > > > > > > the same meaning since neither is providing an
> > >> actual
> > >> > > > >>>> guarantee
> > >> > > > >>>> > at
> > >> > > > >>>> > > > such
> > >> > > > >>>> > > > > > low
> > >> > > > >>>> > > > > > > MS settings, but effectively tying it to both
> the
> > >> > > > frequency
> > >> > > > >>>> of
> > >> > > > >>>> > the
> > >> > > > >>>> > > > log
> > >> > > > >>>> > > > > > > cleaner running and the priority of the given
> > topic
> > >> > > being
> > >> > > > >>>> the
> > >> > > > >>>> > highest
> > >> > > > >>>> > > > > > > priority of all topics that are evaluated for
> > >> cleaning
> > >> > > on
> > >> > > > >>>> the
> > >> > > > >>>> > next
> > >> > > > >>>> > > > cycle.
> > >> > > > >>>> > > > > > > You've captured that nuance with careful
> > "skipped"
> > >> > > wording
> > >> > > > >>>> in
> > >> > > > >>>> > the KIP
> > >> > > > >>>> > > > > > > here "controls
> > >> > > > >>>> > > > > > > the max time interval a message/segment can be
> > >> skipped
> > >> > > for
> > >> > > > >>>> log
> > >> > > > >>>> > > > > > compaction".
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > > How is 0 different to 1, practically? And how
> is
> > it
> > >> > > > >>>> relating to
> > >> > > > >>>> > > > > > segment.ms
> > >> > > > >>>> > > > > > > ?
> > >> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean "use
> > >> > > > segment.ms
> > >> > > > >>>> > > > instead?" as
> > >> > > > >>>> > > > > > a
> > >> > > > >>>> > > > > > > kind of third option?
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
> > >> > > > >>>> xiongqiwu@gmail.com>
> > >> > > > >>>> > > > wrote:
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > > > > To make it clear,
> > >> > > > >>>> > > > > > > > I don't against using -1 as disabled, but we
> > >> need to
> > >> > > > come
> > >> > > > >>>> up
> > >> > > > >>>> > with
> > >> > > > >>>> > > > the
> > >> > > > >>>> > > > > > > > meaning of "0".
> > >> > > > >>>> > > > > > > > If "0" means immediate compaction, but the
> > actual
> > >> > > > >>>> compaction
> > >> > > > >>>> > lag
> > >> > > > >>>> > > > will
> > >> > > > >>>> > > > > > be
> > >> > > > >>>> > > > > > > > segment.ms.
> > >> > > > >>>> > > > > > > > It has longer lag than setting the value to
> be
> > >> half
> > >> > of
> > >> > > > >>>> > segment.ms.
> > >> > > > >>>> > > > > > > > We cannot provide "0" as max compaction lag.
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > > Here are two options.
> > >> > > > >>>> > > > > > > > Option 1:
> > >> > > > >>>> > > > > > > > Keep 0 as disabled
> > >> > > > >>>> > > > > > > > Option 2:
> > >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag =
> > >> segment.ms),
> > >> > > and
> > >> > > > >>>> > others.
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> > >> > > > >>>> > > > <br...@zendesk.com.invalid>
> > >> > > > >>>> > > > > > > > wrote:
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > > > -1 is consistent as "special" with these
> > >> settings
> > >> > > for
> > >> > > > >>>> > example:
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > log.retention.bytes
> > >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > >> > > > >>>> > > > > > > > > queued.max.request.bytes
> > >> > > > >>>> > > > > > > > > retention.bytes
> > >> > > > >>>> > > > > > > > > retention.ms
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > and acks.
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS
> defaults,
> > >> max
> > >> > > > (acks),
> > >> > > > >>>> > etc. I
> > >> > > > >>>> > > > don't
> > >> > > > >>>> > > > > > > see
> > >> > > > >>>> > > > > > > > > much convention of 0 meaning those things.
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > There are some NULLs but it seems convetion
> > >> there
> > >> > is
> > >> > > > >>>> NULL is
> > >> > > > >>>> > used
> > >> > > > >>>> > > > > > where
> > >> > > > >>>> > > > > > > > > there's another setting in the hierarchy
> that
> > >> > would
> > >> > > be
> > >> > > > >>>> used
> > >> > > > >>>> > > > instead.
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann
> <
> > >> > > > >>>> > brann@zendesk.com>
> > >> > > > >>>> > > > > > wrote:
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0, then
> > we're
> > >> not
> > >> > > > being
> > >> > > > >>>> > > > consistent
> > >> > > > >>>> > > > > > > > > > by using 0 for this new setting? I throw
> > out
> > >> -1
> > >> > > for
> > >> > > > >>>> > > > consideration
> > >> > > > >>>> > > > > > > > > > again :)
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi
> wu
> > <
> > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > >> > > > >>>> > > > > > > > wrote:
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is
> > merged.
> > >> > > > >>>> > > > > > > > > >>
> > >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "
> > >> segment.ms"
> > >> > > to
> > >> > > > >>>> 1, we
> > >> > > > >>>> > > > cannot
> > >> > > > >>>> > > > > > set
> > >> > > > >>>> > > > > > > "
> > >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > >> > > > >>>> > > > > > > > > >>
> > >> > > > >>>> > > > > > > > > >> I also updated the title of this KIP.
> > >> > > > >>>> > > > > > > > > >>
> > >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > >> > > > >>>> > > > > > > > > >>
> > >> > > > >>>> > > > > > > > > >>
> > >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett
> Rann
> > >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> wrote:
> > >> > > > >>>> > > > > > > > > >>
> > >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm
> > in
> > >> the
> > >> > > > >>>> minority.
> > >> > > > >>>> > :)
> > >> > > > >>>> > > > Can
> > >> > > > >>>> > > > > > we
> > >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning
> > >> > disabled
> > >> > > > >>>> here:
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > >> > > > >>>> > > > > > <
> > >> https://kafka.apache.org/documentation/#brokerconfigs
> > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > >> > > > >>>> > > > > > > > <
> > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > >> > > > >>>> > > > > > <
> > >> https://kafka.apache.org/documentation/#brokerconfigs
> > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs
> >>>
> > >> > > > >>>> > > > > > > > > >> <
> > >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > >> > > > >>>> > > > > > <
> > >> https://kafka.apache.org/documentation/#brokerconfigs
> > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > >> > > > >>>> > > > > > > > <
> > >> > > https://kafka.apache.org/documentation/#brokerconfigs
> > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > >> > > > >>>> > > > > > <
> > >> https://kafka.apache.org/documentation/#brokerconfigs
> > >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs
> >>>>
> > ?
> > >> > > > >>>> > > > > > > > > >> > And while there it would be good if
> > >> > segment.ms
> > >> > > > is
> > >> > > > >>>> > > > documented
> > >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some
> > >> > hierarchy
> > >> > > > of
> > >> > > > >>>> > configs
> > >> > > > >>>> > > > for
> > >> > > > >>>> > > > > > > that
> > >> > > > >>>> > > > > > > > > too
> > >> > > > >>>> > > > > > > > > >> > if its not set and null for others
> means
> > >> > > > disabled!)
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi
> > wu
> > >> <
> > >> > > > >>>> > > > xiongqiwu@gmail.com>
> > >> > > > >>>> > > > > > > > > wrote:
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate
> > >> > compaction,
> > >> > > > the
> > >> > > > >>>> > > > compaction
> > >> > > > >>>> > > > > > lag
> > >> > > > >>>> > > > > > > > is
> > >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in worst
> > >> case. If
> > >> > > > >>>> segment.ms
> > >> > > > >>>> > is
> > >> > > > >>>> > > > 24
> > >> > > > >>>> > > > > > > > hours,
> > >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker
> > >> > guarantee
> > >> > > > than
> > >> > > > >>>> > setting
> > >> > > > >>>> > > > any
> > >> > > > >>>> > > > > > > > value
> > >> > > > >>>> > > > > > > > > >> less
> > >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the definition of
> > "max
> > >> > > > >>>> compaction
> > >> > > > >>>> > lag",
> > >> > > > >>>> > > > we
> > >> > > > >>>> > > > > > > > cannot
> > >> > > > >>>> > > > > > > > > >> have
> > >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate
> > >> "disable".
> > >> > > > >>>> > > > > > > > > >> > >
> > >> > > > >>>> > > > > > > > > >> > >
> > >> > > > >>>> > > > > > > > > >> > >
> > >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > >> > > > >>>> > > > > > > > > >> > >
> > >> > > > >>>> > > > > > > > > >> > >
> > >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin
> > >> > McCabe <
> > >> > > > >>>> > > > > > cmccabe@apache.org
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> wrote:
> > >> > > > >>>> > > > > > > > > >> > >
> > >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11,
> Brett
> > >> Rann
> > >> > > > >>>> wrote:
> > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should
> > >> make
> > >> > 0 =
> > >> > > > >>>> > disable, to
> > >> > > > >>>> > > > be
> > >> > > > >>>> > > > > > > > > >> consistent
> > >> > > > >>>> > > > > > > > > >> > > with
> > >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > >> > > > >>>> > > > > > > > > >> > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable
> > and
> > >> > when
> > >> > > > >>>> seeing
> > >> > > > >>>> > it
> > >> > > > >>>> > > > in a
> > >> > > > >>>> > > > > > > > config
> > >> > > > >>>> > > > > > > > > >> it's
> > >> > > > >>>> > > > > > > > > >> > > > clear
> > >> > > > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0
> > >> doesn't
> > >> > > have
> > >> > > > >>>> to
> > >> > > > >>>> > mean
> > >> > > > >>>> > > > > > instant,
> > >> > > > >>>> > > > > > > > it
> > >> > > > >>>> > > > > > > > > >> just
> > >> > > > >>>> > > > > > > > > >> > > > means
> > >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't
> > >> think 0
> > >> > > is
> > >> > > > >>>> > intuitive
> > >> > > > >>>> > > > for
> > >> > > > >>>> > > > > > > > > disabled
> > >> > > > >>>> > > > > > > > > >> and
> > >> > > > >>>> > > > > > > > > >> > > it
> > >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't
> aware
> > >> > > > segment.ms=0
> > >> > > > >>>> ==
> > >> > > > >>>> > > > > > disabled,
> > >> > > > >>>> > > > > > > > > but I
> > >> > > > >>>> > > > > > > > > >> > > think
> > >> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> > >> > > > >>>> > > > > > > > > >> > > >
> > >> > > > >>>> > > > > > > > > >> > > > I think there is an argument for
> > >> keeping
> > >> > > > these
> > >> > > > >>>> two
> > >> > > > >>>> > > > > > > > configurations
> > >> > > > >>>> > > > > > > > > >> > > > consistent, since they are so
> > >> similar. I
> > >> > > > agree
> > >> > > > >>>> that
> > >> > > > >>>> > 0
> > >> > > > >>>> > > > was an
> > >> > > > >>>> > > > > > > > > >> > unfortunate
> > >> > > > >>>> > > > > > > > > >> > > > choice.,
> > >> > > > >>>> > > > > > > > > >> > > >
> > >> > > > >>>> > > > > > > > > >> > > > best,
> > >> > > > >>>> > > > > > > > > >> > > > Colin
> > >> > > > >>>> > > > > > > > > >> > > >
> > >> > > > >>>> > > > > > > > > >> > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM
> > >> Colin
> > >> > > > McCabe
> > >> > > > >>>> <
> > >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > >> > > > >>>> > > > > > > > > >> > > wrote:
> > >> > > > >>>> > > > > > > > > >> > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47,
> > >> > xiongqi
> > >> > > wu
> > >> > > > >>>> wrote:
> > >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> > >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply below.
> > >> > > > >>>> > > > > > > > > >> > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > >> > > > >>>> > > > > > > > > >> > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24
> PM
> > >> > Colin
> > >> > > > >>>> McCabe <
> > >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > >> > > > >>>> > > > > > > > > >> > > > wrote:
> > >> > > > >>>> > > > > > > > > >> > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit
> > >> ambiguous.
> > >> > Our
> > >> > > > >>>> > compaction
> > >> > > > >>>> > > > > > > policies
> > >> > > > >>>> > > > > > > > > are
> > >> > > > >>>> > > > > > > > > >> > > already
> > >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It
> > >> seems
> > >> > > like
> > >> > > > >>>> this
> > >> > > > >>>> > > > change is
> > >> > > > >>>> > > > > > > > > focused
> > >> > > > >>>> > > > > > > > > >> > > around
> > >> > > > >>>> > > > > > > > > >> > > > > > adding
> > >> > > > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms
> ."
> > >> > Perhaps
> > >> > > > >>>> the KIP
> > >> > > > >>>> > > > title
> > >> > > > >>>> > > > > > > should
> > >> > > > >>>> > > > > > > > > be
> > >> > > > >>>> > > > > > > > > >> > > > something
> > >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum
> compaction
> > >> lag
> > >> > > > time"?
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will
> > >> change
> > >> > the
> > >> > > > >>>> title.
> > >> > > > >>>> > > > > > > > > >> > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is
> forced
> > >> to
> > >> > > roll
> > >> > > > >>>> when
> > >> > > > >>>> > > > either "
> > >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (
> > >> log.roll.ms
> > >> > and
> > >> > > > >>>> > > > log.roll.hours)
> > >> > > > >>>> > > > > > > has
> > >> > > > >>>> > > > > > > > > >> > reached.
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> > max.compaction.lag.ms
> > >> is
> > >> > > low,
> > >> > > > >>>> it
> > >> > > > >>>> > seems
> > >> > > > >>>> > > > like
> > >> > > > >>>> > > > > > > > > segments
> > >> > > > >>>> > > > > > > > > >> > will
> > >> > > > >>>> > > > > > > > > >> > > > be
> > >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very frequently.
> This
> > >> can
> > >> > > be a
> > >> > > > >>>> > source of
> > >> > > > >>>> > > > > > > problems
> > >> > > > >>>> > > > > > > > > in
> > >> > > > >>>> > > > > > > > > >> the
> > >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > >> > > > >>>> > > > > > > > > >> > > > > > > > since creating many
> > different
> > >> > small
> > >> > > > log
> > >> > > > >>>> > segments
> > >> > > > >>>> > > > > > > > consumes
> > >> > > > >>>> > > > > > > > > a
> > >> > > > >>>> > > > > > > > > >> > huge
> > >> > > > >>>> > > > > > > > > >> > > > > > amount of
> > >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources.
> > Therefore,
> > >> I
> > >> > > would
> > >> > > > >>>> > suggest
> > >> > > > >>>> > > > > > adding a
> > >> > > > >>>> > > > > > > > > >> > > broker-level
> > >> > > > >>>> > > > > > > > > >> > > > > > > > configuration which allows
> > us
> > >> to
> > >> > > set
> > >> > > > a
> > >> > > > >>>> > minimum
> > >> > > > >>>> > > > value
> > >> > > > >>>> > > > > > > for
> > >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If
> > we
> > >> let
> > >> > > > >>>> users set
> > >> > > > >>>> > it
> > >> > > > >>>> > > > on a
> > >> > > > >>>> > > > > > > > > >> per-topic
> > >> > > > >>>> > > > > > > > > >> > > > basis,
> > >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a value
> of
> > >> 1 ms
> > >> > > or
> > >> > > > >>>> > something,
> > >> > > > >>>> > > > and
> > >> > > > >>>> > > > > > > > cause
> > >> > > > >>>> > > > > > > > > >> > chaos.
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this applies to
> > >> > > > segment.ms
> > >> > > > >>>> as
> > >> > > > >>>> > well.
> > >> > > > >>>> > > > > > Today
> > >> > > > >>>> > > > > > > > > users
> > >> > > > >>>> > > > > > > > > >> can
> > >> > > > >>>> > > > > > > > > >> > > > set "
> > >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low
> > >> value,
> > >> > and
> > >> > > > >>>> cause a
> > >> > > > >>>> > > > frequent
> > >> > > > >>>> > > > > > > > > >> rolling of
> > >> > > > >>>> > > > > > > > > >> > > > active
> > >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > >> > > > >>>> > > > > > > > > >> > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > >> > > > >>>> > > > > > > > > >> > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an
> existing
> > >> > > problem
> > >> > > > >>>> with
> > >> > > > >>>> > > > > > segment.ms.
> > >> > > > >>>> > > > > > > > > >> However,
> > >> > > > >>>> > > > > > > > > >> > > > that
> > >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't
> > fix
> > >> > it.
> > >> > > As
> > >> > > > >>>> you
> > >> > > > >>>> > noted,
> > >> > > > >>>> > > > > > there
> > >> > > > >>>> > > > > > > > > will
> > >> > > > >>>> > > > > > > > > >> be
> > >> > > > >>>> > > > > > > > > >> > > more
> > >> > > > >>>> > > > > > > > > >> > > > > > interest in these topic-level
> > >> > retention
> > >> > > > >>>> > settings as
> > >> > > > >>>> > > > a
> > >> > > > >>>> > > > > > > result
> > >> > > > >>>> > > > > > > > > of
> > >> > > > >>>> > > > > > > > > >> > GDPR.
> > >> > > > >>>> > > > > > > > > >> > > > It
> > >> > > > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing
> > >> > problems
> > >> > > > >>>> will
> > >> > > > >>>> > cause
> > >> > > > >>>> > > > more
> > >> > > > >>>> > > > > > > > > trouble.
> > >> > > > >>>> > > > > > > > > >> > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> > >> > > straightforward
> > >> > > > >>>> here --
> > >> > > > >>>> > > > add a
> > >> > > > >>>> > > > > > > > > >> broker-level
> > >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that
> > overrides
> > >> > > > >>>> per-topic
> > >> > > > >>>> > > > minimums.
> > >> > > > >>>> > > > > > We
> > >> > > > >>>> > > > > > > > can
> > >> > > > >>>> > > > > > > > > >> also
> > >> > > > >>>> > > > > > > > > >> > > fail
> > >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error message
> > when
> > >> > > someone
> > >> > > > >>>> > attempts
> > >> > > > >>>> > > > to
> > >> > > > >>>> > > > > > set
> > >> > > > >>>> > > > > > > an
> > >> > > > >>>> > > > > > > > > >> > invalid
> > >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > >> > > > >>>> > > > > > > > > >> > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the minimum
> of "
> > >> > > > >>>> > > > max.compaction.lag.ms"
> > >> > > > >>>> > > > > > > > should
> > >> > > > >>>> > > > > > > > > >> be
> > >> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of "
> > >> > segment.ms
> > >> > > ".
> > >> > > > >>>> Since
> > >> > > > >>>> > > > today the
> > >> > > > >>>> > > > > > > > > >> minimum of
> > >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "
> max.compaction.lag.ms"
> > >> also
> > >> > > > >>>> starts
> > >> > > > >>>> > with
> > >> > > > >>>> > > > 1. "0"
> > >> > > > >>>> > > > > > > > means
> > >> > > > >>>> > > > > > > > > >> > > > disable. I
> > >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but
> it
> > is
> > >> > hard
> > >> > > > to
> > >> > > > >>>> > define
> > >> > > > >>>> > > > the
> > >> > > > >>>> > > > > > > > meaning
> > >> > > > >>>> > > > > > > > > >> of 0
> > >> > > > >>>> > > > > > > > > >> > > > because
> > >> > > > >>>> > > > > > > > > >> > > > > > we
> > >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the active
> > >> segment
> > >> > > > >>>> > immediately.
> > >> > > > >>>> > > > > > > > > >> > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should
> > >> make
> > >> > 0 =
> > >> > > > >>>> > disable, to
> > >> > > > >>>> > > > be
> > >> > > > >>>> > > > > > > > > >> consistent
> > >> > > > >>>> > > > > > > > > >> > > with
> > >> > > > >>>> > > > > > > > > >> > > > the
> > >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > >> > > > >>>> > > > > > > > > >> > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > best,
> > >> > > > >>>> > > > > > > > > >> > > > > > Colin
> > >> > > > >>>> > > > > > > > > >> > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an
> alternative
> > >> > > > >>>> configuration
> > >> > > > >>>> > is to
> > >> > > > >>>> > > > use
> > >> > > > >>>> > > > > > -1
> > >> > > > >>>> > > > > > > > as
> > >> > > > >>>> > > > > > > > > >> > > > "disabled"
> > >> > > > >>>> > > > > > > > > >> > > > > > and
> > >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate
> compaction".
> > >> > > Because
> > >> > > > >>>> > compaction
> > >> > > > >>>> > > > lag
> > >> > > > >>>> > > > > > is
> > >> > > > >>>> > > > > > > > > still
> > >> > > > >>>> > > > > > > > > >> > > > determined
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> > min.compaction.lag
> > >> and
> > >> > > how
> > >> > > > >>>> long
> > >> > > > >>>> > to
> > >> > > > >>>> > > > roll
> > >> > > > >>>> > > > > > an
> > >> > > > >>>> > > > > > > > > active
> > >> > > > >>>> > > > > > > > > >> > > > segment,
> > >> > > > >>>> > > > > > > > > >> > > > > > > > the
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for
> compaction
> > is
> > >> > > > >>>> undetermined
> > >> > > > >>>> > if
> > >> > > > >>>> > > > we
> > >> > > > >>>> > > > > > use
> > >> > > > >>>> > > > > > > > "0".
> > >> > > > >>>> > > > > > > > > >> On
> > >> > > > >>>> > > > > > > > > >> > the
> > >> > > > >>>> > > > > > > > > >> > > > other
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set
> > >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > >> > > > >>>> > > > > > > > to
> > >> > > > >>>> > > > > > > > > >> > achieve
> > >> > > > >>>> > > > > > > > > >> > > > the
> > >> > > > >>>> > > > > > > > > >> > > > > > > > same
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose
> > "0"
> > >> as
> > >> > > > >>>> "disabled".
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be
> the
> > >> > invalid
> > >> > > > >>>> setting.
> > >> > > > >>>> > > > > > Treating
> > >> > > > >>>> > > > > > > 0
> > >> > > > >>>> > > > > > > > > >> > > differently
> > >> > > > >>>> > > > > > > > > >> > > > > > than
> > >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous
> > >> comment, I
> > >> > > am
> > >> > > > >>>> not
> > >> > > > >>>> > > > strongly
> > >> > > > >>>> > > > > > > > against,
> > >> > > > >>>> > > > > > > > > >> but
> > >> > > > >>>> > > > > > > > > >> > 0
> > >> > > > >>>> > > > > > > > > >> > > is
> > >> > > > >>>> > > > > > > > > >> > > > > > not a
> > >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in my
> > >> option.
> > >> > So
> > >> > > I
> > >> > > > >>>> use
> > >> > > > >>>> > "0" as
> > >> > > > >>>> > > > > > > disabled
> > >> > > > >>>> > > > > > > > > >> state.
> > >> > > > >>>> > > > > > > > > >> > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > best,
> > >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at
> > 15:04,
> > >> > > > xiongqi
> > >> > > > >>>> wu
> > >> > > > >>>> > wrote:
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>
> > >> > > > >>>> > > > > > > > <
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> >>>
> > >> > > > >>>> > > > > > > > > >> <
> > >> > > > >>>> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>
> > >> > > > >>>> > > > > > > > <
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> >>>>
> > >> > > > >>>> > > > > > > > > >> > > <
> > >> > > > >>>> > > >
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>
> > >> > > > >>>> > > > > > > > <
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> >>>
> > >> > > > >>>> > > > > > > > > >> <
> > >> > > > >>>> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>
> > >> > > > >>>> > > > > > > > <
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> >>>>>
> > >> > > > >>>> > > > > > > > > >> > > > > > <
> > >> > > > >>>> > > > > > >
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>
> > >> > > > >>>> > > > > > > > <
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> >>>
> > >> > > > >>>> > > > > > > > > >> <
> > >> > > > >>>> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>
> > >> > > > >>>> > > > > > > > <
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> >>>>
> > >> > > > >>>> > > > > > > > > >> > > <
> > >> > > > >>>> > > >
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>
> > >> > > > >>>> > > > > > > > <
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> >>>
> > >> > > > >>>> > > > > > > > > >> <
> > >> > > > >>>> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>
> > >> > > > >>>> > > > > > > > <
> > >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >
> > >> > > > >>>> > > > > > <
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > > > >>>> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >> > >>>>>>
> > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > >> > > > %3A+Time-based+log+compaction+policy
> > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > >> > > > >>>> https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > >> > > > >>>> > > > > > > > > >> <
> > https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > >> > > > >>>> > > > > > > > > >> > > <
> > >> https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > >> > > > >>>> > > > > > > > > >> <
> > https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > >> > > > >>>> > > > > > > > > >> > > > > > <
> > >> > > > https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > >> > > > >>>> > > > > > > > > >> <
> > https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > >> > > > >>>> > > > > > > > > >> > > <
> > >> https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > >> > > > >>>> > > > > > > > > >> <
> > https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> > >> > > > >>>> > > > > > > > > >> > > > > > > >
> > >> > > > >>>> > > > > > > > > >> > > > > >
> > >> > > > >>>> > > > > > > > > >> > > >
> > >> > > > >>>> > > > > > > > > >> > >
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> > --
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> > Brett Rann
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000
> > >> > > Australia
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > >> > > > >>>> > > > > > > > > >> >
> > >> > > > >>>> > > > > > > > > >>
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > > --
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > > Brett Rann
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> > >> Australia
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > >> > > > >>>> > > > > > > > > >
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > --
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > Brett Rann
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > Zendesk International Ltd
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> > >> Australia
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > >> > > > >>>> > > > > > > > >
> > >> > > > >>>> > > > > > > >
> > >> > > > >>>> > > > > > >
> > >> > > > >>>> > > > > >
> > >> > > > >>>> > > >
> > >> > > > >>>> >
> > >> > > > >>>>
> > >> > > > >>>
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by Matt Farmer <ma...@frmr.me>.
I'm a +1 (non-binding) — This looks like it would have saved us a lot of
pain in an issue we had to debug recently. I can't go into details, but
figuring out how to achieve this effect gave me quite a headache. :)
On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu <xi...@gmail.com> wrote:
> Hi all,
>
> Can I have one more vote on this KIP?
> Any comment is appreciated.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
>
>
> Xiongqi (Wesley) Wu
>
>
> On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xi...@gmail.com> wrote:
>
> > Thanks Dong.
> > I have updated the KIP.
> >
> > Xiongqi (Wesley) Wu
> >
> >
> > On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com> wrote:
> >
> >> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
> >>
> >> One minor comment: it may be a bit better to clarify in the public
> >> interface section that the value of the newly added metric is determined
> >> based by applying that formula across all compactable segments. For
> >> example:
> >>
> >> The maximum value of Math.max(now -
> >> earliest_timestamp_in_ms_of_uncompacted_segment - max.compaction.lag.ms
> ,
> >> 0)/1000 across all compactable partitions, where the
> >> max.compaction.lag.ms
> >> can be overridden on per-topic basis.
> >>
> >>
> >>
> >> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com> wrote:
> >>
> >> > Thanks Joel.
> >> > Tracking the delay at second granularity makes sense
> >> > I have updated KIP.
> >> >
> >> > Xiongqi (Wesley) Wu
> >> >
> >> >
> >> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com>
> wrote:
> >> >
> >> > > +1 with one suggestion on the proposed metric. You should probably
> >> > include
> >> > > the unit. So for e.g., max-compaction-delay-secs.
> >> > >
> >> > > Joel
> >> > >
> >> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xi...@gmail.com>
> >> wrote:
> >> > >
> >> > > > bump
> >> > > > Xiongqi (Wesley) Wu
> >> > > >
> >> > > >
> >> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xi...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > > >
> >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for
> >> feedback.
> >> > > > > Can I have more feedback or VOTE on this KIP?
> >> > > > >
> >> > > > >
> >> > > > > Xiongqi (Wesley) Wu
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <
> xiongqiwu@gmail.com>
> >> > > wrote:
> >> > > > >
> >> > > > >> Any other votes or comments?
> >> > > > >>
> >> > > > >> Xiongqi (Wesley) Wu
> >> > > > >>
> >> > > > >>
> >> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <
> xiongqiwu@gmail.com
> >> >
> >> > > > wrote:
> >> > > > >>
> >> > > > >>> Yes, more votes and code review.
> >> > > > >>>
> >> > > > >>> Xiongqi (Wesley) Wu
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> >> > > <brann@zendesk.com.invalid
> >> > > > >
> >> > > > >>> wrote:
> >> > > > >>>
> >> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> >> > > > >>>>
> >> > > > >>>> Where do we go from here? More votes?
> >> > > > >>>>
> >> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> >> cmccabe@apache.org>
> >> > > > >>>> wrote:
> >> > > > >>>>
> >> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> >> > > > >>>> > > Thank you for comments. I will use '0' for now.
> >> > > > >>>> > >
> >> > > > >>>> > > If we create topics through admin client in the future,
> we
> >> > might
> >> > > > >>>> perform
> >> > > > >>>> > > some useful checks. (but the assumption is all brokers in
> >> the
> >> > > same
> >> > > > >>>> > cluster
> >> > > > >>>> > > have the same default configurations value, otherwise,it
> >> might
> >> > > > >>>> still be
> >> > > > >>>> > > tricky to do such cross validation check.)
> >> > > > >>>> >
> >> > > > >>>> > This isn't something that we might do in the future-- this
> is
> >> > > > >>>> something we
> >> > > > >>>> > are doing now. We already have Create Topic policies which
> >> are
> >> > > > >>>> enforced by
> >> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details. This is
> >> one
> >> > of
> >> > > > the
> >> > > > >>>> > motivations for getting rid of direct ZK access-- making
> sure
> >> > that
> >> > > > >>>> these
> >> > > > >>>> > policies are applied.
> >> > > > >>>> >
> >> > > > >>>> > I agree that having different configurations on different
> >> > brokers
> >> > > > can
> >> > > > >>>> be
> >> > > > >>>> > confusing and frustrating . That's why more configurations
> >> are
> >> > > being
> >> > > > >>>> made
> >> > > > >>>> > dynamic using KIP-226. Dynamic configurations are stored
> >> > centrally
> >> > > > in
> >> > > > >>>> ZK,
> >> > > > >>>> > so they are the same on all brokers (modulo propagation
> >> delays).
> >> > > In
> >> > > > >>>> any
> >> > > > >>>> > case, this is a general issue, not specific to "create
> >> topics".
> >> > > > >>>> >
> >> > > > >>>> > cheers,
> >> > > > >>>> > Colin
> >> > > > >>>> >
> >> > > > >>>> >
> >> > > > >>>> > >
> >> > > > >>>> > >
> >> > > > >>>> > > Xiongqi (Wesley) Wu
> >> > > > >>>> > >
> >> > > > >>>> > >
> >> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> >> > > cmccabe@apache.org
> >> > > > >
> >> > > > >>>> > wrote:
> >> > > > >>>> > >
> >> > > > >>>> > > > I don't have a strong opinion. But I think we should
> >> > probably
> >> > > be
> >> > > > >>>> > > > consistent with how segment.ms works, and just use 0.
> >> > > > >>>> > > >
> >> > > > >>>> > > > best,
> >> > > > >>>> > > > Colin
> >> > > > >>>> > > >
> >> > > > >>>> > > >
> >> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> >> > > > >>>> > > > > OK thanks for that clarification. I see why you're
> >> > > > uncomfortable
> >> > > > >>>> > with 0
> >> > > > >>>> > > > now.
> >> > > > >>>> > > > >
> >> > > > >>>> > > > > I'm not really fussed. I just prefer consistency in
> >> > > > >>>> configuration
> >> > > > >>>> > > > options.
> >> > > > >>>> > > > >
> >> > > > >>>> > > > > Personally I lean towards treating 0 and 1 similarly
> in
> >> > that
> >> > > > >>>> > scenario,
> >> > > > >>>> > > > > because it favours the person thinking about setting
> >> the
> >> > > > >>>> > configurations,
> >> > > > >>>> > > > > and a person doesn't care about a 1ms edge case
> >> especially
> >> > > > when
> >> > > > >>>> the
> >> > > > >>>> > > > context
> >> > > > >>>> > > > > is the true minimum is tied to the log cleaner
> cadence.
> >> > > > >>>> > > > >
> >> > > > >>>> > > > > Introducing 0 to mean "disabled" because there is
> some
> >> > > > >>>> uniquness in
> >> > > > >>>> > > > > segment.ms not being able to be set to 0, reduces
> >> > > > configuration
> >> > > > >>>> > > > consistency
> >> > > > >>>> > > > > in favour of capturing a MS gap in an edge case that
> >> > nobody
> >> > > > >>>> would
> >> > > > >>>> > ever
> >> > > > >>>> > > > > notice. For someone to understand why everywhere else
> >> -1
> >> > is
> >> > > > >>>> used to
> >> > > > >>>> > > > > disable, but here 0 is used, they would need to learn
> >> > about
> >> > > > >>>> > segment.ms
> >> > > > >>>> > > > > having a 1ms minimum and then after learning would
> >> think
> >> > > "who
> >> > > > >>>> cares
> >> > > > >>>> > about
> >> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> >> > > > >>>> > > > >
> >> > > > >>>> > > > > my 2c anyway. Will again defer to majority. Curious
> >> which
> >> > > way
> >> > > > >>>> Colin
> >> > > > >>>> > falls
> >> > > > >>>> > > > > now.
> >> > > > >>>> > > > >
> >> > > > >>>> > > > > Don't want to spend more time on this though, It's
> well
> >> > into
> >> > > > >>>> > > > bikeshedding
> >> > > > >>>> > > > > territory now. :)
> >> > > > >>>> > > > >
> >> > > > >>>> > > > >
> >> > > > >>>> > > > >
> >> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> >> > > > xiongqiwu@gmail.com>
> >> > > > >>>> > wrote:
> >> > > > >>>> > > > >
> >> > > > >>>> > > > > > I want to honor the minimum value of segment.ms
> >> (which
> >> > is
> >> > > > >>>> 1ms) to
> >> > > > >>>> > > > force
> >> > > > >>>> > > > > > roll an active segment.
> >> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any value >
> 0,
> >> the
> >> > > > >>>> minimum of
> >> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be used
> to
> >> > seal
> >> > > > an
> >> > > > >>>> > active
> >> > > > >>>> > > > > > segment.
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the current
> >> > > > >>>> implementation
> >> > > > >>>> > will
> >> > > > >>>> > > > > > treat it as disabled.
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > > It is a little bit weird to treat
> >> max.compaction.lag=0
> >> > the
> >> > > > >>>> same as
> >> > > > >>>> > > > > > max.compaction.lag=1.
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > > There might be a reason why we set the minimum of
> >> > > > segment.ms
> >> > > > >>>> to 1,
> >> > > > >>>> > > > and I
> >> > > > >>>> > > > > > don't want to break this assumption.
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> >> > > > >>>> > <br...@zendesk.com.invalid>
> >> > > > >>>> > > > > > wrote:
> >> > > > >>>> > > > > >
> >> > > > >>>> > > > > > > You're rolling a new segment if the condition is
> >> met
> >> > > > right?
> >> > > > >>>> So
> >> > > > >>>> > I'm
> >> > > > >>>> > > > > > > struggling to understand the relevance of
> >> segment.ms
> >> > > > here.
> >> > > > >>>> > Maybe an
> >> > > > >>>> > > > > > > example
> >> > > > >>>> > > > > > > would help my understanding:
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > > segment.ms=9999999
> >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> >> > > > >>>> > > > > > > max.compaction.lag.ms=1
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > > When a duplicate message comes in, after 1ms the
> >> topic
> >> > > > >>>> should be
> >> > > > >>>> > > > eligible
> >> > > > >>>> > > > > > > for compaction when the log compaction thread
> gets
> >> > > around
> >> > > > to
> >> > > > >>>> > > > evaluating
> >> > > > >>>> > > > > > the
> >> > > > >>>> > > > > > > topic.
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > > if we have
> >> > > > >>>> > > > > > > segment.ms=9999999
> >> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> >> > > > >>>> > > > > > > max.compaction.lag.ms=0
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > > When a duplicate message comes in, after 0ms the
> >> topic
> >> > > > >>>> should be
> >> > > > >>>> > > > eligible
> >> > > > >>>> > > > > > > for compaction when the log compaction thread
> gets
> >> > > around
> >> > > > to
> >> > > > >>>> > > > evaluating
> >> > > > >>>> > > > > > the
> >> > > > >>>> > > > > > > topic.
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > > In both of those cases the change would mean a
> new
> >> > > segment
> >> > > > >>>> is
> >> > > > >>>> > rolled
> >> > > > >>>> > > > so
> >> > > > >>>> > > > > > the
> >> > > > >>>> > > > > > > new message would be part of the compaction
> task. 0
> >> > and
> >> > > 1
> >> > > > >>>> are
> >> > > > >>>> > > > practically
> >> > > > >>>> > > > > > > the same meaning since neither is providing an
> >> actual
> >> > > > >>>> guarantee
> >> > > > >>>> > at
> >> > > > >>>> > > > such
> >> > > > >>>> > > > > > low
> >> > > > >>>> > > > > > > MS settings, but effectively tying it to both the
> >> > > > frequency
> >> > > > >>>> of
> >> > > > >>>> > the
> >> > > > >>>> > > > log
> >> > > > >>>> > > > > > > cleaner running and the priority of the given
> topic
> >> > > being
> >> > > > >>>> the
> >> > > > >>>> > highest
> >> > > > >>>> > > > > > > priority of all topics that are evaluated for
> >> cleaning
> >> > > on
> >> > > > >>>> the
> >> > > > >>>> > next
> >> > > > >>>> > > > cycle.
> >> > > > >>>> > > > > > > You've captured that nuance with careful
> "skipped"
> >> > > wording
> >> > > > >>>> in
> >> > > > >>>> > the KIP
> >> > > > >>>> > > > > > > here "controls
> >> > > > >>>> > > > > > > the max time interval a message/segment can be
> >> skipped
> >> > > for
> >> > > > >>>> log
> >> > > > >>>> > > > > > compaction".
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > > How is 0 different to 1, practically? And how is
> it
> >> > > > >>>> relating to
> >> > > > >>>> > > > > > segment.ms
> >> > > > >>>> > > > > > > ?
> >> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean "use
> >> > > > segment.ms
> >> > > > >>>> > > > instead?" as
> >> > > > >>>> > > > > > a
> >> > > > >>>> > > > > > > kind of third option?
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
> >> > > > >>>> xiongqiwu@gmail.com>
> >> > > > >>>> > > > wrote:
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > > > > To make it clear,
> >> > > > >>>> > > > > > > > I don't against using -1 as disabled, but we
> >> need to
> >> > > > come
> >> > > > >>>> up
> >> > > > >>>> > with
> >> > > > >>>> > > > the
> >> > > > >>>> > > > > > > > meaning of "0".
> >> > > > >>>> > > > > > > > If "0" means immediate compaction, but the
> actual
> >> > > > >>>> compaction
> >> > > > >>>> > lag
> >> > > > >>>> > > > will
> >> > > > >>>> > > > > > be
> >> > > > >>>> > > > > > > > segment.ms.
> >> > > > >>>> > > > > > > > It has longer lag than setting the value to be
> >> half
> >> > of
> >> > > > >>>> > segment.ms.
> >> > > > >>>> > > > > > > > We cannot provide "0" as max compaction lag.
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > > Here are two options.
> >> > > > >>>> > > > > > > > Option 1:
> >> > > > >>>> > > > > > > > Keep 0 as disabled
> >> > > > >>>> > > > > > > > Option 2:
> >> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag =
> >> segment.ms),
> >> > > and
> >> > > > >>>> > others.
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> >> > > > >>>> > > > <br...@zendesk.com.invalid>
> >> > > > >>>> > > > > > > > wrote:
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > > > -1 is consistent as "special" with these
> >> settings
> >> > > for
> >> > > > >>>> > example:
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > log.retention.bytes
> >> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> >> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> >> > > > >>>> > > > > > > > > queued.max.request.bytes
> >> > > > >>>> > > > > > > > > retention.bytes
> >> > > > >>>> > > > > > > > > retention.ms
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > and acks.
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > Where it may mean no limit, use OS defaults,
> >> max
> >> > > > (acks),
> >> > > > >>>> > etc. I
> >> > > > >>>> > > > don't
> >> > > > >>>> > > > > > > see
> >> > > > >>>> > > > > > > > > much convention of 0 meaning those things.
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > There are some NULLs but it seems convetion
> >> there
> >> > is
> >> > > > >>>> NULL is
> >> > > > >>>> > used
> >> > > > >>>> > > > > > where
> >> > > > >>>> > > > > > > > > there's another setting in the hierarchy that
> >> > would
> >> > > be
> >> > > > >>>> used
> >> > > > >>>> > > > instead.
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann <
> >> > > > >>>> > brann@zendesk.com>
> >> > > > >>>> > > > > > wrote:
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0, then
> we're
> >> not
> >> > > > being
> >> > > > >>>> > > > consistent
> >> > > > >>>> > > > > > > > > > by using 0 for this new setting? I throw
> out
> >> -1
> >> > > for
> >> > > > >>>> > > > consideration
> >> > > > >>>> > > > > > > > > > again :)
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu
> <
> >> > > > >>>> > > > xiongqiwu@gmail.com>
> >> > > > >>>> > > > > > > > wrote:
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is
> merged.
> >> > > > >>>> > > > > > > > > >>
> >> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "
> >> segment.ms"
> >> > > to
> >> > > > >>>> 1, we
> >> > > > >>>> > > > cannot
> >> > > > >>>> > > > > > set
> >> > > > >>>> > > > > > > "
> >> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> >> > > > >>>> > > > > > > > > >>
> >> > > > >>>> > > > > > > > > >> I also updated the title of this KIP.
> >> > > > >>>> > > > > > > > > >>
> >> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> >> > > > >>>> > > > > > > > > >>
> >> > > > >>>> > > > > > > > > >>
> >> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett Rann
> >> > > > >>>> > > > > > <brann@zendesk.com.invalid
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > > > >> wrote:
> >> > > > >>>> > > > > > > > > >>
> >> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm
> in
> >> the
> >> > > > >>>> minority.
> >> > > > >>>> > :)
> >> > > > >>>> > > > Can
> >> > > > >>>> > > > > > we
> >> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning
> >> > disabled
> >> > > > >>>> here:
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> >> > > > >>>> > > > > > <
> >> https://kafka.apache.org/documentation/#brokerconfigs
> >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> >> > > > >>>> > > > > > > > <
> >> > > https://kafka.apache.org/documentation/#brokerconfigs
> >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> >> > > > >>>> > > > > > <
> >> https://kafka.apache.org/documentation/#brokerconfigs
> >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>
> >> > > > >>>> > > > > > > > > >> <
> >> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> >> > > > >>>> > > > > > <
> >> https://kafka.apache.org/documentation/#brokerconfigs
> >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> >> > > > >>>> > > > > > > > <
> >> > > https://kafka.apache.org/documentation/#brokerconfigs
> >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> >> > > > >>>> > > > > > <
> >> https://kafka.apache.org/documentation/#brokerconfigs
> >> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>>
> ?
> >> > > > >>>> > > > > > > > > >> > And while there it would be good if
> >> > segment.ms
> >> > > > is
> >> > > > >>>> > > > documented
> >> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some
> >> > hierarchy
> >> > > > of
> >> > > > >>>> > configs
> >> > > > >>>> > > > for
> >> > > > >>>> > > > > > > that
> >> > > > >>>> > > > > > > > > too
> >> > > > >>>> > > > > > > > > >> > if its not set and null for others means
> >> > > > disabled!)
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi
> wu
> >> <
> >> > > > >>>> > > > xiongqiwu@gmail.com>
> >> > > > >>>> > > > > > > > > wrote:
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate
> >> > compaction,
> >> > > > the
> >> > > > >>>> > > > compaction
> >> > > > >>>> > > > > > lag
> >> > > > >>>> > > > > > > > is
> >> > > > >>>> > > > > > > > > >> > > determined by segment.ms in worst
> >> case. If
> >> > > > >>>> segment.ms
> >> > > > >>>> > is
> >> > > > >>>> > > > 24
> >> > > > >>>> > > > > > > > hours,
> >> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker
> >> > guarantee
> >> > > > than
> >> > > > >>>> > setting
> >> > > > >>>> > > > any
> >> > > > >>>> > > > > > > > value
> >> > > > >>>> > > > > > > > > >> less
> >> > > > >>>> > > > > > > > > >> > > than 24 hours. By the definition of
> "max
> >> > > > >>>> compaction
> >> > > > >>>> > lag",
> >> > > > >>>> > > > we
> >> > > > >>>> > > > > > > > cannot
> >> > > > >>>> > > > > > > > > >> have
> >> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate
> >> "disable".
> >> > > > >>>> > > > > > > > > >> > >
> >> > > > >>>> > > > > > > > > >> > >
> >> > > > >>>> > > > > > > > > >> > >
> >> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> >> > > > >>>> > > > > > > > > >> > >
> >> > > > >>>> > > > > > > > > >> > >
> >> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin
> >> > McCabe <
> >> > > > >>>> > > > > > cmccabe@apache.org
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > > > > >> wrote:
> >> > > > >>>> > > > > > > > > >> > >
> >> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, Brett
> >> Rann
> >> > > > >>>> wrote:
> >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should
> >> make
> >> > 0 =
> >> > > > >>>> > disable, to
> >> > > > >>>> > > > be
> >> > > > >>>> > > > > > > > > >> consistent
> >> > > > >>>> > > > > > > > > >> > > with
> >> > > > >>>> > > > > > > > > >> > > > > the other settings.
> >> > > > >>>> > > > > > > > > >> > > > >
> >> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable
> and
> >> > when
> >> > > > >>>> seeing
> >> > > > >>>> > it
> >> > > > >>>> > > > in a
> >> > > > >>>> > > > > > > > config
> >> > > > >>>> > > > > > > > > >> it's
> >> > > > >>>> > > > > > > > > >> > > > clear
> >> > > > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0
> >> doesn't
> >> > > have
> >> > > > >>>> to
> >> > > > >>>> > mean
> >> > > > >>>> > > > > > instant,
> >> > > > >>>> > > > > > > > it
> >> > > > >>>> > > > > > > > > >> just
> >> > > > >>>> > > > > > > > > >> > > > means
> >> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't
> >> think 0
> >> > > is
> >> > > > >>>> > intuitive
> >> > > > >>>> > > > for
> >> > > > >>>> > > > > > > > > disabled
> >> > > > >>>> > > > > > > > > >> and
> >> > > > >>>> > > > > > > > > >> > > it
> >> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't aware
> >> > > > segment.ms=0
> >> > > > >>>> ==
> >> > > > >>>> > > > > > disabled,
> >> > > > >>>> > > > > > > > > but I
> >> > > > >>>> > > > > > > > > >> > > think
> >> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> >> > > > >>>> > > > > > > > > >> > > >
> >> > > > >>>> > > > > > > > > >> > > > I think there is an argument for
> >> keeping
> >> > > > these
> >> > > > >>>> two
> >> > > > >>>> > > > > > > > configurations
> >> > > > >>>> > > > > > > > > >> > > > consistent, since they are so
> >> similar. I
> >> > > > agree
> >> > > > >>>> that
> >> > > > >>>> > 0
> >> > > > >>>> > > > was an
> >> > > > >>>> > > > > > > > > >> > unfortunate
> >> > > > >>>> > > > > > > > > >> > > > choice.,
> >> > > > >>>> > > > > > > > > >> > > >
> >> > > > >>>> > > > > > > > > >> > > > best,
> >> > > > >>>> > > > > > > > > >> > > > Colin
> >> > > > >>>> > > > > > > > > >> > > >
> >> > > > >>>> > > > > > > > > >> > > > >
> >> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM
> >> Colin
> >> > > > McCabe
> >> > > > >>>> <
> >> > > > >>>> > > > > > > > > cmccabe@apache.org>
> >> > > > >>>> > > > > > > > > >> > > wrote:
> >> > > > >>>> > > > > > > > > >> > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47,
> >> > xiongqi
> >> > > wu
> >> > > > >>>> wrote:
> >> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> >> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> >> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply below.
> >> > > > >>>> > > > > > > > > >> > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> >> > > > >>>> > > > > > > > > >> > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 PM
> >> > Colin
> >> > > > >>>> McCabe <
> >> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> >> > > > >>>> > > > > > > > > >> > > > wrote:
> >> > > > >>>> > > > > > > > > >> > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit
> >> ambiguous.
> >> > Our
> >> > > > >>>> > compaction
> >> > > > >>>> > > > > > > policies
> >> > > > >>>> > > > > > > > > are
> >> > > > >>>> > > > > > > > > >> > > already
> >> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It
> >> seems
> >> > > like
> >> > > > >>>> this
> >> > > > >>>> > > > change is
> >> > > > >>>> > > > > > > > > focused
> >> > > > >>>> > > > > > > > > >> > > around
> >> > > > >>>> > > > > > > > > >> > > > > > adding
> >> > > > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms."
> >> > Perhaps
> >> > > > >>>> the KIP
> >> > > > >>>> > > > title
> >> > > > >>>> > > > > > > should
> >> > > > >>>> > > > > > > > > be
> >> > > > >>>> > > > > > > > > >> > > > something
> >> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum compaction
> >> lag
> >> > > > time"?
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will
> >> change
> >> > the
> >> > > > >>>> title.
> >> > > > >>>> > > > > > > > > >> > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is forced
> >> to
> >> > > roll
> >> > > > >>>> when
> >> > > > >>>> > > > either "
> >> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> >> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (
> >> log.roll.ms
> >> > and
> >> > > > >>>> > > > log.roll.hours)
> >> > > > >>>> > > > > > > has
> >> > > > >>>> > > > > > > > > >> > reached.
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > If the
> max.compaction.lag.ms
> >> is
> >> > > low,
> >> > > > >>>> it
> >> > > > >>>> > seems
> >> > > > >>>> > > > like
> >> > > > >>>> > > > > > > > > segments
> >> > > > >>>> > > > > > > > > >> > will
> >> > > > >>>> > > > > > > > > >> > > > be
> >> > > > >>>> > > > > > > > > >> > > > > > > > rolled very frequently. This
> >> can
> >> > > be a
> >> > > > >>>> > source of
> >> > > > >>>> > > > > > > problems
> >> > > > >>>> > > > > > > > > in
> >> > > > >>>> > > > > > > > > >> the
> >> > > > >>>> > > > > > > > > >> > > > > > cluster,
> >> > > > >>>> > > > > > > > > >> > > > > > > > since creating many
> different
> >> > small
> >> > > > log
> >> > > > >>>> > segments
> >> > > > >>>> > > > > > > > consumes
> >> > > > >>>> > > > > > > > > a
> >> > > > >>>> > > > > > > > > >> > huge
> >> > > > >>>> > > > > > > > > >> > > > > > amount of
> >> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources.
> Therefore,
> >> I
> >> > > would
> >> > > > >>>> > suggest
> >> > > > >>>> > > > > > adding a
> >> > > > >>>> > > > > > > > > >> > > broker-level
> >> > > > >>>> > > > > > > > > >> > > > > > > > configuration which allows
> us
> >> to
> >> > > set
> >> > > > a
> >> > > > >>>> > minimum
> >> > > > >>>> > > > value
> >> > > > >>>> > > > > > > for
> >> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If
> we
> >> let
> >> > > > >>>> users set
> >> > > > >>>> > it
> >> > > > >>>> > > > on a
> >> > > > >>>> > > > > > > > > >> per-topic
> >> > > > >>>> > > > > > > > > >> > > > basis,
> >> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a value of
> >> 1 ms
> >> > > or
> >> > > > >>>> > something,
> >> > > > >>>> > > > and
> >> > > > >>>> > > > > > > > cause
> >> > > > >>>> > > > > > > > > >> > chaos.
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > =========> this applies to
> >> > > > segment.ms
> >> > > > >>>> as
> >> > > > >>>> > well.
> >> > > > >>>> > > > > > Today
> >> > > > >>>> > > > > > > > > users
> >> > > > >>>> > > > > > > > > >> can
> >> > > > >>>> > > > > > > > > >> > > > set "
> >> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low
> >> value,
> >> > and
> >> > > > >>>> cause a
> >> > > > >>>> > > > frequent
> >> > > > >>>> > > > > > > > > >> rolling of
> >> > > > >>>> > > > > > > > > >> > > > active
> >> > > > >>>> > > > > > > > > >> > > > > > > segments.
> >> > > > >>>> > > > > > > > > >> > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> >> > > > >>>> > > > > > > > > >> > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an existing
> >> > > problem
> >> > > > >>>> with
> >> > > > >>>> > > > > > segment.ms.
> >> > > > >>>> > > > > > > > > >> However,
> >> > > > >>>> > > > > > > > > >> > > > that
> >> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't
> fix
> >> > it.
> >> > > As
> >> > > > >>>> you
> >> > > > >>>> > noted,
> >> > > > >>>> > > > > > there
> >> > > > >>>> > > > > > > > > will
> >> > > > >>>> > > > > > > > > >> be
> >> > > > >>>> > > > > > > > > >> > > more
> >> > > > >>>> > > > > > > > > >> > > > > > interest in these topic-level
> >> > retention
> >> > > > >>>> > settings as
> >> > > > >>>> > > > a
> >> > > > >>>> > > > > > > result
> >> > > > >>>> > > > > > > > > of
> >> > > > >>>> > > > > > > > > >> > GDPR.
> >> > > > >>>> > > > > > > > > >> > > > It
> >> > > > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing
> >> > problems
> >> > > > >>>> will
> >> > > > >>>> > cause
> >> > > > >>>> > > > more
> >> > > > >>>> > > > > > > > > trouble.
> >> > > > >>>> > > > > > > > > >> > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> >> > > straightforward
> >> > > > >>>> here --
> >> > > > >>>> > > > add a
> >> > > > >>>> > > > > > > > > >> broker-level
> >> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that
> overrides
> >> > > > >>>> per-topic
> >> > > > >>>> > > > minimums.
> >> > > > >>>> > > > > > We
> >> > > > >>>> > > > > > > > can
> >> > > > >>>> > > > > > > > > >> also
> >> > > > >>>> > > > > > > > > >> > > fail
> >> > > > >>>> > > > > > > > > >> > > > > > with a helpful error message
> when
> >> > > someone
> >> > > > >>>> > attempts
> >> > > > >>>> > > > to
> >> > > > >>>> > > > > > set
> >> > > > >>>> > > > > > > an
> >> > > > >>>> > > > > > > > > >> > invalid
> >> > > > >>>> > > > > > > > > >> > > > > > configuration.
> >> > > > >>>> > > > > > > > > >> > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > In my option, the minimum of "
> >> > > > >>>> > > > max.compaction.lag.ms"
> >> > > > >>>> > > > > > > > should
> >> > > > >>>> > > > > > > > > >> be
> >> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of "
> >> > segment.ms
> >> > > ".
> >> > > > >>>> Since
> >> > > > >>>> > > > today the
> >> > > > >>>> > > > > > > > > >> minimum of
> >> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> >> > > > >>>> > > > > > > > > >> > > > > > > is 1, "max.compaction.lag.ms"
> >> also
> >> > > > >>>> starts
> >> > > > >>>> > with
> >> > > > >>>> > > > 1. "0"
> >> > > > >>>> > > > > > > > means
> >> > > > >>>> > > > > > > > > >> > > > disable. I
> >> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but it
> is
> >> > hard
> >> > > > to
> >> > > > >>>> > define
> >> > > > >>>> > > > the
> >> > > > >>>> > > > > > > > meaning
> >> > > > >>>> > > > > > > > > >> of 0
> >> > > > >>>> > > > > > > > > >> > > > because
> >> > > > >>>> > > > > > > > > >> > > > > > we
> >> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the active
> >> segment
> >> > > > >>>> > immediately.
> >> > > > >>>> > > > > > > > > >> > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should
> >> make
> >> > 0 =
> >> > > > >>>> > disable, to
> >> > > > >>>> > > > be
> >> > > > >>>> > > > > > > > > >> consistent
> >> > > > >>>> > > > > > > > > >> > > with
> >> > > > >>>> > > > > > > > > >> > > > the
> >> > > > >>>> > > > > > > > > >> > > > > > other settings.
> >> > > > >>>> > > > > > > > > >> > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > best,
> >> > > > >>>> > > > > > > > > >> > > > > > Colin
> >> > > > >>>> > > > > > > > > >> > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an alternative
> >> > > > >>>> configuration
> >> > > > >>>> > is to
> >> > > > >>>> > > > use
> >> > > > >>>> > > > > > -1
> >> > > > >>>> > > > > > > > as
> >> > > > >>>> > > > > > > > > >> > > > "disabled"
> >> > > > >>>> > > > > > > > > >> > > > > > and
> >> > > > >>>> > > > > > > > > >> > > > > > > > 0
> >> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate compaction".
> >> > > Because
> >> > > > >>>> > compaction
> >> > > > >>>> > > > lag
> >> > > > >>>> > > > > > is
> >> > > > >>>> > > > > > > > > still
> >> > > > >>>> > > > > > > > > >> > > > determined
> >> > > > >>>> > > > > > > > > >> > > > > > > > > based on
> min.compaction.lag
> >> and
> >> > > how
> >> > > > >>>> long
> >> > > > >>>> > to
> >> > > > >>>> > > > roll
> >> > > > >>>> > > > > > an
> >> > > > >>>> > > > > > > > > active
> >> > > > >>>> > > > > > > > > >> > > > segment,
> >> > > > >>>> > > > > > > > > >> > > > > > > > the
> >> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for compaction
> is
> >> > > > >>>> undetermined
> >> > > > >>>> > if
> >> > > > >>>> > > > we
> >> > > > >>>> > > > > > use
> >> > > > >>>> > > > > > > > "0".
> >> > > > >>>> > > > > > > > > >> On
> >> > > > >>>> > > > > > > > > >> > the
> >> > > > >>>> > > > > > > > > >> > > > other
> >> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set
> >> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> >> > > > >>>> > > > > > > > to
> >> > > > >>>> > > > > > > > > >> > achieve
> >> > > > >>>> > > > > > > > > >> > > > the
> >> > > > >>>> > > > > > > > > >> > > > > > > > same
> >> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose
> "0"
> >> as
> >> > > > >>>> "disabled".
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be the
> >> > invalid
> >> > > > >>>> setting.
> >> > > > >>>> > > > > > Treating
> >> > > > >>>> > > > > > > 0
> >> > > > >>>> > > > > > > > > >> > > differently
> >> > > > >>>> > > > > > > > > >> > > > > > than
> >> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous
> >> comment, I
> >> > > am
> >> > > > >>>> not
> >> > > > >>>> > > > strongly
> >> > > > >>>> > > > > > > > against,
> >> > > > >>>> > > > > > > > > >> but
> >> > > > >>>> > > > > > > > > >> > 0
> >> > > > >>>> > > > > > > > > >> > > is
> >> > > > >>>> > > > > > > > > >> > > > > > not a
> >> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in my
> >> option.
> >> > So
> >> > > I
> >> > > > >>>> use
> >> > > > >>>> > "0" as
> >> > > > >>>> > > > > > > disabled
> >> > > > >>>> > > > > > > > > >> state.
> >> > > > >>>> > > > > > > > > >> > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > best,
> >> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at
> 15:04,
> >> > > > xiongqi
> >> > > > >>>> wu
> >> > > > >>>> > wrote:
> >> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
> >> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> >> > > > >>>> > > > > > > > > >> > > > > > > > >
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>
> >> > > > >>>> > > > > > > > <
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> >>>
> >> > > > >>>> > > > > > > > > >> <
> >> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>
> >> > > > >>>> > > > > > > > <
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> >>>>
> >> > > > >>>> > > > > > > > > >> > > <
> >> > > > >>>> > > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>
> >> > > > >>>> > > > > > > > <
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> >>>
> >> > > > >>>> > > > > > > > > >> <
> >> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>
> >> > > > >>>> > > > > > > > <
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> >>>>>
> >> > > > >>>> > > > > > > > > >> > > > > > <
> >> > > > >>>> > > > > > >
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>
> >> > > > >>>> > > > > > > > <
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> >>>
> >> > > > >>>> > > > > > > > > >> <
> >> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>
> >> > > > >>>> > > > > > > > <
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> >>>>
> >> > > > >>>> > > > > > > > > >> > > <
> >> > > > >>>> > > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>
> >> > > > >>>> > > > > > > > <
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> >>>
> >> > > > >>>> > > > > > > > > >> <
> >> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>
> >> > > > >>>> > > > > > > > <
> >> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >
> >> > > > >>>> > > > > > <
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >> > >>>>>>
> >> > > > >>>> > > > > > > > > >> > > > > > > > >
> >> > > > %3A+Time-based+log+compaction+policy
> >> > > > >>>> > > > > > > > > >> > > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> >> > > > >>>> > > > > > > > > >> > > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > >
> >> > > > >>>> https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> >> > > > >>>> > > > > > > > > >> <
> https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> >> > > > >>>> > > > > > > > > >> > > <
> >> https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> >> > > > >>>> > > > > > > > > >> <
> https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> >> > > > >>>> > > > > > > > > >> > > > > > <
> >> > > > https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> >> > > > >>>> > > > > > > > > >> <
> https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> >> > > > >>>> > > > > > > > > >> > > <
> >> https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> >> > > > >>>> > > > > > > > > >> <
> https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> >> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> >> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> >> > > > >>>> > > > > > > > > >> > > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> >> > > > >>>> > > > > > > > > >> > > > > > > >
> >> > > > >>>> > > > > > > > > >> > > > > >
> >> > > > >>>> > > > > > > > > >> > > >
> >> > > > >>>> > > > > > > > > >> > >
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> > --
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> > Brett Rann
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000
> >> > > Australia
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> >> > > > >>>> > > > > > > > > >> >
> >> > > > >>>> > > > > > > > > >>
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > > --
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > > Brett Rann
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > > Zendesk International Ltd
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> >> Australia
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> >> > > > >>>> > > > > > > > > >
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > --
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > Brett Rann
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > Senior DevOps Engineer
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > Zendesk International Ltd
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> >> Australia
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> >> > > > >>>> > > > > > > > >
> >> > > > >>>> > > > > > > >
> >> > > > >>>> > > > > > >
> >> > > > >>>> > > > > >
> >> > > > >>>> > > >
> >> > > > >>>> >
> >> > > > >>>>
> >> > > > >>>
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by xiongqi wu <xi...@gmail.com>.
Hi all,
Can I have one more vote on this KIP?
Any comment is appreciated.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-354%3A+Add+a+Maximum+Log+Compaction+Lag
Xiongqi (Wesley) Wu
On Fri, Nov 9, 2018 at 7:56 PM xiongqi wu <xi...@gmail.com> wrote:
> Thanks Dong.
> I have updated the KIP.
>
> Xiongqi (Wesley) Wu
>
>
> On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com> wrote:
>
>> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
>>
>> One minor comment: it may be a bit better to clarify in the public
>> interface section that the value of the newly added metric is determined
>> based by applying that formula across all compactable segments. For
>> example:
>>
>> The maximum value of Math.max(now -
>> earliest_timestamp_in_ms_of_uncompacted_segment - max.compaction.lag.ms,
>> 0)/1000 across all compactable partitions, where the
>> max.compaction.lag.ms
>> can be overridden on per-topic basis.
>>
>>
>>
>> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com> wrote:
>>
>> > Thanks Joel.
>> > Tracking the delay at second granularity makes sense
>> > I have updated KIP.
>> >
>> > Xiongqi (Wesley) Wu
>> >
>> >
>> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com> wrote:
>> >
>> > > +1 with one suggestion on the proposed metric. You should probably
>> > include
>> > > the unit. So for e.g., max-compaction-delay-secs.
>> > >
>> > > Joel
>> > >
>> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xi...@gmail.com>
>> wrote:
>> > >
>> > > > bump
>> > > > Xiongqi (Wesley) Wu
>> > > >
>> > > >
>> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xi...@gmail.com>
>> > wrote:
>> > > >
>> > > > >
>> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for
>> feedback.
>> > > > > Can I have more feedback or VOTE on this KIP?
>> > > > >
>> > > > >
>> > > > > Xiongqi (Wesley) Wu
>> > > > >
>> > > > >
>> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <xi...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > >> Any other votes or comments?
>> > > > >>
>> > > > >> Xiongqi (Wesley) Wu
>> > > > >>
>> > > > >>
>> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <xiongqiwu@gmail.com
>> >
>> > > > wrote:
>> > > > >>
>> > > > >>> Yes, more votes and code review.
>> > > > >>>
>> > > > >>> Xiongqi (Wesley) Wu
>> > > > >>>
>> > > > >>>
>> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
>> > > <brann@zendesk.com.invalid
>> > > > >
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
>> > > > >>>>
>> > > > >>>> Where do we go from here? More votes?
>> > > > >>>>
>> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
>> cmccabe@apache.org>
>> > > > >>>> wrote:
>> > > > >>>>
>> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
>> > > > >>>> > > Thank you for comments. I will use '0' for now.
>> > > > >>>> > >
>> > > > >>>> > > If we create topics through admin client in the future, we
>> > might
>> > > > >>>> perform
>> > > > >>>> > > some useful checks. (but the assumption is all brokers in
>> the
>> > > same
>> > > > >>>> > cluster
>> > > > >>>> > > have the same default configurations value, otherwise,it
>> might
>> > > > >>>> still be
>> > > > >>>> > > tricky to do such cross validation check.)
>> > > > >>>> >
>> > > > >>>> > This isn't something that we might do in the future-- this is
>> > > > >>>> something we
>> > > > >>>> > are doing now. We already have Create Topic policies which
>> are
>> > > > >>>> enforced by
>> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details. This is
>> one
>> > of
>> > > > the
>> > > > >>>> > motivations for getting rid of direct ZK access-- making sure
>> > that
>> > > > >>>> these
>> > > > >>>> > policies are applied.
>> > > > >>>> >
>> > > > >>>> > I agree that having different configurations on different
>> > brokers
>> > > > can
>> > > > >>>> be
>> > > > >>>> > confusing and frustrating . That's why more configurations
>> are
>> > > being
>> > > > >>>> made
>> > > > >>>> > dynamic using KIP-226. Dynamic configurations are stored
>> > centrally
>> > > > in
>> > > > >>>> ZK,
>> > > > >>>> > so they are the same on all brokers (modulo propagation
>> delays).
>> > > In
>> > > > >>>> any
>> > > > >>>> > case, this is a general issue, not specific to "create
>> topics".
>> > > > >>>> >
>> > > > >>>> > cheers,
>> > > > >>>> > Colin
>> > > > >>>> >
>> > > > >>>> >
>> > > > >>>> > >
>> > > > >>>> > >
>> > > > >>>> > > Xiongqi (Wesley) Wu
>> > > > >>>> > >
>> > > > >>>> > >
>> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
>> > > cmccabe@apache.org
>> > > > >
>> > > > >>>> > wrote:
>> > > > >>>> > >
>> > > > >>>> > > > I don't have a strong opinion. But I think we should
>> > probably
>> > > be
>> > > > >>>> > > > consistent with how segment.ms works, and just use 0.
>> > > > >>>> > > >
>> > > > >>>> > > > best,
>> > > > >>>> > > > Colin
>> > > > >>>> > > >
>> > > > >>>> > > >
>> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
>> > > > >>>> > > > > OK thanks for that clarification. I see why you're
>> > > > uncomfortable
>> > > > >>>> > with 0
>> > > > >>>> > > > now.
>> > > > >>>> > > > >
>> > > > >>>> > > > > I'm not really fussed. I just prefer consistency in
>> > > > >>>> configuration
>> > > > >>>> > > > options.
>> > > > >>>> > > > >
>> > > > >>>> > > > > Personally I lean towards treating 0 and 1 similarly in
>> > that
>> > > > >>>> > scenario,
>> > > > >>>> > > > > because it favours the person thinking about setting
>> the
>> > > > >>>> > configurations,
>> > > > >>>> > > > > and a person doesn't care about a 1ms edge case
>> especially
>> > > > when
>> > > > >>>> the
>> > > > >>>> > > > context
>> > > > >>>> > > > > is the true minimum is tied to the log cleaner cadence.
>> > > > >>>> > > > >
>> > > > >>>> > > > > Introducing 0 to mean "disabled" because there is some
>> > > > >>>> uniquness in
>> > > > >>>> > > > > segment.ms not being able to be set to 0, reduces
>> > > > configuration
>> > > > >>>> > > > consistency
>> > > > >>>> > > > > in favour of capturing a MS gap in an edge case that
>> > nobody
>> > > > >>>> would
>> > > > >>>> > ever
>> > > > >>>> > > > > notice. For someone to understand why everywhere else
>> -1
>> > is
>> > > > >>>> used to
>> > > > >>>> > > > > disable, but here 0 is used, they would need to learn
>> > about
>> > > > >>>> > segment.ms
>> > > > >>>> > > > > having a 1ms minimum and then after learning would
>> think
>> > > "who
>> > > > >>>> cares
>> > > > >>>> > about
>> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
>> > > > >>>> > > > >
>> > > > >>>> > > > > my 2c anyway. Will again defer to majority. Curious
>> which
>> > > way
>> > > > >>>> Colin
>> > > > >>>> > falls
>> > > > >>>> > > > > now.
>> > > > >>>> > > > >
>> > > > >>>> > > > > Don't want to spend more time on this though, It's well
>> > into
>> > > > >>>> > > > bikeshedding
>> > > > >>>> > > > > territory now. :)
>> > > > >>>> > > > >
>> > > > >>>> > > > >
>> > > > >>>> > > > >
>> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
>> > > > xiongqiwu@gmail.com>
>> > > > >>>> > wrote:
>> > > > >>>> > > > >
>> > > > >>>> > > > > > I want to honor the minimum value of segment.ms
>> (which
>> > is
>> > > > >>>> 1ms) to
>> > > > >>>> > > > force
>> > > > >>>> > > > > > roll an active segment.
>> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any value > 0,
>> the
>> > > > >>>> minimum of
>> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be used to
>> > seal
>> > > > an
>> > > > >>>> > active
>> > > > >>>> > > > > > segment.
>> > > > >>>> > > > > >
>> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the current
>> > > > >>>> implementation
>> > > > >>>> > will
>> > > > >>>> > > > > > treat it as disabled.
>> > > > >>>> > > > > >
>> > > > >>>> > > > > > It is a little bit weird to treat
>> max.compaction.lag=0
>> > the
>> > > > >>>> same as
>> > > > >>>> > > > > > max.compaction.lag=1.
>> > > > >>>> > > > > >
>> > > > >>>> > > > > > There might be a reason why we set the minimum of
>> > > > segment.ms
>> > > > >>>> to 1,
>> > > > >>>> > > > and I
>> > > > >>>> > > > > > don't want to break this assumption.
>> > > > >>>> > > > > >
>> > > > >>>> > > > > >
>> > > > >>>> > > > > >
>> > > > >>>> > > > > > Xiongqi (Wesley) Wu
>> > > > >>>> > > > > >
>> > > > >>>> > > > > >
>> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
>> > > > >>>> > <br...@zendesk.com.invalid>
>> > > > >>>> > > > > > wrote:
>> > > > >>>> > > > > >
>> > > > >>>> > > > > > > You're rolling a new segment if the condition is
>> met
>> > > > right?
>> > > > >>>> So
>> > > > >>>> > I'm
>> > > > >>>> > > > > > > struggling to understand the relevance of
>> segment.ms
>> > > > here.
>> > > > >>>> > Maybe an
>> > > > >>>> > > > > > > example
>> > > > >>>> > > > > > > would help my understanding:
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > > segment.ms=9999999
>> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
>> > > > >>>> > > > > > > max.compaction.lag.ms=1
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > > When a duplicate message comes in, after 1ms the
>> topic
>> > > > >>>> should be
>> > > > >>>> > > > eligible
>> > > > >>>> > > > > > > for compaction when the log compaction thread gets
>> > > around
>> > > > to
>> > > > >>>> > > > evaluating
>> > > > >>>> > > > > > the
>> > > > >>>> > > > > > > topic.
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > > if we have
>> > > > >>>> > > > > > > segment.ms=9999999
>> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
>> > > > >>>> > > > > > > max.compaction.lag.ms=0
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > > When a duplicate message comes in, after 0ms the
>> topic
>> > > > >>>> should be
>> > > > >>>> > > > eligible
>> > > > >>>> > > > > > > for compaction when the log compaction thread gets
>> > > around
>> > > > to
>> > > > >>>> > > > evaluating
>> > > > >>>> > > > > > the
>> > > > >>>> > > > > > > topic.
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > > In both of those cases the change would mean a new
>> > > segment
>> > > > >>>> is
>> > > > >>>> > rolled
>> > > > >>>> > > > so
>> > > > >>>> > > > > > the
>> > > > >>>> > > > > > > new message would be part of the compaction task. 0
>> > and
>> > > 1
>> > > > >>>> are
>> > > > >>>> > > > practically
>> > > > >>>> > > > > > > the same meaning since neither is providing an
>> actual
>> > > > >>>> guarantee
>> > > > >>>> > at
>> > > > >>>> > > > such
>> > > > >>>> > > > > > low
>> > > > >>>> > > > > > > MS settings, but effectively tying it to both the
>> > > > frequency
>> > > > >>>> of
>> > > > >>>> > the
>> > > > >>>> > > > log
>> > > > >>>> > > > > > > cleaner running and the priority of the given topic
>> > > being
>> > > > >>>> the
>> > > > >>>> > highest
>> > > > >>>> > > > > > > priority of all topics that are evaluated for
>> cleaning
>> > > on
>> > > > >>>> the
>> > > > >>>> > next
>> > > > >>>> > > > cycle.
>> > > > >>>> > > > > > > You've captured that nuance with careful "skipped"
>> > > wording
>> > > > >>>> in
>> > > > >>>> > the KIP
>> > > > >>>> > > > > > > here "controls
>> > > > >>>> > > > > > > the max time interval a message/segment can be
>> skipped
>> > > for
>> > > > >>>> log
>> > > > >>>> > > > > > compaction".
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > > How is 0 different to 1, practically? And how is it
>> > > > >>>> relating to
>> > > > >>>> > > > > > segment.ms
>> > > > >>>> > > > > > > ?
>> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean "use
>> > > > segment.ms
>> > > > >>>> > > > instead?" as
>> > > > >>>> > > > > > a
>> > > > >>>> > > > > > > kind of third option?
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
>> > > > >>>> xiongqiwu@gmail.com>
>> > > > >>>> > > > wrote:
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > > > > To make it clear,
>> > > > >>>> > > > > > > > I don't against using -1 as disabled, but we
>> need to
>> > > > come
>> > > > >>>> up
>> > > > >>>> > with
>> > > > >>>> > > > the
>> > > > >>>> > > > > > > > meaning of "0".
>> > > > >>>> > > > > > > > If "0" means immediate compaction, but the actual
>> > > > >>>> compaction
>> > > > >>>> > lag
>> > > > >>>> > > > will
>> > > > >>>> > > > > > be
>> > > > >>>> > > > > > > > segment.ms.
>> > > > >>>> > > > > > > > It has longer lag than setting the value to be
>> half
>> > of
>> > > > >>>> > segment.ms.
>> > > > >>>> > > > > > > > We cannot provide "0" as max compaction lag.
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > > Here are two options.
>> > > > >>>> > > > > > > > Option 1:
>> > > > >>>> > > > > > > > Keep 0 as disabled
>> > > > >>>> > > > > > > > Option 2:
>> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag =
>> segment.ms),
>> > > and
>> > > > >>>> > others.
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
>> > > > >>>> > > > <br...@zendesk.com.invalid>
>> > > > >>>> > > > > > > > wrote:
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > > > -1 is consistent as "special" with these
>> settings
>> > > for
>> > > > >>>> > example:
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > log.retention.bytes
>> > > > >>>> > > > > > > > > socket.received.buffer.bytes
>> > > > >>>> > > > > > > > > socket.send.buffer.bytes
>> > > > >>>> > > > > > > > > queued.max.request.bytes
>> > > > >>>> > > > > > > > > retention.bytes
>> > > > >>>> > > > > > > > > retention.ms
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > and acks.
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > Where it may mean no limit, use OS defaults,
>> max
>> > > > (acks),
>> > > > >>>> > etc. I
>> > > > >>>> > > > don't
>> > > > >>>> > > > > > > see
>> > > > >>>> > > > > > > > > much convention of 0 meaning those things.
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > There are some NULLs but it seems convetion
>> there
>> > is
>> > > > >>>> NULL is
>> > > > >>>> > used
>> > > > >>>> > > > > > where
>> > > > >>>> > > > > > > > > there's another setting in the hierarchy that
>> > would
>> > > be
>> > > > >>>> used
>> > > > >>>> > > > instead.
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann <
>> > > > >>>> > brann@zendesk.com>
>> > > > >>>> > > > > > wrote:
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0, then we're
>> not
>> > > > being
>> > > > >>>> > > > consistent
>> > > > >>>> > > > > > > > > > by using 0 for this new setting? I throw out
>> -1
>> > > for
>> > > > >>>> > > > consideration
>> > > > >>>> > > > > > > > > > again :)
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu <
>> > > > >>>> > > > xiongqiwu@gmail.com>
>> > > > >>>> > > > > > > > wrote:
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is merged.
>> > > > >>>> > > > > > > > > >>
>> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "
>> segment.ms"
>> > > to
>> > > > >>>> 1, we
>> > > > >>>> > > > cannot
>> > > > >>>> > > > > > set
>> > > > >>>> > > > > > > "
>> > > > >>>> > > > > > > > > >> segment.ms" to 0.
>> > > > >>>> > > > > > > > > >>
>> > > > >>>> > > > > > > > > >> I also updated the title of this KIP.
>> > > > >>>> > > > > > > > > >>
>> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
>> > > > >>>> > > > > > > > > >>
>> > > > >>>> > > > > > > > > >>
>> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett Rann
>> > > > >>>> > > > > > <brann@zendesk.com.invalid
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > > > >> wrote:
>> > > > >>>> > > > > > > > > >>
>> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm in
>> the
>> > > > >>>> minority.
>> > > > >>>> > :)
>> > > > >>>> > > > Can
>> > > > >>>> > > > > > we
>> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning
>> > disabled
>> > > > >>>> here:
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
>> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
>> > > > >>>> > > > > > <
>> https://kafka.apache.org/documentation/#brokerconfigs
>> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
>> > > > >>>> > > > > > > > <
>> > > https://kafka.apache.org/documentation/#brokerconfigs
>> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
>> > > > >>>> > > > > > <
>> https://kafka.apache.org/documentation/#brokerconfigs
>> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>
>> > > > >>>> > > > > > > > > >> <
>> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
>> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
>> > > > >>>> > > > > > <
>> https://kafka.apache.org/documentation/#brokerconfigs
>> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
>> > > > >>>> > > > > > > > <
>> > > https://kafka.apache.org/documentation/#brokerconfigs
>> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
>> > > > >>>> > > > > > <
>> https://kafka.apache.org/documentation/#brokerconfigs
>> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>> ?
>> > > > >>>> > > > > > > > > >> > And while there it would be good if
>> > segment.ms
>> > > > is
>> > > > >>>> > > > documented
>> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some
>> > hierarchy
>> > > > of
>> > > > >>>> > configs
>> > > > >>>> > > > for
>> > > > >>>> > > > > > > that
>> > > > >>>> > > > > > > > > too
>> > > > >>>> > > > > > > > > >> > if its not set and null for others means
>> > > > disabled!)
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi wu
>> <
>> > > > >>>> > > > xiongqiwu@gmail.com>
>> > > > >>>> > > > > > > > > wrote:
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate
>> > compaction,
>> > > > the
>> > > > >>>> > > > compaction
>> > > > >>>> > > > > > lag
>> > > > >>>> > > > > > > > is
>> > > > >>>> > > > > > > > > >> > > determined by segment.ms in worst
>> case. If
>> > > > >>>> segment.ms
>> > > > >>>> > is
>> > > > >>>> > > > 24
>> > > > >>>> > > > > > > > hours,
>> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker
>> > guarantee
>> > > > than
>> > > > >>>> > setting
>> > > > >>>> > > > any
>> > > > >>>> > > > > > > > value
>> > > > >>>> > > > > > > > > >> less
>> > > > >>>> > > > > > > > > >> > > than 24 hours. By the definition of "max
>> > > > >>>> compaction
>> > > > >>>> > lag",
>> > > > >>>> > > > we
>> > > > >>>> > > > > > > > cannot
>> > > > >>>> > > > > > > > > >> have
>> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate
>> "disable".
>> > > > >>>> > > > > > > > > >> > >
>> > > > >>>> > > > > > > > > >> > >
>> > > > >>>> > > > > > > > > >> > >
>> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
>> > > > >>>> > > > > > > > > >> > >
>> > > > >>>> > > > > > > > > >> > >
>> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin
>> > McCabe <
>> > > > >>>> > > > > > cmccabe@apache.org
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > > > > >> wrote:
>> > > > >>>> > > > > > > > > >> > >
>> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, Brett
>> Rann
>> > > > >>>> wrote:
>> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should
>> make
>> > 0 =
>> > > > >>>> > disable, to
>> > > > >>>> > > > be
>> > > > >>>> > > > > > > > > >> consistent
>> > > > >>>> > > > > > > > > >> > > with
>> > > > >>>> > > > > > > > > >> > > > > the other settings.
>> > > > >>>> > > > > > > > > >> > > > >
>> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable and
>> > when
>> > > > >>>> seeing
>> > > > >>>> > it
>> > > > >>>> > > > in a
>> > > > >>>> > > > > > > > config
>> > > > >>>> > > > > > > > > >> it's
>> > > > >>>> > > > > > > > > >> > > > clear
>> > > > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0
>> doesn't
>> > > have
>> > > > >>>> to
>> > > > >>>> > mean
>> > > > >>>> > > > > > instant,
>> > > > >>>> > > > > > > > it
>> > > > >>>> > > > > > > > > >> just
>> > > > >>>> > > > > > > > > >> > > > means
>> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't
>> think 0
>> > > is
>> > > > >>>> > intuitive
>> > > > >>>> > > > for
>> > > > >>>> > > > > > > > > disabled
>> > > > >>>> > > > > > > > > >> and
>> > > > >>>> > > > > > > > > >> > > it
>> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't aware
>> > > > segment.ms=0
>> > > > >>>> ==
>> > > > >>>> > > > > > disabled,
>> > > > >>>> > > > > > > > > but I
>> > > > >>>> > > > > > > > > >> > > think
>> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
>> > > > >>>> > > > > > > > > >> > > >
>> > > > >>>> > > > > > > > > >> > > > I think there is an argument for
>> keeping
>> > > > these
>> > > > >>>> two
>> > > > >>>> > > > > > > > configurations
>> > > > >>>> > > > > > > > > >> > > > consistent, since they are so
>> similar. I
>> > > > agree
>> > > > >>>> that
>> > > > >>>> > 0
>> > > > >>>> > > > was an
>> > > > >>>> > > > > > > > > >> > unfortunate
>> > > > >>>> > > > > > > > > >> > > > choice.,
>> > > > >>>> > > > > > > > > >> > > >
>> > > > >>>> > > > > > > > > >> > > > best,
>> > > > >>>> > > > > > > > > >> > > > Colin
>> > > > >>>> > > > > > > > > >> > > >
>> > > > >>>> > > > > > > > > >> > > > >
>> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM
>> Colin
>> > > > McCabe
>> > > > >>>> <
>> > > > >>>> > > > > > > > > cmccabe@apache.org>
>> > > > >>>> > > > > > > > > >> > > wrote:
>> > > > >>>> > > > > > > > > >> > > > >
>> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47,
>> > xiongqi
>> > > wu
>> > > > >>>> wrote:
>> > > > >>>> > > > > > > > > >> > > > > > > Colin,
>> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
>> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply below.
>> > > > >>>> > > > > > > > > >> > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
>> > > > >>>> > > > > > > > > >> > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 PM
>> > Colin
>> > > > >>>> McCabe <
>> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
>> > > > >>>> > > > > > > > > >> > > > wrote:
>> > > > >>>> > > > > > > > > >> > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit
>> ambiguous.
>> > Our
>> > > > >>>> > compaction
>> > > > >>>> > > > > > > policies
>> > > > >>>> > > > > > > > > are
>> > > > >>>> > > > > > > > > >> > > already
>> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It
>> seems
>> > > like
>> > > > >>>> this
>> > > > >>>> > > > change is
>> > > > >>>> > > > > > > > > focused
>> > > > >>>> > > > > > > > > >> > > around
>> > > > >>>> > > > > > > > > >> > > > > > adding
>> > > > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms."
>> > Perhaps
>> > > > >>>> the KIP
>> > > > >>>> > > > title
>> > > > >>>> > > > > > > should
>> > > > >>>> > > > > > > > > be
>> > > > >>>> > > > > > > > > >> > > > something
>> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum compaction
>> lag
>> > > > time"?
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will
>> change
>> > the
>> > > > >>>> title.
>> > > > >>>> > > > > > > > > >> > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is forced
>> to
>> > > roll
>> > > > >>>> when
>> > > > >>>> > > > either "
>> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
>> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (
>> log.roll.ms
>> > and
>> > > > >>>> > > > log.roll.hours)
>> > > > >>>> > > > > > > has
>> > > > >>>> > > > > > > > > >> > reached.
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > If the max.compaction.lag.ms
>> is
>> > > low,
>> > > > >>>> it
>> > > > >>>> > seems
>> > > > >>>> > > > like
>> > > > >>>> > > > > > > > > segments
>> > > > >>>> > > > > > > > > >> > will
>> > > > >>>> > > > > > > > > >> > > > be
>> > > > >>>> > > > > > > > > >> > > > > > > > rolled very frequently. This
>> can
>> > > be a
>> > > > >>>> > source of
>> > > > >>>> > > > > > > problems
>> > > > >>>> > > > > > > > > in
>> > > > >>>> > > > > > > > > >> the
>> > > > >>>> > > > > > > > > >> > > > > > cluster,
>> > > > >>>> > > > > > > > > >> > > > > > > > since creating many different
>> > small
>> > > > log
>> > > > >>>> > segments
>> > > > >>>> > > > > > > > consumes
>> > > > >>>> > > > > > > > > a
>> > > > >>>> > > > > > > > > >> > huge
>> > > > >>>> > > > > > > > > >> > > > > > amount of
>> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources. Therefore,
>> I
>> > > would
>> > > > >>>> > suggest
>> > > > >>>> > > > > > adding a
>> > > > >>>> > > > > > > > > >> > > broker-level
>> > > > >>>> > > > > > > > > >> > > > > > > > configuration which allows us
>> to
>> > > set
>> > > > a
>> > > > >>>> > minimum
>> > > > >>>> > > > value
>> > > > >>>> > > > > > > for
>> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If we
>> let
>> > > > >>>> users set
>> > > > >>>> > it
>> > > > >>>> > > > on a
>> > > > >>>> > > > > > > > > >> per-topic
>> > > > >>>> > > > > > > > > >> > > > basis,
>> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a value of
>> 1 ms
>> > > or
>> > > > >>>> > something,
>> > > > >>>> > > > and
>> > > > >>>> > > > > > > > cause
>> > > > >>>> > > > > > > > > >> > chaos.
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > =========> this applies to
>> > > > segment.ms
>> > > > >>>> as
>> > > > >>>> > well.
>> > > > >>>> > > > > > Today
>> > > > >>>> > > > > > > > > users
>> > > > >>>> > > > > > > > > >> can
>> > > > >>>> > > > > > > > > >> > > > set "
>> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low
>> value,
>> > and
>> > > > >>>> cause a
>> > > > >>>> > > > frequent
>> > > > >>>> > > > > > > > > >> rolling of
>> > > > >>>> > > > > > > > > >> > > > active
>> > > > >>>> > > > > > > > > >> > > > > > > segments.
>> > > > >>>> > > > > > > > > >> > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
>> > > > >>>> > > > > > > > > >> > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an existing
>> > > problem
>> > > > >>>> with
>> > > > >>>> > > > > > segment.ms.
>> > > > >>>> > > > > > > > > >> However,
>> > > > >>>> > > > > > > > > >> > > > that
>> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't fix
>> > it.
>> > > As
>> > > > >>>> you
>> > > > >>>> > noted,
>> > > > >>>> > > > > > there
>> > > > >>>> > > > > > > > > will
>> > > > >>>> > > > > > > > > >> be
>> > > > >>>> > > > > > > > > >> > > more
>> > > > >>>> > > > > > > > > >> > > > > > interest in these topic-level
>> > retention
>> > > > >>>> > settings as
>> > > > >>>> > > > a
>> > > > >>>> > > > > > > result
>> > > > >>>> > > > > > > > > of
>> > > > >>>> > > > > > > > > >> > GDPR.
>> > > > >>>> > > > > > > > > >> > > > It
>> > > > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing
>> > problems
>> > > > >>>> will
>> > > > >>>> > cause
>> > > > >>>> > > > more
>> > > > >>>> > > > > > > > > trouble.
>> > > > >>>> > > > > > > > > >> > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
>> > > straightforward
>> > > > >>>> here --
>> > > > >>>> > > > add a
>> > > > >>>> > > > > > > > > >> broker-level
>> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that overrides
>> > > > >>>> per-topic
>> > > > >>>> > > > minimums.
>> > > > >>>> > > > > > We
>> > > > >>>> > > > > > > > can
>> > > > >>>> > > > > > > > > >> also
>> > > > >>>> > > > > > > > > >> > > fail
>> > > > >>>> > > > > > > > > >> > > > > > with a helpful error message when
>> > > someone
>> > > > >>>> > attempts
>> > > > >>>> > > > to
>> > > > >>>> > > > > > set
>> > > > >>>> > > > > > > an
>> > > > >>>> > > > > > > > > >> > invalid
>> > > > >>>> > > > > > > > > >> > > > > > configuration.
>> > > > >>>> > > > > > > > > >> > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > In my option, the minimum of "
>> > > > >>>> > > > max.compaction.lag.ms"
>> > > > >>>> > > > > > > > should
>> > > > >>>> > > > > > > > > >> be
>> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of "
>> > segment.ms
>> > > ".
>> > > > >>>> Since
>> > > > >>>> > > > today the
>> > > > >>>> > > > > > > > > >> minimum of
>> > > > >>>> > > > > > > > > >> > > > > > segment.ms
>> > > > >>>> > > > > > > > > >> > > > > > > is 1, "max.compaction.lag.ms"
>> also
>> > > > >>>> starts
>> > > > >>>> > with
>> > > > >>>> > > > 1. "0"
>> > > > >>>> > > > > > > > means
>> > > > >>>> > > > > > > > > >> > > > disable. I
>> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but it is
>> > hard
>> > > > to
>> > > > >>>> > define
>> > > > >>>> > > > the
>> > > > >>>> > > > > > > > meaning
>> > > > >>>> > > > > > > > > >> of 0
>> > > > >>>> > > > > > > > > >> > > > because
>> > > > >>>> > > > > > > > > >> > > > > > we
>> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the active
>> segment
>> > > > >>>> > immediately.
>> > > > >>>> > > > > > > > > >> > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should
>> make
>> > 0 =
>> > > > >>>> > disable, to
>> > > > >>>> > > > be
>> > > > >>>> > > > > > > > > >> consistent
>> > > > >>>> > > > > > > > > >> > > with
>> > > > >>>> > > > > > > > > >> > > > the
>> > > > >>>> > > > > > > > > >> > > > > > other settings.
>> > > > >>>> > > > > > > > > >> > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > best,
>> > > > >>>> > > > > > > > > >> > > > > > Colin
>> > > > >>>> > > > > > > > > >> > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an alternative
>> > > > >>>> configuration
>> > > > >>>> > is to
>> > > > >>>> > > > use
>> > > > >>>> > > > > > -1
>> > > > >>>> > > > > > > > as
>> > > > >>>> > > > > > > > > >> > > > "disabled"
>> > > > >>>> > > > > > > > > >> > > > > > and
>> > > > >>>> > > > > > > > > >> > > > > > > > 0
>> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate compaction".
>> > > Because
>> > > > >>>> > compaction
>> > > > >>>> > > > lag
>> > > > >>>> > > > > > is
>> > > > >>>> > > > > > > > > still
>> > > > >>>> > > > > > > > > >> > > > determined
>> > > > >>>> > > > > > > > > >> > > > > > > > > based on min.compaction.lag
>> and
>> > > how
>> > > > >>>> long
>> > > > >>>> > to
>> > > > >>>> > > > roll
>> > > > >>>> > > > > > an
>> > > > >>>> > > > > > > > > active
>> > > > >>>> > > > > > > > > >> > > > segment,
>> > > > >>>> > > > > > > > > >> > > > > > > > the
>> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for compaction is
>> > > > >>>> undetermined
>> > > > >>>> > if
>> > > > >>>> > > > we
>> > > > >>>> > > > > > use
>> > > > >>>> > > > > > > > "0".
>> > > > >>>> > > > > > > > > >> On
>> > > > >>>> > > > > > > > > >> > the
>> > > > >>>> > > > > > > > > >> > > > other
>> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set
>> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
>> > > > >>>> > > > > > > > to
>> > > > >>>> > > > > > > > > >> > achieve
>> > > > >>>> > > > > > > > > >> > > > the
>> > > > >>>> > > > > > > > > >> > > > > > > > same
>> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose "0"
>> as
>> > > > >>>> "disabled".
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be the
>> > invalid
>> > > > >>>> setting.
>> > > > >>>> > > > > > Treating
>> > > > >>>> > > > > > > 0
>> > > > >>>> > > > > > > > > >> > > differently
>> > > > >>>> > > > > > > > > >> > > > > > than
>> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous
>> comment, I
>> > > am
>> > > > >>>> not
>> > > > >>>> > > > strongly
>> > > > >>>> > > > > > > > against,
>> > > > >>>> > > > > > > > > >> but
>> > > > >>>> > > > > > > > > >> > 0
>> > > > >>>> > > > > > > > > >> > > is
>> > > > >>>> > > > > > > > > >> > > > > > not a
>> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in my
>> option.
>> > So
>> > > I
>> > > > >>>> use
>> > > > >>>> > "0" as
>> > > > >>>> > > > > > > disabled
>> > > > >>>> > > > > > > > > >> state.
>> > > > >>>> > > > > > > > > >> > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > best,
>> > > > >>>> > > > > > > > > >> > > > > > > > Colin
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at 15:04,
>> > > > xiongqi
>> > > > >>>> wu
>> > > > >>>> > wrote:
>> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
>> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
>> > > > >>>> > > > > > > > > >> > > > > > > > >
>> > > > >>>> > > > > > > > >
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>> > > > >>>> > > > > > > > <
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> >>>
>> > > > >>>> > > > > > > > > >> <
>> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>> > > > >>>> > > > > > > > <
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> >>>>
>> > > > >>>> > > > > > > > > >> > > <
>> > > > >>>> > > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>> > > > >>>> > > > > > > > <
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> >>>
>> > > > >>>> > > > > > > > > >> <
>> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>> > > > >>>> > > > > > > > <
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> >>>>>
>> > > > >>>> > > > > > > > > >> > > > > > <
>> > > > >>>> > > > > > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>> > > > >>>> > > > > > > > <
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> >>>
>> > > > >>>> > > > > > > > > >> <
>> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>> > > > >>>> > > > > > > > <
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> >>>>
>> > > > >>>> > > > > > > > > >> > > <
>> > > > >>>> > > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>> > > > >>>> > > > > > > > <
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> >>>
>> > > > >>>> > > > > > > > > >> <
>> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
>> > > > >>>> > > > > > > > <
>> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
>> > > > >>>> > > > > > <
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
>> > >>>>>>
>> > > > >>>> > > > > > > > > >> > > > > > > > >
>> > > > %3A+Time-based+log+compaction+policy
>> > > > >>>> > > > > > > > > >> > > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
>> > > > >>>> > > > > > > > > >> > > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > >
>> > > > >>>> https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
>> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
>> > > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
>> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
>> > > > >>>> > > > > > > > > >> > > <
>> https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
>> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
>> > > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
>> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
>> > > > >>>> > > > > > > > > >> > > > > > <
>> > > > https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
>> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
>> > > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
>> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
>> > > > >>>> > > > > > > > > >> > > <
>> https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
>> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
>> > > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
>> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
>> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
>> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
>> > > > >>>> > > > > > > > > >> > > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
>> > > > >>>> > > > > > > > > >> > > > > > > >
>> > > > >>>> > > > > > > > > >> > > > > >
>> > > > >>>> > > > > > > > > >> > > >
>> > > > >>>> > > > > > > > > >> > >
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> > --
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> > Brett Rann
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000
>> > > Australia
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
>> > > > >>>> > > > > > > > > >> >
>> > > > >>>> > > > > > > > > >>
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > > --
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > > Brett Rann
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > > Senior DevOps Engineer
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > > Zendesk International Ltd
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000
>> Australia
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
>> > > > >>>> > > > > > > > > >
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > --
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > Brett Rann
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > Senior DevOps Engineer
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > Zendesk International Ltd
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000
>> Australia
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
>> > > > >>>> > > > > > > > >
>> > > > >>>> > > > > > > >
>> > > > >>>> > > > > > >
>> > > > >>>> > > > > >
>> > > > >>>> > > >
>> > > > >>>> >
>> > > > >>>>
>> > > > >>>
>> > > >
>> > >
>> >
>>
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by xiongqi wu <xi...@gmail.com>.
Thanks Dong.
I have updated the KIP.
Xiongqi (Wesley) Wu
On Fri, Nov 9, 2018 at 5:31 PM Dong Lin <li...@gmail.com> wrote:
> Thanks for the KIP Xiongqi. LGTM. +1 (binding)
>
> One minor comment: it may be a bit better to clarify in the public
> interface section that the value of the newly added metric is determined
> based by applying that formula across all compactable segments. For
> example:
>
> The maximum value of Math.max(now -
> earliest_timestamp_in_ms_of_uncompacted_segment - max.compaction.lag.ms,
> 0)/1000 across all compactable partitions, where the max.compaction.lag.ms
> can be overridden on per-topic basis.
>
>
>
> On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com> wrote:
>
> > Thanks Joel.
> > Tracking the delay at second granularity makes sense
> > I have updated KIP.
> >
> > Xiongqi (Wesley) Wu
> >
> >
> > On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com> wrote:
> >
> > > +1 with one suggestion on the proposed metric. You should probably
> > include
> > > the unit. So for e.g., max-compaction-delay-secs.
> > >
> > > Joel
> > >
> > > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xi...@gmail.com> wrote:
> > >
> > > > bump
> > > > Xiongqi (Wesley) Wu
> > > >
> > > >
> > > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xi...@gmail.com>
> > wrote:
> > > >
> > > > >
> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for feedback.
> > > > > Can I have more feedback or VOTE on this KIP?
> > > > >
> > > > >
> > > > > Xiongqi (Wesley) Wu
> > > > >
> > > > >
> > > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <xi...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Any other votes or comments?
> > > > >>
> > > > >> Xiongqi (Wesley) Wu
> > > > >>
> > > > >>
> > > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <xi...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >>> Yes, more votes and code review.
> > > > >>>
> > > > >>> Xiongqi (Wesley) Wu
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > > <brann@zendesk.com.invalid
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > > > >>>>
> > > > >>>> Where do we go from here? More votes?
> > > > >>>>
> > > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <
> cmccabe@apache.org>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > > > >>>> > > Thank you for comments. I will use '0' for now.
> > > > >>>> > >
> > > > >>>> > > If we create topics through admin client in the future, we
> > might
> > > > >>>> perform
> > > > >>>> > > some useful checks. (but the assumption is all brokers in
> the
> > > same
> > > > >>>> > cluster
> > > > >>>> > > have the same default configurations value, otherwise,it
> might
> > > > >>>> still be
> > > > >>>> > > tricky to do such cross validation check.)
> > > > >>>> >
> > > > >>>> > This isn't something that we might do in the future-- this is
> > > > >>>> something we
> > > > >>>> > are doing now. We already have Create Topic policies which are
> > > > >>>> enforced by
> > > > >>>> > the broker. Check KIP-108 and KIP-170 for details. This is one
> > of
> > > > the
> > > > >>>> > motivations for getting rid of direct ZK access-- making sure
> > that
> > > > >>>> these
> > > > >>>> > policies are applied.
> > > > >>>> >
> > > > >>>> > I agree that having different configurations on different
> > brokers
> > > > can
> > > > >>>> be
> > > > >>>> > confusing and frustrating . That's why more configurations are
> > > being
> > > > >>>> made
> > > > >>>> > dynamic using KIP-226. Dynamic configurations are stored
> > centrally
> > > > in
> > > > >>>> ZK,
> > > > >>>> > so they are the same on all brokers (modulo propagation
> delays).
> > > In
> > > > >>>> any
> > > > >>>> > case, this is a general issue, not specific to "create
> topics".
> > > > >>>> >
> > > > >>>> > cheers,
> > > > >>>> > Colin
> > > > >>>> >
> > > > >>>> >
> > > > >>>> > >
> > > > >>>> > >
> > > > >>>> > > Xiongqi (Wesley) Wu
> > > > >>>> > >
> > > > >>>> > >
> > > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> > > cmccabe@apache.org
> > > > >
> > > > >>>> > wrote:
> > > > >>>> > >
> > > > >>>> > > > I don't have a strong opinion. But I think we should
> > probably
> > > be
> > > > >>>> > > > consistent with how segment.ms works, and just use 0.
> > > > >>>> > > >
> > > > >>>> > > > best,
> > > > >>>> > > > Colin
> > > > >>>> > > >
> > > > >>>> > > >
> > > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> > > > >>>> > > > > OK thanks for that clarification. I see why you're
> > > > uncomfortable
> > > > >>>> > with 0
> > > > >>>> > > > now.
> > > > >>>> > > > >
> > > > >>>> > > > > I'm not really fussed. I just prefer consistency in
> > > > >>>> configuration
> > > > >>>> > > > options.
> > > > >>>> > > > >
> > > > >>>> > > > > Personally I lean towards treating 0 and 1 similarly in
> > that
> > > > >>>> > scenario,
> > > > >>>> > > > > because it favours the person thinking about setting the
> > > > >>>> > configurations,
> > > > >>>> > > > > and a person doesn't care about a 1ms edge case
> especially
> > > > when
> > > > >>>> the
> > > > >>>> > > > context
> > > > >>>> > > > > is the true minimum is tied to the log cleaner cadence.
> > > > >>>> > > > >
> > > > >>>> > > > > Introducing 0 to mean "disabled" because there is some
> > > > >>>> uniquness in
> > > > >>>> > > > > segment.ms not being able to be set to 0, reduces
> > > > configuration
> > > > >>>> > > > consistency
> > > > >>>> > > > > in favour of capturing a MS gap in an edge case that
> > nobody
> > > > >>>> would
> > > > >>>> > ever
> > > > >>>> > > > > notice. For someone to understand why everywhere else -1
> > is
> > > > >>>> used to
> > > > >>>> > > > > disable, but here 0 is used, they would need to learn
> > about
> > > > >>>> > segment.ms
> > > > >>>> > > > > having a 1ms minimum and then after learning would think
> > > "who
> > > > >>>> cares
> > > > >>>> > about
> > > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > > > >>>> > > > >
> > > > >>>> > > > > my 2c anyway. Will again defer to majority. Curious
> which
> > > way
> > > > >>>> Colin
> > > > >>>> > falls
> > > > >>>> > > > > now.
> > > > >>>> > > > >
> > > > >>>> > > > > Don't want to spend more time on this though, It's well
> > into
> > > > >>>> > > > bikeshedding
> > > > >>>> > > > > territory now. :)
> > > > >>>> > > > >
> > > > >>>> > > > >
> > > > >>>> > > > >
> > > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > > > xiongqiwu@gmail.com>
> > > > >>>> > wrote:
> > > > >>>> > > > >
> > > > >>>> > > > > > I want to honor the minimum value of segment.ms
> (which
> > is
> > > > >>>> 1ms) to
> > > > >>>> > > > force
> > > > >>>> > > > > > roll an active segment.
> > > > >>>> > > > > > So if we set "max.compaction.lag.ms" any value > 0,
> the
> > > > >>>> minimum of
> > > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be used to
> > seal
> > > > an
> > > > >>>> > active
> > > > >>>> > > > > > segment.
> > > > >>>> > > > > >
> > > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the current
> > > > >>>> implementation
> > > > >>>> > will
> > > > >>>> > > > > > treat it as disabled.
> > > > >>>> > > > > >
> > > > >>>> > > > > > It is a little bit weird to treat max.compaction.lag=0
> > the
> > > > >>>> same as
> > > > >>>> > > > > > max.compaction.lag=1.
> > > > >>>> > > > > >
> > > > >>>> > > > > > There might be a reason why we set the minimum of
> > > > segment.ms
> > > > >>>> to 1,
> > > > >>>> > > > and I
> > > > >>>> > > > > > don't want to break this assumption.
> > > > >>>> > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > > >>>> > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > > > >>>> > <br...@zendesk.com.invalid>
> > > > >>>> > > > > > wrote:
> > > > >>>> > > > > >
> > > > >>>> > > > > > > You're rolling a new segment if the condition is met
> > > > right?
> > > > >>>> So
> > > > >>>> > I'm
> > > > >>>> > > > > > > struggling to understand the relevance of
> segment.ms
> > > > here.
> > > > >>>> > Maybe an
> > > > >>>> > > > > > > example
> > > > >>>> > > > > > > would help my understanding:
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > segment.ms=9999999
> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > >>>> > > > > > > max.compaction.lag.ms=1
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > When a duplicate message comes in, after 1ms the
> topic
> > > > >>>> should be
> > > > >>>> > > > eligible
> > > > >>>> > > > > > > for compaction when the log compaction thread gets
> > > around
> > > > to
> > > > >>>> > > > evaluating
> > > > >>>> > > > > > the
> > > > >>>> > > > > > > topic.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > if we have
> > > > >>>> > > > > > > segment.ms=9999999
> > > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > > >>>> > > > > > > max.compaction.lag.ms=0
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > When a duplicate message comes in, after 0ms the
> topic
> > > > >>>> should be
> > > > >>>> > > > eligible
> > > > >>>> > > > > > > for compaction when the log compaction thread gets
> > > around
> > > > to
> > > > >>>> > > > evaluating
> > > > >>>> > > > > > the
> > > > >>>> > > > > > > topic.
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > In both of those cases the change would mean a new
> > > segment
> > > > >>>> is
> > > > >>>> > rolled
> > > > >>>> > > > so
> > > > >>>> > > > > > the
> > > > >>>> > > > > > > new message would be part of the compaction task. 0
> > and
> > > 1
> > > > >>>> are
> > > > >>>> > > > practically
> > > > >>>> > > > > > > the same meaning since neither is providing an
> actual
> > > > >>>> guarantee
> > > > >>>> > at
> > > > >>>> > > > such
> > > > >>>> > > > > > low
> > > > >>>> > > > > > > MS settings, but effectively tying it to both the
> > > > frequency
> > > > >>>> of
> > > > >>>> > the
> > > > >>>> > > > log
> > > > >>>> > > > > > > cleaner running and the priority of the given topic
> > > being
> > > > >>>> the
> > > > >>>> > highest
> > > > >>>> > > > > > > priority of all topics that are evaluated for
> cleaning
> > > on
> > > > >>>> the
> > > > >>>> > next
> > > > >>>> > > > cycle.
> > > > >>>> > > > > > > You've captured that nuance with careful "skipped"
> > > wording
> > > > >>>> in
> > > > >>>> > the KIP
> > > > >>>> > > > > > > here "controls
> > > > >>>> > > > > > > the max time interval a message/segment can be
> skipped
> > > for
> > > > >>>> log
> > > > >>>> > > > > > compaction".
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > How is 0 different to 1, practically? And how is it
> > > > >>>> relating to
> > > > >>>> > > > > > segment.ms
> > > > >>>> > > > > > > ?
> > > > >>>> > > > > > > Is it that you're proposing to have 0 mean "use
> > > > segment.ms
> > > > >>>> > > > instead?" as
> > > > >>>> > > > > > a
> > > > >>>> > > > > > > kind of third option?
> > > > >>>> > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
> > > > >>>> xiongqiwu@gmail.com>
> > > > >>>> > > > wrote:
> > > > >>>> > > > > > >
> > > > >>>> > > > > > > > To make it clear,
> > > > >>>> > > > > > > > I don't against using -1 as disabled, but we need
> to
> > > > come
> > > > >>>> up
> > > > >>>> > with
> > > > >>>> > > > the
> > > > >>>> > > > > > > > meaning of "0".
> > > > >>>> > > > > > > > If "0" means immediate compaction, but the actual
> > > > >>>> compaction
> > > > >>>> > lag
> > > > >>>> > > > will
> > > > >>>> > > > > > be
> > > > >>>> > > > > > > > segment.ms.
> > > > >>>> > > > > > > > It has longer lag than setting the value to be
> half
> > of
> > > > >>>> > segment.ms.
> > > > >>>> > > > > > > > We cannot provide "0" as max compaction lag.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Here are two options.
> > > > >>>> > > > > > > > Option 1:
> > > > >>>> > > > > > > > Keep 0 as disabled
> > > > >>>> > > > > > > > Option 2:
> > > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag = segment.ms
> ),
> > > and
> > > > >>>> > others.
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> > > > >>>> > > > <br...@zendesk.com.invalid>
> > > > >>>> > > > > > > > wrote:
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > > -1 is consistent as "special" with these
> settings
> > > for
> > > > >>>> > example:
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > log.retention.bytes
> > > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > > >>>> > > > > > > > > queued.max.request.bytes
> > > > >>>> > > > > > > > > retention.bytes
> > > > >>>> > > > > > > > > retention.ms
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > and acks.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Where it may mean no limit, use OS defaults, max
> > > > (acks),
> > > > >>>> > etc. I
> > > > >>>> > > > don't
> > > > >>>> > > > > > > see
> > > > >>>> > > > > > > > > much convention of 0 meaning those things.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > There are some NULLs but it seems convetion
> there
> > is
> > > > >>>> NULL is
> > > > >>>> > used
> > > > >>>> > > > > > where
> > > > >>>> > > > > > > > > there's another setting in the hierarchy that
> > would
> > > be
> > > > >>>> used
> > > > >>>> > > > instead.
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann <
> > > > >>>> > brann@zendesk.com>
> > > > >>>> > > > > > wrote:
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > > If segment.ms can't be set to 0, then we're
> not
> > > > being
> > > > >>>> > > > consistent
> > > > >>>> > > > > > > > > > by using 0 for this new setting? I throw out
> -1
> > > for
> > > > >>>> > > > consideration
> > > > >>>> > > > > > > > > > again :)
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu <
> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > >>>> > > > > > > > wrote:
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >> Thanks. I will document after PR is merged.
> > > > >>>> > > > > > > > > >>
> > > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "
> segment.ms"
> > > to
> > > > >>>> 1, we
> > > > >>>> > > > cannot
> > > > >>>> > > > > > set
> > > > >>>> > > > > > > "
> > > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > > >>>> > > > > > > > > >>
> > > > >>>> > > > > > > > > >> I also updated the title of this KIP.
> > > > >>>> > > > > > > > > >>
> > > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > > >>>> > > > > > > > > >>
> > > > >>>> > > > > > > > > >>
> > > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett Rann
> > > > >>>> > > > > > <brann@zendesk.com.invalid
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > > >> wrote:
> > > > >>>> > > > > > > > > >>
> > > > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm in
> the
> > > > >>>> minority.
> > > > >>>> > :)
> > > > >>>> > > > Can
> > > > >>>> > > > > > we
> > > > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning
> > disabled
> > > > >>>> here:
> > > > >>>> > > > > > > > > >> >
> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >>>> > > > > > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > >>>> > > > > > > > <
> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >>>> > > > > > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>
> > > > >>>> > > > > > > > > >> <
> > > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >>>> > > > > > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > > > >>>> > > > > > > > <
> > > https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > > >>>> > > > > > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>> ?
> > > > >>>> > > > > > > > > >> > And while there it would be good if
> > segment.ms
> > > > is
> > > > >>>> > > > documented
> > > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some
> > hierarchy
> > > > of
> > > > >>>> > configs
> > > > >>>> > > > for
> > > > >>>> > > > > > > that
> > > > >>>> > > > > > > > > too
> > > > >>>> > > > > > > > > >> > if its not set and null for others means
> > > > disabled!)
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi wu <
> > > > >>>> > > > xiongqiwu@gmail.com>
> > > > >>>> > > > > > > > > wrote:
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate
> > compaction,
> > > > the
> > > > >>>> > > > compaction
> > > > >>>> > > > > > lag
> > > > >>>> > > > > > > > is
> > > > >>>> > > > > > > > > >> > > determined by segment.ms in worst case.
> If
> > > > >>>> segment.ms
> > > > >>>> > is
> > > > >>>> > > > 24
> > > > >>>> > > > > > > > hours,
> > > > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker
> > guarantee
> > > > than
> > > > >>>> > setting
> > > > >>>> > > > any
> > > > >>>> > > > > > > > value
> > > > >>>> > > > > > > > > >> less
> > > > >>>> > > > > > > > > >> > > than 24 hours. By the definition of "max
> > > > >>>> compaction
> > > > >>>> > lag",
> > > > >>>> > > > we
> > > > >>>> > > > > > > > cannot
> > > > >>>> > > > > > > > > >> have
> > > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate
> "disable".
> > > > >>>> > > > > > > > > >> > >
> > > > >>>> > > > > > > > > >> > >
> > > > >>>> > > > > > > > > >> > >
> > > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > > >>>> > > > > > > > > >> > >
> > > > >>>> > > > > > > > > >> > >
> > > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin
> > McCabe <
> > > > >>>> > > > > > cmccabe@apache.org
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > > > > >> wrote:
> > > > >>>> > > > > > > > > >> > >
> > > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, Brett
> Rann
> > > > >>>> wrote:
> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should make
> > 0 =
> > > > >>>> > disable, to
> > > > >>>> > > > be
> > > > >>>> > > > > > > > > >> consistent
> > > > >>>> > > > > > > > > >> > > with
> > > > >>>> > > > > > > > > >> > > > > the other settings.
> > > > >>>> > > > > > > > > >> > > > >
> > > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable and
> > when
> > > > >>>> seeing
> > > > >>>> > it
> > > > >>>> > > > in a
> > > > >>>> > > > > > > > config
> > > > >>>> > > > > > > > > >> it's
> > > > >>>> > > > > > > > > >> > > > clear
> > > > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0
> doesn't
> > > have
> > > > >>>> to
> > > > >>>> > mean
> > > > >>>> > > > > > instant,
> > > > >>>> > > > > > > > it
> > > > >>>> > > > > > > > > >> just
> > > > >>>> > > > > > > > > >> > > > means
> > > > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't
> think 0
> > > is
> > > > >>>> > intuitive
> > > > >>>> > > > for
> > > > >>>> > > > > > > > > disabled
> > > > >>>> > > > > > > > > >> and
> > > > >>>> > > > > > > > > >> > > it
> > > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't aware
> > > > segment.ms=0
> > > > >>>> ==
> > > > >>>> > > > > > disabled,
> > > > >>>> > > > > > > > > but I
> > > > >>>> > > > > > > > > >> > > think
> > > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> > > > >>>> > > > > > > > > >> > > >
> > > > >>>> > > > > > > > > >> > > > I think there is an argument for
> keeping
> > > > these
> > > > >>>> two
> > > > >>>> > > > > > > > configurations
> > > > >>>> > > > > > > > > >> > > > consistent, since they are so similar.
> I
> > > > agree
> > > > >>>> that
> > > > >>>> > 0
> > > > >>>> > > > was an
> > > > >>>> > > > > > > > > >> > unfortunate
> > > > >>>> > > > > > > > > >> > > > choice.,
> > > > >>>> > > > > > > > > >> > > >
> > > > >>>> > > > > > > > > >> > > > best,
> > > > >>>> > > > > > > > > >> > > > Colin
> > > > >>>> > > > > > > > > >> > > >
> > > > >>>> > > > > > > > > >> > > > >
> > > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM Colin
> > > > McCabe
> > > > >>>> <
> > > > >>>> > > > > > > > > cmccabe@apache.org>
> > > > >>>> > > > > > > > > >> > > wrote:
> > > > >>>> > > > > > > > > >> > > > >
> > > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47,
> > xiongqi
> > > wu
> > > > >>>> wrote:
> > > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> > > > >>>> > > > > > > > > >> > > > > > > see my inline reply below.
> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 PM
> > Colin
> > > > >>>> McCabe <
> > > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > > >>>> > > > > > > > > >> > > > wrote:
> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit ambiguous.
> > Our
> > > > >>>> > compaction
> > > > >>>> > > > > > > policies
> > > > >>>> > > > > > > > > are
> > > > >>>> > > > > > > > > >> > > already
> > > > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It seems
> > > like
> > > > >>>> this
> > > > >>>> > > > change is
> > > > >>>> > > > > > > > > focused
> > > > >>>> > > > > > > > > >> > > around
> > > > >>>> > > > > > > > > >> > > > > > adding
> > > > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms."
> > Perhaps
> > > > >>>> the KIP
> > > > >>>> > > > title
> > > > >>>> > > > > > > should
> > > > >>>> > > > > > > > > be
> > > > >>>> > > > > > > > > >> > > > something
> > > > >>>> > > > > > > > > >> > > > > > > > like "add maximum compaction
> lag
> > > > time"?
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will change
> > the
> > > > >>>> title.
> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > The active segment is forced to
> > > roll
> > > > >>>> when
> > > > >>>> > > > either "
> > > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> > > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (log.roll.ms
> > and
> > > > >>>> > > > log.roll.hours)
> > > > >>>> > > > > > > has
> > > > >>>> > > > > > > > > >> > reached.
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > If the max.compaction.lag.ms
> is
> > > low,
> > > > >>>> it
> > > > >>>> > seems
> > > > >>>> > > > like
> > > > >>>> > > > > > > > > segments
> > > > >>>> > > > > > > > > >> > will
> > > > >>>> > > > > > > > > >> > > > be
> > > > >>>> > > > > > > > > >> > > > > > > > rolled very frequently. This
> can
> > > be a
> > > > >>>> > source of
> > > > >>>> > > > > > > problems
> > > > >>>> > > > > > > > > in
> > > > >>>> > > > > > > > > >> the
> > > > >>>> > > > > > > > > >> > > > > > cluster,
> > > > >>>> > > > > > > > > >> > > > > > > > since creating many different
> > small
> > > > log
> > > > >>>> > segments
> > > > >>>> > > > > > > > consumes
> > > > >>>> > > > > > > > > a
> > > > >>>> > > > > > > > > >> > huge
> > > > >>>> > > > > > > > > >> > > > > > amount of
> > > > >>>> > > > > > > > > >> > > > > > > > cluster resources. Therefore, I
> > > would
> > > > >>>> > suggest
> > > > >>>> > > > > > adding a
> > > > >>>> > > > > > > > > >> > > broker-level
> > > > >>>> > > > > > > > > >> > > > > > > > configuration which allows us
> to
> > > set
> > > > a
> > > > >>>> > minimum
> > > > >>>> > > > value
> > > > >>>> > > > > > > for
> > > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If we
> let
> > > > >>>> users set
> > > > >>>> > it
> > > > >>>> > > > on a
> > > > >>>> > > > > > > > > >> per-topic
> > > > >>>> > > > > > > > > >> > > > basis,
> > > > >>>> > > > > > > > > >> > > > > > > > someone could set a value of 1
> ms
> > > or
> > > > >>>> > something,
> > > > >>>> > > > and
> > > > >>>> > > > > > > > cause
> > > > >>>> > > > > > > > > >> > chaos.
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > =========> this applies to
> > > > segment.ms
> > > > >>>> as
> > > > >>>> > well.
> > > > >>>> > > > > > Today
> > > > >>>> > > > > > > > > users
> > > > >>>> > > > > > > > > >> can
> > > > >>>> > > > > > > > > >> > > > set "
> > > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low value,
> > and
> > > > >>>> cause a
> > > > >>>> > > > frequent
> > > > >>>> > > > > > > > > >> rolling of
> > > > >>>> > > > > > > > > >> > > > active
> > > > >>>> > > > > > > > > >> > > > > > > segments.
> > > > >>>> > > > > > > > > >> > > > > >
> > > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > > >>>> > > > > > > > > >> > > > > >
> > > > >>>> > > > > > > > > >> > > > > > I agree that this is an existing
> > > problem
> > > > >>>> with
> > > > >>>> > > > > > segment.ms.
> > > > >>>> > > > > > > > > >> However,
> > > > >>>> > > > > > > > > >> > > > that
> > > > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't fix
> > it.
> > > As
> > > > >>>> you
> > > > >>>> > noted,
> > > > >>>> > > > > > there
> > > > >>>> > > > > > > > > will
> > > > >>>> > > > > > > > > >> be
> > > > >>>> > > > > > > > > >> > > more
> > > > >>>> > > > > > > > > >> > > > > > interest in these topic-level
> > retention
> > > > >>>> > settings as
> > > > >>>> > > > a
> > > > >>>> > > > > > > result
> > > > >>>> > > > > > > > > of
> > > > >>>> > > > > > > > > >> > GDPR.
> > > > >>>> > > > > > > > > >> > > > It
> > > > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing
> > problems
> > > > >>>> will
> > > > >>>> > cause
> > > > >>>> > > > more
> > > > >>>> > > > > > > > > trouble.
> > > > >>>> > > > > > > > > >> > > > > >
> > > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> > > straightforward
> > > > >>>> here --
> > > > >>>> > > > add a
> > > > >>>> > > > > > > > > >> broker-level
> > > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that overrides
> > > > >>>> per-topic
> > > > >>>> > > > minimums.
> > > > >>>> > > > > > We
> > > > >>>> > > > > > > > can
> > > > >>>> > > > > > > > > >> also
> > > > >>>> > > > > > > > > >> > > fail
> > > > >>>> > > > > > > > > >> > > > > > with a helpful error message when
> > > someone
> > > > >>>> > attempts
> > > > >>>> > > > to
> > > > >>>> > > > > > set
> > > > >>>> > > > > > > an
> > > > >>>> > > > > > > > > >> > invalid
> > > > >>>> > > > > > > > > >> > > > > > configuration.
> > > > >>>> > > > > > > > > >> > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > In my option, the minimum of "
> > > > >>>> > > > max.compaction.lag.ms"
> > > > >>>> > > > > > > > should
> > > > >>>> > > > > > > > > >> be
> > > > >>>> > > > > > > > > >> > > > > > > based on the minimum of "
> > segment.ms
> > > ".
> > > > >>>> Since
> > > > >>>> > > > today the
> > > > >>>> > > > > > > > > >> minimum of
> > > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > > >>>> > > > > > > > > >> > > > > > > is 1, "max.compaction.lag.ms"
> also
> > > > >>>> starts
> > > > >>>> > with
> > > > >>>> > > > 1. "0"
> > > > >>>> > > > > > > > means
> > > > >>>> > > > > > > > > >> > > > disable. I
> > > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but it is
> > hard
> > > > to
> > > > >>>> > define
> > > > >>>> > > > the
> > > > >>>> > > > > > > > meaning
> > > > >>>> > > > > > > > > >> of 0
> > > > >>>> > > > > > > > > >> > > > because
> > > > >>>> > > > > > > > > >> > > > > > we
> > > > >>>> > > > > > > > > >> > > > > > > cannot just roll the active
> segment
> > > > >>>> > immediately.
> > > > >>>> > > > > > > > > >> > > > > >
> > > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should make
> > 0 =
> > > > >>>> > disable, to
> > > > >>>> > > > be
> > > > >>>> > > > > > > > > >> consistent
> > > > >>>> > > > > > > > > >> > > with
> > > > >>>> > > > > > > > > >> > > > the
> > > > >>>> > > > > > > > > >> > > > > > other settings.
> > > > >>>> > > > > > > > > >> > > > > >
> > > > >>>> > > > > > > > > >> > > > > > best,
> > > > >>>> > > > > > > > > >> > > > > > Colin
> > > > >>>> > > > > > > > > >> > > > > >
> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > -- Note that an alternative
> > > > >>>> configuration
> > > > >>>> > is to
> > > > >>>> > > > use
> > > > >>>> > > > > > -1
> > > > >>>> > > > > > > > as
> > > > >>>> > > > > > > > > >> > > > "disabled"
> > > > >>>> > > > > > > > > >> > > > > > and
> > > > >>>> > > > > > > > > >> > > > > > > > 0
> > > > >>>> > > > > > > > > >> > > > > > > > > as "immediate compaction".
> > > Because
> > > > >>>> > compaction
> > > > >>>> > > > lag
> > > > >>>> > > > > > is
> > > > >>>> > > > > > > > > still
> > > > >>>> > > > > > > > > >> > > > determined
> > > > >>>> > > > > > > > > >> > > > > > > > > based on min.compaction.lag
> and
> > > how
> > > > >>>> long
> > > > >>>> > to
> > > > >>>> > > > roll
> > > > >>>> > > > > > an
> > > > >>>> > > > > > > > > active
> > > > >>>> > > > > > > > > >> > > > segment,
> > > > >>>> > > > > > > > > >> > > > > > > > the
> > > > >>>> > > > > > > > > >> > > > > > > > > actual lag for compaction is
> > > > >>>> undetermined
> > > > >>>> > if
> > > > >>>> > > > we
> > > > >>>> > > > > > use
> > > > >>>> > > > > > > > "0".
> > > > >>>> > > > > > > > > >> On
> > > > >>>> > > > > > > > > >> > the
> > > > >>>> > > > > > > > > >> > > > other
> > > > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set
> > > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > > >>>> > > > > > > > to
> > > > >>>> > > > > > > > > >> > achieve
> > > > >>>> > > > > > > > > >> > > > the
> > > > >>>> > > > > > > > > >> > > > > > > > same
> > > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose "0"
> as
> > > > >>>> "disabled".
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be the
> > invalid
> > > > >>>> setting.
> > > > >>>> > > > > > Treating
> > > > >>>> > > > > > > 0
> > > > >>>> > > > > > > > > >> > > differently
> > > > >>>> > > > > > > > > >> > > > > > than
> > > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous
> comment, I
> > > am
> > > > >>>> not
> > > > >>>> > > > strongly
> > > > >>>> > > > > > > > against,
> > > > >>>> > > > > > > > > >> but
> > > > >>>> > > > > > > > > >> > 0
> > > > >>>> > > > > > > > > >> > > is
> > > > >>>> > > > > > > > > >> > > > > > not a
> > > > >>>> > > > > > > > > >> > > > > > > valid configuration in my option.
> > So
> > > I
> > > > >>>> use
> > > > >>>> > "0" as
> > > > >>>> > > > > > > disabled
> > > > >>>> > > > > > > > > >> state.
> > > > >>>> > > > > > > > > >> > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > best,
> > > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at 15:04,
> > > > xiongqi
> > > > >>>> wu
> > > > >>>> > wrote:
> > > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
> > > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > > >>>> > > > > > > > <
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > > > >>>> > > > > > > > > >> <
> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > > >>>> > > > > > > > <
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>>
> > > > >>>> > > > > > > > > >> > > <
> > > > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > > >>>> > > > > > > > <
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > > > >>>> > > > > > > > > >> <
> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > > >>>> > > > > > > > <
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>>>
> > > > >>>> > > > > > > > > >> > > > > > <
> > > > >>>> > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > > >>>> > > > > > > > <
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > > > >>>> > > > > > > > > >> <
> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > > >>>> > > > > > > > <
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>>
> > > > >>>> > > > > > > > > >> > > <
> > > > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > > >>>> > > > > > > > <
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > > > >>>> > > > > > > > > >> <
> > > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > > >>>> > > > > > > > <
> > > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > > >>>> > > > > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>>>>
> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > %3A+Time-based+log+compaction+policy
> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >>>> https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > >>>> > > > > > > > > >> > > <
> https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > > > >>>> > > > > > > > > >> > > > > > <
> > > > https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > > >>>> > > > > > > > > >> > > <
> https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> > > > >>>> > > > > > > > > >> > > > > > > >
> > > > >>>> > > > > > > > > >> > > > > >
> > > > >>>> > > > > > > > > >> > > >
> > > > >>>> > > > > > > > > >> > >
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> > --
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> > Brett Rann
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000
> > > Australia
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > > > >>>> > > > > > > > > >> >
> > > > >>>> > > > > > > > > >>
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > --
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > Brett Rann
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > Zendesk International Ltd
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000
> Australia
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > >>>> > > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > --
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Brett Rann
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Senior DevOps Engineer
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Zendesk International Ltd
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > > >>>> > > > > > > > >
> > > > >>>> > > > > > > >
> > > > >>>> > > > > > >
> > > > >>>> > > > > >
> > > > >>>> > > >
> > > > >>>> >
> > > > >>>>
> > > > >>>
> > > >
> > >
> >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by Dong Lin <li...@gmail.com>.
Thanks for the KIP Xiongqi. LGTM. +1 (binding)
One minor comment: it may be a bit better to clarify in the public
interface section that the value of the newly added metric is determined
based by applying that formula across all compactable segments. For example:
The maximum value of Math.max(now -
earliest_timestamp_in_ms_of_uncompacted_segment - max.compaction.lag.ms,
0)/1000 across all compactable partitions, where the max.compaction.lag.ms
can be overridden on per-topic basis.
On Fri, Nov 9, 2018 at 5:16 PM xiongqi wu <xi...@gmail.com> wrote:
> Thanks Joel.
> Tracking the delay at second granularity makes sense
> I have updated KIP.
>
> Xiongqi (Wesley) Wu
>
>
> On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com> wrote:
>
> > +1 with one suggestion on the proposed metric. You should probably
> include
> > the unit. So for e.g., max-compaction-delay-secs.
> >
> > Joel
> >
> > On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xi...@gmail.com> wrote:
> >
> > > bump
> > > Xiongqi (Wesley) Wu
> > >
> > >
> > > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > > >
> > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for feedback.
> > > > Can I have more feedback or VOTE on this KIP?
> > > >
> > > >
> > > > Xiongqi (Wesley) Wu
> > > >
> > > >
> > > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <xi...@gmail.com>
> > wrote:
> > > >
> > > >> Any other votes or comments?
> > > >>
> > > >> Xiongqi (Wesley) Wu
> > > >>
> > > >>
> > > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <xi...@gmail.com>
> > > wrote:
> > > >>
> > > >>> Yes, more votes and code review.
> > > >>>
> > > >>> Xiongqi (Wesley) Wu
> > > >>>
> > > >>>
> > > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> > <brann@zendesk.com.invalid
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > > >>>>
> > > >>>> Where do we go from here? More votes?
> > > >>>>
> > > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <cm...@apache.org>
> > > >>>> wrote:
> > > >>>>
> > > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > > >>>> > > Thank you for comments. I will use '0' for now.
> > > >>>> > >
> > > >>>> > > If we create topics through admin client in the future, we
> might
> > > >>>> perform
> > > >>>> > > some useful checks. (but the assumption is all brokers in the
> > same
> > > >>>> > cluster
> > > >>>> > > have the same default configurations value, otherwise,it might
> > > >>>> still be
> > > >>>> > > tricky to do such cross validation check.)
> > > >>>> >
> > > >>>> > This isn't something that we might do in the future-- this is
> > > >>>> something we
> > > >>>> > are doing now. We already have Create Topic policies which are
> > > >>>> enforced by
> > > >>>> > the broker. Check KIP-108 and KIP-170 for details. This is one
> of
> > > the
> > > >>>> > motivations for getting rid of direct ZK access-- making sure
> that
> > > >>>> these
> > > >>>> > policies are applied.
> > > >>>> >
> > > >>>> > I agree that having different configurations on different
> brokers
> > > can
> > > >>>> be
> > > >>>> > confusing and frustrating . That's why more configurations are
> > being
> > > >>>> made
> > > >>>> > dynamic using KIP-226. Dynamic configurations are stored
> centrally
> > > in
> > > >>>> ZK,
> > > >>>> > so they are the same on all brokers (modulo propagation delays).
> > In
> > > >>>> any
> > > >>>> > case, this is a general issue, not specific to "create topics".
> > > >>>> >
> > > >>>> > cheers,
> > > >>>> > Colin
> > > >>>> >
> > > >>>> >
> > > >>>> > >
> > > >>>> > >
> > > >>>> > > Xiongqi (Wesley) Wu
> > > >>>> > >
> > > >>>> > >
> > > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> > cmccabe@apache.org
> > > >
> > > >>>> > wrote:
> > > >>>> > >
> > > >>>> > > > I don't have a strong opinion. But I think we should
> probably
> > be
> > > >>>> > > > consistent with how segment.ms works, and just use 0.
> > > >>>> > > >
> > > >>>> > > > best,
> > > >>>> > > > Colin
> > > >>>> > > >
> > > >>>> > > >
> > > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> > > >>>> > > > > OK thanks for that clarification. I see why you're
> > > uncomfortable
> > > >>>> > with 0
> > > >>>> > > > now.
> > > >>>> > > > >
> > > >>>> > > > > I'm not really fussed. I just prefer consistency in
> > > >>>> configuration
> > > >>>> > > > options.
> > > >>>> > > > >
> > > >>>> > > > > Personally I lean towards treating 0 and 1 similarly in
> that
> > > >>>> > scenario,
> > > >>>> > > > > because it favours the person thinking about setting the
> > > >>>> > configurations,
> > > >>>> > > > > and a person doesn't care about a 1ms edge case especially
> > > when
> > > >>>> the
> > > >>>> > > > context
> > > >>>> > > > > is the true minimum is tied to the log cleaner cadence.
> > > >>>> > > > >
> > > >>>> > > > > Introducing 0 to mean "disabled" because there is some
> > > >>>> uniquness in
> > > >>>> > > > > segment.ms not being able to be set to 0, reduces
> > > configuration
> > > >>>> > > > consistency
> > > >>>> > > > > in favour of capturing a MS gap in an edge case that
> nobody
> > > >>>> would
> > > >>>> > ever
> > > >>>> > > > > notice. For someone to understand why everywhere else -1
> is
> > > >>>> used to
> > > >>>> > > > > disable, but here 0 is used, they would need to learn
> about
> > > >>>> > segment.ms
> > > >>>> > > > > having a 1ms minimum and then after learning would think
> > "who
> > > >>>> cares
> > > >>>> > about
> > > >>>> > > > > 1ms?" in this context. I would anyway :)
> > > >>>> > > > >
> > > >>>> > > > > my 2c anyway. Will again defer to majority. Curious which
> > way
> > > >>>> Colin
> > > >>>> > falls
> > > >>>> > > > > now.
> > > >>>> > > > >
> > > >>>> > > > > Don't want to spend more time on this though, It's well
> into
> > > >>>> > > > bikeshedding
> > > >>>> > > > > territory now. :)
> > > >>>> > > > >
> > > >>>> > > > >
> > > >>>> > > > >
> > > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > > xiongqiwu@gmail.com>
> > > >>>> > wrote:
> > > >>>> > > > >
> > > >>>> > > > > > I want to honor the minimum value of segment.ms (which
> is
> > > >>>> 1ms) to
> > > >>>> > > > force
> > > >>>> > > > > > roll an active segment.
> > > >>>> > > > > > So if we set "max.compaction.lag.ms" any value > 0, the
> > > >>>> minimum of
> > > >>>> > > > > > max.compaction.lag.ms and segment.ms will be used to
> seal
> > > an
> > > >>>> > active
> > > >>>> > > > > > segment.
> > > >>>> > > > > >
> > > >>>> > > > > > If we set max.compaction.lag.ms to 0, the current
> > > >>>> implementation
> > > >>>> > will
> > > >>>> > > > > > treat it as disabled.
> > > >>>> > > > > >
> > > >>>> > > > > > It is a little bit weird to treat max.compaction.lag=0
> the
> > > >>>> same as
> > > >>>> > > > > > max.compaction.lag=1.
> > > >>>> > > > > >
> > > >>>> > > > > > There might be a reason why we set the minimum of
> > > segment.ms
> > > >>>> to 1,
> > > >>>> > > > and I
> > > >>>> > > > > > don't want to break this assumption.
> > > >>>> > > > > >
> > > >>>> > > > > >
> > > >>>> > > > > >
> > > >>>> > > > > > Xiongqi (Wesley) Wu
> > > >>>> > > > > >
> > > >>>> > > > > >
> > > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > > >>>> > <br...@zendesk.com.invalid>
> > > >>>> > > > > > wrote:
> > > >>>> > > > > >
> > > >>>> > > > > > > You're rolling a new segment if the condition is met
> > > right?
> > > >>>> So
> > > >>>> > I'm
> > > >>>> > > > > > > struggling to understand the relevance of segment.ms
> > > here.
> > > >>>> > Maybe an
> > > >>>> > > > > > > example
> > > >>>> > > > > > > would help my understanding:
> > > >>>> > > > > > >
> > > >>>> > > > > > > segment.ms=9999999
> > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > >>>> > > > > > > max.compaction.lag.ms=1
> > > >>>> > > > > > >
> > > >>>> > > > > > > When a duplicate message comes in, after 1ms the topic
> > > >>>> should be
> > > >>>> > > > eligible
> > > >>>> > > > > > > for compaction when the log compaction thread gets
> > around
> > > to
> > > >>>> > > > evaluating
> > > >>>> > > > > > the
> > > >>>> > > > > > > topic.
> > > >>>> > > > > > >
> > > >>>> > > > > > > if we have
> > > >>>> > > > > > > segment.ms=9999999
> > > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > > >>>> > > > > > > max.compaction.lag.ms=0
> > > >>>> > > > > > >
> > > >>>> > > > > > > When a duplicate message comes in, after 0ms the topic
> > > >>>> should be
> > > >>>> > > > eligible
> > > >>>> > > > > > > for compaction when the log compaction thread gets
> > around
> > > to
> > > >>>> > > > evaluating
> > > >>>> > > > > > the
> > > >>>> > > > > > > topic.
> > > >>>> > > > > > >
> > > >>>> > > > > > > In both of those cases the change would mean a new
> > segment
> > > >>>> is
> > > >>>> > rolled
> > > >>>> > > > so
> > > >>>> > > > > > the
> > > >>>> > > > > > > new message would be part of the compaction task. 0
> and
> > 1
> > > >>>> are
> > > >>>> > > > practically
> > > >>>> > > > > > > the same meaning since neither is providing an actual
> > > >>>> guarantee
> > > >>>> > at
> > > >>>> > > > such
> > > >>>> > > > > > low
> > > >>>> > > > > > > MS settings, but effectively tying it to both the
> > > frequency
> > > >>>> of
> > > >>>> > the
> > > >>>> > > > log
> > > >>>> > > > > > > cleaner running and the priority of the given topic
> > being
> > > >>>> the
> > > >>>> > highest
> > > >>>> > > > > > > priority of all topics that are evaluated for cleaning
> > on
> > > >>>> the
> > > >>>> > next
> > > >>>> > > > cycle.
> > > >>>> > > > > > > You've captured that nuance with careful "skipped"
> > wording
> > > >>>> in
> > > >>>> > the KIP
> > > >>>> > > > > > > here "controls
> > > >>>> > > > > > > the max time interval a message/segment can be skipped
> > for
> > > >>>> log
> > > >>>> > > > > > compaction".
> > > >>>> > > > > > >
> > > >>>> > > > > > > How is 0 different to 1, practically? And how is it
> > > >>>> relating to
> > > >>>> > > > > > segment.ms
> > > >>>> > > > > > > ?
> > > >>>> > > > > > > Is it that you're proposing to have 0 mean "use
> > > segment.ms
> > > >>>> > > > instead?" as
> > > >>>> > > > > > a
> > > >>>> > > > > > > kind of third option?
> > > >>>> > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
> > > >>>> xiongqiwu@gmail.com>
> > > >>>> > > > wrote:
> > > >>>> > > > > > >
> > > >>>> > > > > > > > To make it clear,
> > > >>>> > > > > > > > I don't against using -1 as disabled, but we need to
> > > come
> > > >>>> up
> > > >>>> > with
> > > >>>> > > > the
> > > >>>> > > > > > > > meaning of "0".
> > > >>>> > > > > > > > If "0" means immediate compaction, but the actual
> > > >>>> compaction
> > > >>>> > lag
> > > >>>> > > > will
> > > >>>> > > > > > be
> > > >>>> > > > > > > > segment.ms.
> > > >>>> > > > > > > > It has longer lag than setting the value to be half
> of
> > > >>>> > segment.ms.
> > > >>>> > > > > > > > We cannot provide "0" as max compaction lag.
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > Here are two options.
> > > >>>> > > > > > > > Option 1:
> > > >>>> > > > > > > > Keep 0 as disabled
> > > >>>> > > > > > > > Option 2:
> > > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag = segment.ms),
> > and
> > > >>>> > others.
> > > >>>> > > > > > > >
> > > >>>> > > > > > > >
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > > >>>> > > > > > > >
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> > > >>>> > > > <br...@zendesk.com.invalid>
> > > >>>> > > > > > > > wrote:
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > > -1 is consistent as "special" with these settings
> > for
> > > >>>> > example:
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > log.retention.bytes
> > > >>>> > > > > > > > > socket.received.buffer.bytes
> > > >>>> > > > > > > > > socket.send.buffer.bytes
> > > >>>> > > > > > > > > queued.max.request.bytes
> > > >>>> > > > > > > > > retention.bytes
> > > >>>> > > > > > > > > retention.ms
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > and acks.
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > Where it may mean no limit, use OS defaults, max
> > > (acks),
> > > >>>> > etc. I
> > > >>>> > > > don't
> > > >>>> > > > > > > see
> > > >>>> > > > > > > > > much convention of 0 meaning those things.
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > There are some NULLs but it seems convetion there
> is
> > > >>>> NULL is
> > > >>>> > used
> > > >>>> > > > > > where
> > > >>>> > > > > > > > > there's another setting in the hierarchy that
> would
> > be
> > > >>>> used
> > > >>>> > > > instead.
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann <
> > > >>>> > brann@zendesk.com>
> > > >>>> > > > > > wrote:
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > > If segment.ms can't be set to 0, then we're not
> > > being
> > > >>>> > > > consistent
> > > >>>> > > > > > > > > > by using 0 for this new setting? I throw out -1
> > for
> > > >>>> > > > consideration
> > > >>>> > > > > > > > > > again :)
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu <
> > > >>>> > > > xiongqiwu@gmail.com>
> > > >>>> > > > > > > > wrote:
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > >> Thanks. I will document after PR is merged.
> > > >>>> > > > > > > > > >>
> > > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "segment.ms"
> > to
> > > >>>> 1, we
> > > >>>> > > > cannot
> > > >>>> > > > > > set
> > > >>>> > > > > > > "
> > > >>>> > > > > > > > > >> segment.ms" to 0.
> > > >>>> > > > > > > > > >>
> > > >>>> > > > > > > > > >> I also updated the title of this KIP.
> > > >>>> > > > > > > > > >>
> > > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > > >>>> > > > > > > > > >>
> > > >>>> > > > > > > > > >>
> > > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett Rann
> > > >>>> > > > > > <brann@zendesk.com.invalid
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > > >> wrote:
> > > >>>> > > > > > > > > >>
> > > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm in the
> > > >>>> minority.
> > > >>>> > :)
> > > >>>> > > > Can
> > > >>>> > > > > > we
> > > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning
> disabled
> > > >>>> here:
> > > >>>> > > > > > > > > >> >
> > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > > >>>> > > > > > > > <
> > https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>
> > > >>>> > > > > > > > > >> <
> > > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > > >>>> > > > > > > > <
> > https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > > >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> > > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>> ?
> > > >>>> > > > > > > > > >> > And while there it would be good if
> segment.ms
> > > is
> > > >>>> > > > documented
> > > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some
> hierarchy
> > > of
> > > >>>> > configs
> > > >>>> > > > for
> > > >>>> > > > > > > that
> > > >>>> > > > > > > > > too
> > > >>>> > > > > > > > > >> > if its not set and null for others means
> > > disabled!)
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi wu <
> > > >>>> > > > xiongqiwu@gmail.com>
> > > >>>> > > > > > > > > wrote:
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate
> compaction,
> > > the
> > > >>>> > > > compaction
> > > >>>> > > > > > lag
> > > >>>> > > > > > > > is
> > > >>>> > > > > > > > > >> > > determined by segment.ms in worst case. If
> > > >>>> segment.ms
> > > >>>> > is
> > > >>>> > > > 24
> > > >>>> > > > > > > > hours,
> > > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker
> guarantee
> > > than
> > > >>>> > setting
> > > >>>> > > > any
> > > >>>> > > > > > > > value
> > > >>>> > > > > > > > > >> less
> > > >>>> > > > > > > > > >> > > than 24 hours. By the definition of "max
> > > >>>> compaction
> > > >>>> > lag",
> > > >>>> > > > we
> > > >>>> > > > > > > > cannot
> > > >>>> > > > > > > > > >> have
> > > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate "disable".
> > > >>>> > > > > > > > > >> > >
> > > >>>> > > > > > > > > >> > >
> > > >>>> > > > > > > > > >> > >
> > > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > > >>>> > > > > > > > > >> > >
> > > >>>> > > > > > > > > >> > >
> > > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin
> McCabe <
> > > >>>> > > > > > cmccabe@apache.org
> > > >>>> > > > > > > >
> > > >>>> > > > > > > > > >> wrote:
> > > >>>> > > > > > > > > >> > >
> > > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, Brett Rann
> > > >>>> wrote:
> > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should make
> 0 =
> > > >>>> > disable, to
> > > >>>> > > > be
> > > >>>> > > > > > > > > >> consistent
> > > >>>> > > > > > > > > >> > > with
> > > >>>> > > > > > > > > >> > > > > the other settings.
> > > >>>> > > > > > > > > >> > > > >
> > > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable and
> when
> > > >>>> seeing
> > > >>>> > it
> > > >>>> > > > in a
> > > >>>> > > > > > > > config
> > > >>>> > > > > > > > > >> it's
> > > >>>> > > > > > > > > >> > > > clear
> > > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0 doesn't
> > have
> > > >>>> to
> > > >>>> > mean
> > > >>>> > > > > > instant,
> > > >>>> > > > > > > > it
> > > >>>> > > > > > > > > >> just
> > > >>>> > > > > > > > > >> > > > means
> > > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't think 0
> > is
> > > >>>> > intuitive
> > > >>>> > > > for
> > > >>>> > > > > > > > > disabled
> > > >>>> > > > > > > > > >> and
> > > >>>> > > > > > > > > >> > > it
> > > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't aware
> > > segment.ms=0
> > > >>>> ==
> > > >>>> > > > > > disabled,
> > > >>>> > > > > > > > > but I
> > > >>>> > > > > > > > > >> > > think
> > > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> > > >>>> > > > > > > > > >> > > >
> > > >>>> > > > > > > > > >> > > > I think there is an argument for keeping
> > > these
> > > >>>> two
> > > >>>> > > > > > > > configurations
> > > >>>> > > > > > > > > >> > > > consistent, since they are so similar. I
> > > agree
> > > >>>> that
> > > >>>> > 0
> > > >>>> > > > was an
> > > >>>> > > > > > > > > >> > unfortunate
> > > >>>> > > > > > > > > >> > > > choice.,
> > > >>>> > > > > > > > > >> > > >
> > > >>>> > > > > > > > > >> > > > best,
> > > >>>> > > > > > > > > >> > > > Colin
> > > >>>> > > > > > > > > >> > > >
> > > >>>> > > > > > > > > >> > > > >
> > > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM Colin
> > > McCabe
> > > >>>> <
> > > >>>> > > > > > > > > cmccabe@apache.org>
> > > >>>> > > > > > > > > >> > > wrote:
> > > >>>> > > > > > > > > >> > > > >
> > > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47,
> xiongqi
> > wu
> > > >>>> wrote:
> > > >>>> > > > > > > > > >> > > > > > > Colin,
> > > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> > > >>>> > > > > > > > > >> > > > > > > see my inline reply below.
> > > >>>> > > > > > > > > >> > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > > >>>> > > > > > > > > >> > > > > > >
> > > >>>> > > > > > > > > >> > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 PM
> Colin
> > > >>>> McCabe <
> > > >>>> > > > > > > > > >> cmccabe@apache.org>
> > > >>>> > > > > > > > > >> > > > wrote:
> > > >>>> > > > > > > > > >> > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > The name seems a bit ambiguous.
> Our
> > > >>>> > compaction
> > > >>>> > > > > > > policies
> > > >>>> > > > > > > > > are
> > > >>>> > > > > > > > > >> > > already
> > > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It seems
> > like
> > > >>>> this
> > > >>>> > > > change is
> > > >>>> > > > > > > > > focused
> > > >>>> > > > > > > > > >> > > around
> > > >>>> > > > > > > > > >> > > > > > adding
> > > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms."
> Perhaps
> > > >>>> the KIP
> > > >>>> > > > title
> > > >>>> > > > > > > should
> > > >>>> > > > > > > > > be
> > > >>>> > > > > > > > > >> > > > something
> > > >>>> > > > > > > > > >> > > > > > > > like "add maximum compaction lag
> > > time"?
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will change
> the
> > > >>>> title.
> > > >>>> > > > > > > > > >> > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > The active segment is forced to
> > roll
> > > >>>> when
> > > >>>> > > > either "
> > > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> > > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (log.roll.ms
> and
> > > >>>> > > > log.roll.hours)
> > > >>>> > > > > > > has
> > > >>>> > > > > > > > > >> > reached.
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > If the max.compaction.lag.ms is
> > low,
> > > >>>> it
> > > >>>> > seems
> > > >>>> > > > like
> > > >>>> > > > > > > > > segments
> > > >>>> > > > > > > > > >> > will
> > > >>>> > > > > > > > > >> > > > be
> > > >>>> > > > > > > > > >> > > > > > > > rolled very frequently. This can
> > be a
> > > >>>> > source of
> > > >>>> > > > > > > problems
> > > >>>> > > > > > > > > in
> > > >>>> > > > > > > > > >> the
> > > >>>> > > > > > > > > >> > > > > > cluster,
> > > >>>> > > > > > > > > >> > > > > > > > since creating many different
> small
> > > log
> > > >>>> > segments
> > > >>>> > > > > > > > consumes
> > > >>>> > > > > > > > > a
> > > >>>> > > > > > > > > >> > huge
> > > >>>> > > > > > > > > >> > > > > > amount of
> > > >>>> > > > > > > > > >> > > > > > > > cluster resources. Therefore, I
> > would
> > > >>>> > suggest
> > > >>>> > > > > > adding a
> > > >>>> > > > > > > > > >> > > broker-level
> > > >>>> > > > > > > > > >> > > > > > > > configuration which allows us to
> > set
> > > a
> > > >>>> > minimum
> > > >>>> > > > value
> > > >>>> > > > > > > for
> > > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If we let
> > > >>>> users set
> > > >>>> > it
> > > >>>> > > > on a
> > > >>>> > > > > > > > > >> per-topic
> > > >>>> > > > > > > > > >> > > > basis,
> > > >>>> > > > > > > > > >> > > > > > > > someone could set a value of 1 ms
> > or
> > > >>>> > something,
> > > >>>> > > > and
> > > >>>> > > > > > > > cause
> > > >>>> > > > > > > > > >> > chaos.
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > =========> this applies to
> > > segment.ms
> > > >>>> as
> > > >>>> > well.
> > > >>>> > > > > > Today
> > > >>>> > > > > > > > > users
> > > >>>> > > > > > > > > >> can
> > > >>>> > > > > > > > > >> > > > set "
> > > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low value,
> and
> > > >>>> cause a
> > > >>>> > > > frequent
> > > >>>> > > > > > > > > >> rolling of
> > > >>>> > > > > > > > > >> > > > active
> > > >>>> > > > > > > > > >> > > > > > > segments.
> > > >>>> > > > > > > > > >> > > > > >
> > > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > > >>>> > > > > > > > > >> > > > > >
> > > >>>> > > > > > > > > >> > > > > > I agree that this is an existing
> > problem
> > > >>>> with
> > > >>>> > > > > > segment.ms.
> > > >>>> > > > > > > > > >> However,
> > > >>>> > > > > > > > > >> > > > that
> > > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't fix
> it.
> > As
> > > >>>> you
> > > >>>> > noted,
> > > >>>> > > > > > there
> > > >>>> > > > > > > > > will
> > > >>>> > > > > > > > > >> be
> > > >>>> > > > > > > > > >> > > more
> > > >>>> > > > > > > > > >> > > > > > interest in these topic-level
> retention
> > > >>>> > settings as
> > > >>>> > > > a
> > > >>>> > > > > > > result
> > > >>>> > > > > > > > > of
> > > >>>> > > > > > > > > >> > GDPR.
> > > >>>> > > > > > > > > >> > > > It
> > > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing
> problems
> > > >>>> will
> > > >>>> > cause
> > > >>>> > > > more
> > > >>>> > > > > > > > > trouble.
> > > >>>> > > > > > > > > >> > > > > >
> > > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> > straightforward
> > > >>>> here --
> > > >>>> > > > add a
> > > >>>> > > > > > > > > >> broker-level
> > > >>>> > > > > > > > > >> > > > > > minimum segment.ms that overrides
> > > >>>> per-topic
> > > >>>> > > > minimums.
> > > >>>> > > > > > We
> > > >>>> > > > > > > > can
> > > >>>> > > > > > > > > >> also
> > > >>>> > > > > > > > > >> > > fail
> > > >>>> > > > > > > > > >> > > > > > with a helpful error message when
> > someone
> > > >>>> > attempts
> > > >>>> > > > to
> > > >>>> > > > > > set
> > > >>>> > > > > > > an
> > > >>>> > > > > > > > > >> > invalid
> > > >>>> > > > > > > > > >> > > > > > configuration.
> > > >>>> > > > > > > > > >> > > > > >
> > > >>>> > > > > > > > > >> > > > > > > In my option, the minimum of "
> > > >>>> > > > max.compaction.lag.ms"
> > > >>>> > > > > > > > should
> > > >>>> > > > > > > > > >> be
> > > >>>> > > > > > > > > >> > > > > > > based on the minimum of "
> segment.ms
> > ".
> > > >>>> Since
> > > >>>> > > > today the
> > > >>>> > > > > > > > > >> minimum of
> > > >>>> > > > > > > > > >> > > > > > segment.ms
> > > >>>> > > > > > > > > >> > > > > > > is 1, "max.compaction.lag.ms" also
> > > >>>> starts
> > > >>>> > with
> > > >>>> > > > 1. "0"
> > > >>>> > > > > > > > means
> > > >>>> > > > > > > > > >> > > > disable. I
> > > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but it is
> hard
> > > to
> > > >>>> > define
> > > >>>> > > > the
> > > >>>> > > > > > > > meaning
> > > >>>> > > > > > > > > >> of 0
> > > >>>> > > > > > > > > >> > > > because
> > > >>>> > > > > > > > > >> > > > > > we
> > > >>>> > > > > > > > > >> > > > > > > cannot just roll the active segment
> > > >>>> > immediately.
> > > >>>> > > > > > > > > >> > > > > >
> > > >>>> > > > > > > > > >> > > > > > That's a fair point. We should make
> 0 =
> > > >>>> > disable, to
> > > >>>> > > > be
> > > >>>> > > > > > > > > >> consistent
> > > >>>> > > > > > > > > >> > > with
> > > >>>> > > > > > > > > >> > > > the
> > > >>>> > > > > > > > > >> > > > > > other settings.
> > > >>>> > > > > > > > > >> > > > > >
> > > >>>> > > > > > > > > >> > > > > > best,
> > > >>>> > > > > > > > > >> > > > > > Colin
> > > >>>> > > > > > > > > >> > > > > >
> > > >>>> > > > > > > > > >> > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > -- Note that an alternative
> > > >>>> configuration
> > > >>>> > is to
> > > >>>> > > > use
> > > >>>> > > > > > -1
> > > >>>> > > > > > > > as
> > > >>>> > > > > > > > > >> > > > "disabled"
> > > >>>> > > > > > > > > >> > > > > > and
> > > >>>> > > > > > > > > >> > > > > > > > 0
> > > >>>> > > > > > > > > >> > > > > > > > > as "immediate compaction".
> > Because
> > > >>>> > compaction
> > > >>>> > > > lag
> > > >>>> > > > > > is
> > > >>>> > > > > > > > > still
> > > >>>> > > > > > > > > >> > > > determined
> > > >>>> > > > > > > > > >> > > > > > > > > based on min.compaction.lag and
> > how
> > > >>>> long
> > > >>>> > to
> > > >>>> > > > roll
> > > >>>> > > > > > an
> > > >>>> > > > > > > > > active
> > > >>>> > > > > > > > > >> > > > segment,
> > > >>>> > > > > > > > > >> > > > > > > > the
> > > >>>> > > > > > > > > >> > > > > > > > > actual lag for compaction is
> > > >>>> undetermined
> > > >>>> > if
> > > >>>> > > > we
> > > >>>> > > > > > use
> > > >>>> > > > > > > > "0".
> > > >>>> > > > > > > > > >> On
> > > >>>> > > > > > > > > >> > the
> > > >>>> > > > > > > > > >> > > > other
> > > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set
> > > >>>> > > > > > "min.cleanable.dirty.ratio"
> > > >>>> > > > > > > > to
> > > >>>> > > > > > > > > >> > achieve
> > > >>>> > > > > > > > > >> > > > the
> > > >>>> > > > > > > > > >> > > > > > > > same
> > > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose "0" as
> > > >>>> "disabled".
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be the
> invalid
> > > >>>> setting.
> > > >>>> > > > > > Treating
> > > >>>> > > > > > > 0
> > > >>>> > > > > > > > > >> > > differently
> > > >>>> > > > > > > > > >> > > > > > than
> > > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > =====> see my previous comment, I
> > am
> > > >>>> not
> > > >>>> > > > strongly
> > > >>>> > > > > > > > against,
> > > >>>> > > > > > > > > >> but
> > > >>>> > > > > > > > > >> > 0
> > > >>>> > > > > > > > > >> > > is
> > > >>>> > > > > > > > > >> > > > > > not a
> > > >>>> > > > > > > > > >> > > > > > > valid configuration in my option.
> So
> > I
> > > >>>> use
> > > >>>> > "0" as
> > > >>>> > > > > > > disabled
> > > >>>> > > > > > > > > >> state.
> > > >>>> > > > > > > > > >> > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > best,
> > > >>>> > > > > > > > > >> > > > > > > > Colin
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at 15:04,
> > > xiongqi
> > > >>>> wu
> > > >>>> > wrote:
> > > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
> > > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > > >>>> > > > > > > > > >> > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > >>>> > > > > > > > <
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > > >>>> > > > > > > > > >> <
> > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > >>>> > > > > > > > <
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>
> > > >>>> > > > > > > > > >> > > <
> > > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > >>>> > > > > > > > <
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > > >>>> > > > > > > > > >> <
> > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > >>>> > > > > > > > <
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>>
> > > >>>> > > > > > > > > >> > > > > > <
> > > >>>> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > >>>> > > > > > > > <
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > > >>>> > > > > > > > > >> <
> > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > >>>> > > > > > > > <
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>
> > > >>>> > > > > > > > > >> > > <
> > > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > >>>> > > > > > > > <
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > > >>>> > > > > > > > > >> <
> > > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > > >>>> > > > > > > > <
> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > > >>>> > > > > > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>>>>
> > > >>>> > > > > > > > > >> > > > > > > > >
> > > %3A+Time-based+log+compaction+policy
> > > >>>> > > > > > > > > >> > > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > > >>>> > > > > > > > > >> > > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > >
> > > >>>> https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > >>>> > > > > > > > > >> > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > > >>>> > > > > > > > > >> > > > > > <
> > > https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > > >>>> > > > > > > > > >> > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>
> > > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > > >>>> > > > > > > > > >> > > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > >
> > > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> > > >>>> > > > > > > > > >> > > > > > > >
> > > >>>> > > > > > > > > >> > > > > >
> > > >>>> > > > > > > > > >> > > >
> > > >>>> > > > > > > > > >> > >
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> > --
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> > Brett Rann
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> > Zendesk International Ltd
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000
> > Australia
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > > >>>> > > > > > > > > >> >
> > > >>>> > > > > > > > > >>
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > --
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > Brett Rann
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > Senior DevOps Engineer
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > Zendesk International Ltd
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > > >>>> > > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > --
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > Brett Rann
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > Senior DevOps Engineer
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > Zendesk International Ltd
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > > >>>> > > > > > > > >
> > > >>>> > > > > > > >
> > > >>>> > > > > > >
> > > >>>> > > > > >
> > > >>>> > > >
> > > >>>> >
> > > >>>>
> > > >>>
> > >
> >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by xiongqi wu <xi...@gmail.com>.
Thanks Joel.
Tracking the delay at second granularity makes sense
I have updated KIP.
Xiongqi (Wesley) Wu
On Fri, Nov 9, 2018 at 5:05 PM Joel Koshy <jj...@gmail.com> wrote:
> +1 with one suggestion on the proposed metric. You should probably include
> the unit. So for e.g., max-compaction-delay-secs.
>
> Joel
>
> On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xi...@gmail.com> wrote:
>
> > bump
> > Xiongqi (Wesley) Wu
> >
> >
> > On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xi...@gmail.com> wrote:
> >
> > >
> > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for feedback.
> > > Can I have more feedback or VOTE on this KIP?
> > >
> > >
> > > Xiongqi (Wesley) Wu
> > >
> > >
> > > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <xi...@gmail.com>
> wrote:
> > >
> > >> Any other votes or comments?
> > >>
> > >> Xiongqi (Wesley) Wu
> > >>
> > >>
> > >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <xi...@gmail.com>
> > wrote:
> > >>
> > >>> Yes, more votes and code review.
> > >>>
> > >>> Xiongqi (Wesley) Wu
> > >>>
> > >>>
> > >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann
> <brann@zendesk.com.invalid
> > >
> > >>> wrote:
> > >>>
> > >>>> +1 (non binding) from on 0 then, and on the KIP.
> > >>>>
> > >>>> Where do we go from here? More votes?
> > >>>>
> > >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <cm...@apache.org>
> > >>>> wrote:
> > >>>>
> > >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> > >>>> > > Thank you for comments. I will use '0' for now.
> > >>>> > >
> > >>>> > > If we create topics through admin client in the future, we might
> > >>>> perform
> > >>>> > > some useful checks. (but the assumption is all brokers in the
> same
> > >>>> > cluster
> > >>>> > > have the same default configurations value, otherwise,it might
> > >>>> still be
> > >>>> > > tricky to do such cross validation check.)
> > >>>> >
> > >>>> > This isn't something that we might do in the future-- this is
> > >>>> something we
> > >>>> > are doing now. We already have Create Topic policies which are
> > >>>> enforced by
> > >>>> > the broker. Check KIP-108 and KIP-170 for details. This is one of
> > the
> > >>>> > motivations for getting rid of direct ZK access-- making sure that
> > >>>> these
> > >>>> > policies are applied.
> > >>>> >
> > >>>> > I agree that having different configurations on different brokers
> > can
> > >>>> be
> > >>>> > confusing and frustrating . That's why more configurations are
> being
> > >>>> made
> > >>>> > dynamic using KIP-226. Dynamic configurations are stored centrally
> > in
> > >>>> ZK,
> > >>>> > so they are the same on all brokers (modulo propagation delays).
> In
> > >>>> any
> > >>>> > case, this is a general issue, not specific to "create topics".
> > >>>> >
> > >>>> > cheers,
> > >>>> > Colin
> > >>>> >
> > >>>> >
> > >>>> > >
> > >>>> > >
> > >>>> > > Xiongqi (Wesley) Wu
> > >>>> > >
> > >>>> > >
> > >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <
> cmccabe@apache.org
> > >
> > >>>> > wrote:
> > >>>> > >
> > >>>> > > > I don't have a strong opinion. But I think we should probably
> be
> > >>>> > > > consistent with how segment.ms works, and just use 0.
> > >>>> > > >
> > >>>> > > > best,
> > >>>> > > > Colin
> > >>>> > > >
> > >>>> > > >
> > >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> > >>>> > > > > OK thanks for that clarification. I see why you're
> > uncomfortable
> > >>>> > with 0
> > >>>> > > > now.
> > >>>> > > > >
> > >>>> > > > > I'm not really fussed. I just prefer consistency in
> > >>>> configuration
> > >>>> > > > options.
> > >>>> > > > >
> > >>>> > > > > Personally I lean towards treating 0 and 1 similarly in that
> > >>>> > scenario,
> > >>>> > > > > because it favours the person thinking about setting the
> > >>>> > configurations,
> > >>>> > > > > and a person doesn't care about a 1ms edge case especially
> > when
> > >>>> the
> > >>>> > > > context
> > >>>> > > > > is the true minimum is tied to the log cleaner cadence.
> > >>>> > > > >
> > >>>> > > > > Introducing 0 to mean "disabled" because there is some
> > >>>> uniquness in
> > >>>> > > > > segment.ms not being able to be set to 0, reduces
> > configuration
> > >>>> > > > consistency
> > >>>> > > > > in favour of capturing a MS gap in an edge case that nobody
> > >>>> would
> > >>>> > ever
> > >>>> > > > > notice. For someone to understand why everywhere else -1 is
> > >>>> used to
> > >>>> > > > > disable, but here 0 is used, they would need to learn about
> > >>>> > segment.ms
> > >>>> > > > > having a 1ms minimum and then after learning would think
> "who
> > >>>> cares
> > >>>> > about
> > >>>> > > > > 1ms?" in this context. I would anyway :)
> > >>>> > > > >
> > >>>> > > > > my 2c anyway. Will again defer to majority. Curious which
> way
> > >>>> Colin
> > >>>> > falls
> > >>>> > > > > now.
> > >>>> > > > >
> > >>>> > > > > Don't want to spend more time on this though, It's well into
> > >>>> > > > bikeshedding
> > >>>> > > > > territory now. :)
> > >>>> > > > >
> > >>>> > > > >
> > >>>> > > > >
> > >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> > xiongqiwu@gmail.com>
> > >>>> > wrote:
> > >>>> > > > >
> > >>>> > > > > > I want to honor the minimum value of segment.ms (which is
> > >>>> 1ms) to
> > >>>> > > > force
> > >>>> > > > > > roll an active segment.
> > >>>> > > > > > So if we set "max.compaction.lag.ms" any value > 0, the
> > >>>> minimum of
> > >>>> > > > > > max.compaction.lag.ms and segment.ms will be used to seal
> > an
> > >>>> > active
> > >>>> > > > > > segment.
> > >>>> > > > > >
> > >>>> > > > > > If we set max.compaction.lag.ms to 0, the current
> > >>>> implementation
> > >>>> > will
> > >>>> > > > > > treat it as disabled.
> > >>>> > > > > >
> > >>>> > > > > > It is a little bit weird to treat max.compaction.lag=0 the
> > >>>> same as
> > >>>> > > > > > max.compaction.lag=1.
> > >>>> > > > > >
> > >>>> > > > > > There might be a reason why we set the minimum of
> > segment.ms
> > >>>> to 1,
> > >>>> > > > and I
> > >>>> > > > > > don't want to break this assumption.
> > >>>> > > > > >
> > >>>> > > > > >
> > >>>> > > > > >
> > >>>> > > > > > Xiongqi (Wesley) Wu
> > >>>> > > > > >
> > >>>> > > > > >
> > >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> > >>>> > <br...@zendesk.com.invalid>
> > >>>> > > > > > wrote:
> > >>>> > > > > >
> > >>>> > > > > > > You're rolling a new segment if the condition is met
> > right?
> > >>>> So
> > >>>> > I'm
> > >>>> > > > > > > struggling to understand the relevance of segment.ms
> > here.
> > >>>> > Maybe an
> > >>>> > > > > > > example
> > >>>> > > > > > > would help my understanding:
> > >>>> > > > > > >
> > >>>> > > > > > > segment.ms=9999999
> > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > >>>> > > > > > > max.compaction.lag.ms=1
> > >>>> > > > > > >
> > >>>> > > > > > > When a duplicate message comes in, after 1ms the topic
> > >>>> should be
> > >>>> > > > eligible
> > >>>> > > > > > > for compaction when the log compaction thread gets
> around
> > to
> > >>>> > > > evaluating
> > >>>> > > > > > the
> > >>>> > > > > > > topic.
> > >>>> > > > > > >
> > >>>> > > > > > > if we have
> > >>>> > > > > > > segment.ms=9999999
> > >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> > >>>> > > > > > > max.compaction.lag.ms=0
> > >>>> > > > > > >
> > >>>> > > > > > > When a duplicate message comes in, after 0ms the topic
> > >>>> should be
> > >>>> > > > eligible
> > >>>> > > > > > > for compaction when the log compaction thread gets
> around
> > to
> > >>>> > > > evaluating
> > >>>> > > > > > the
> > >>>> > > > > > > topic.
> > >>>> > > > > > >
> > >>>> > > > > > > In both of those cases the change would mean a new
> segment
> > >>>> is
> > >>>> > rolled
> > >>>> > > > so
> > >>>> > > > > > the
> > >>>> > > > > > > new message would be part of the compaction task. 0 and
> 1
> > >>>> are
> > >>>> > > > practically
> > >>>> > > > > > > the same meaning since neither is providing an actual
> > >>>> guarantee
> > >>>> > at
> > >>>> > > > such
> > >>>> > > > > > low
> > >>>> > > > > > > MS settings, but effectively tying it to both the
> > frequency
> > >>>> of
> > >>>> > the
> > >>>> > > > log
> > >>>> > > > > > > cleaner running and the priority of the given topic
> being
> > >>>> the
> > >>>> > highest
> > >>>> > > > > > > priority of all topics that are evaluated for cleaning
> on
> > >>>> the
> > >>>> > next
> > >>>> > > > cycle.
> > >>>> > > > > > > You've captured that nuance with careful "skipped"
> wording
> > >>>> in
> > >>>> > the KIP
> > >>>> > > > > > > here "controls
> > >>>> > > > > > > the max time interval a message/segment can be skipped
> for
> > >>>> log
> > >>>> > > > > > compaction".
> > >>>> > > > > > >
> > >>>> > > > > > > How is 0 different to 1, practically? And how is it
> > >>>> relating to
> > >>>> > > > > > segment.ms
> > >>>> > > > > > > ?
> > >>>> > > > > > > Is it that you're proposing to have 0 mean "use
> > segment.ms
> > >>>> > > > instead?" as
> > >>>> > > > > > a
> > >>>> > > > > > > kind of third option?
> > >>>> > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
> > >>>> xiongqiwu@gmail.com>
> > >>>> > > > wrote:
> > >>>> > > > > > >
> > >>>> > > > > > > > To make it clear,
> > >>>> > > > > > > > I don't against using -1 as disabled, but we need to
> > come
> > >>>> up
> > >>>> > with
> > >>>> > > > the
> > >>>> > > > > > > > meaning of "0".
> > >>>> > > > > > > > If "0" means immediate compaction, but the actual
> > >>>> compaction
> > >>>> > lag
> > >>>> > > > will
> > >>>> > > > > > be
> > >>>> > > > > > > > segment.ms.
> > >>>> > > > > > > > It has longer lag than setting the value to be half of
> > >>>> > segment.ms.
> > >>>> > > > > > > > We cannot provide "0" as max compaction lag.
> > >>>> > > > > > > >
> > >>>> > > > > > > > Here are two options.
> > >>>> > > > > > > > Option 1:
> > >>>> > > > > > > > Keep 0 as disabled
> > >>>> > > > > > > > Option 2:
> > >>>> > > > > > > > -1 (disabled), 0 (max compaction lag = segment.ms),
> and
> > >>>> > others.
> > >>>> > > > > > > >
> > >>>> > > > > > > >
> > >>>> > > > > > > >
> > >>>> > > > > > > > Xiongqi (Wesley) Wu
> > >>>> > > > > > > >
> > >>>> > > > > > > >
> > >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> > >>>> > > > <br...@zendesk.com.invalid>
> > >>>> > > > > > > > wrote:
> > >>>> > > > > > > >
> > >>>> > > > > > > > > -1 is consistent as "special" with these settings
> for
> > >>>> > example:
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > log.retention.bytes
> > >>>> > > > > > > > > socket.received.buffer.bytes
> > >>>> > > > > > > > > socket.send.buffer.bytes
> > >>>> > > > > > > > > queued.max.request.bytes
> > >>>> > > > > > > > > retention.bytes
> > >>>> > > > > > > > > retention.ms
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > and acks.
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Where it may mean no limit, use OS defaults, max
> > (acks),
> > >>>> > etc. I
> > >>>> > > > don't
> > >>>> > > > > > > see
> > >>>> > > > > > > > > much convention of 0 meaning those things.
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > There are some NULLs but it seems convetion there is
> > >>>> NULL is
> > >>>> > used
> > >>>> > > > > > where
> > >>>> > > > > > > > > there's another setting in the hierarchy that would
> be
> > >>>> used
> > >>>> > > > instead.
> > >>>> > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann <
> > >>>> > brann@zendesk.com>
> > >>>> > > > > > wrote:
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > > If segment.ms can't be set to 0, then we're not
> > being
> > >>>> > > > consistent
> > >>>> > > > > > > > > > by using 0 for this new setting? I throw out -1
> for
> > >>>> > > > consideration
> > >>>> > > > > > > > > > again :)
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu <
> > >>>> > > > xiongqiwu@gmail.com>
> > >>>> > > > > > > > wrote:
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >> Thanks. I will document after PR is merged.
> > >>>> > > > > > > > > >>
> > >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "segment.ms"
> to
> > >>>> 1, we
> > >>>> > > > cannot
> > >>>> > > > > > set
> > >>>> > > > > > > "
> > >>>> > > > > > > > > >> segment.ms" to 0.
> > >>>> > > > > > > > > >>
> > >>>> > > > > > > > > >> I also updated the title of this KIP.
> > >>>> > > > > > > > > >>
> > >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> > >>>> > > > > > > > > >>
> > >>>> > > > > > > > > >>
> > >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett Rann
> > >>>> > > > > > <brann@zendesk.com.invalid
> > >>>> > > > > > > >
> > >>>> > > > > > > > > >> wrote:
> > >>>> > > > > > > > > >>
> > >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm in the
> > >>>> minority.
> > >>>> > :)
> > >>>> > > > Can
> > >>>> > > > > > we
> > >>>> > > > > > > > > >> > make sure 0 gets documented as meaning disabled
> > >>>> here:
> > >>>> > > > > > > > > >> >
> > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > >>>> > > > > > > > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>
> > >>>> > > > > > > > > >> <
> > >>>> https://kafka.apache.org/documentation/#brokerconfigs
> > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> > >>>> > > > > > > > <
> https://kafka.apache.org/documentation/#brokerconfigs
> > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> > >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> > >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>> ?
> > >>>> > > > > > > > > >> > And while there it would be good if segment.ms
> > is
> > >>>> > > > documented
> > >>>> > > > > > > > > >> > that 0 is disabled too. (there's some hierarchy
> > of
> > >>>> > configs
> > >>>> > > > for
> > >>>> > > > > > > that
> > >>>> > > > > > > > > too
> > >>>> > > > > > > > > >> > if its not set and null for others means
> > disabled!)
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi wu <
> > >>>> > > > xiongqiwu@gmail.com>
> > >>>> > > > > > > > > wrote:
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> > > If we use 0 to indicate immediate compaction,
> > the
> > >>>> > > > compaction
> > >>>> > > > > > lag
> > >>>> > > > > > > > is
> > >>>> > > > > > > > > >> > > determined by segment.ms in worst case. If
> > >>>> segment.ms
> > >>>> > is
> > >>>> > > > 24
> > >>>> > > > > > > > hours,
> > >>>> > > > > > > > > >> > > "immediate compaction" is a weaker guarantee
> > than
> > >>>> > setting
> > >>>> > > > any
> > >>>> > > > > > > > value
> > >>>> > > > > > > > > >> less
> > >>>> > > > > > > > > >> > > than 24 hours. By the definition of "max
> > >>>> compaction
> > >>>> > lag",
> > >>>> > > > we
> > >>>> > > > > > > > cannot
> > >>>> > > > > > > > > >> have
> > >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate "disable".
> > >>>> > > > > > > > > >> > >
> > >>>> > > > > > > > > >> > >
> > >>>> > > > > > > > > >> > >
> > >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> > >>>> > > > > > > > > >> > >
> > >>>> > > > > > > > > >> > >
> > >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin McCabe <
> > >>>> > > > > > cmccabe@apache.org
> > >>>> > > > > > > >
> > >>>> > > > > > > > > >> wrote:
> > >>>> > > > > > > > > >> > >
> > >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, Brett Rann
> > >>>> wrote:
> > >>>> > > > > > > > > >> > > > > > That's a fair point. We should make 0 =
> > >>>> > disable, to
> > >>>> > > > be
> > >>>> > > > > > > > > >> consistent
> > >>>> > > > > > > > > >> > > with
> > >>>> > > > > > > > > >> > > > > the other settings.
> > >>>> > > > > > > > > >> > > > >
> > >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable and when
> > >>>> seeing
> > >>>> > it
> > >>>> > > > in a
> > >>>> > > > > > > > config
> > >>>> > > > > > > > > >> it's
> > >>>> > > > > > > > > >> > > > clear
> > >>>> > > > > > > > > >> > > > > that it's a special meaning. 0 doesn't
> have
> > >>>> to
> > >>>> > mean
> > >>>> > > > > > instant,
> > >>>> > > > > > > > it
> > >>>> > > > > > > > > >> just
> > >>>> > > > > > > > > >> > > > means
> > >>>> > > > > > > > > >> > > > > as quickly as possible. I don't think 0
> is
> > >>>> > intuitive
> > >>>> > > > for
> > >>>> > > > > > > > > disabled
> > >>>> > > > > > > > > >> and
> > >>>> > > > > > > > > >> > > it
> > >>>> > > > > > > > > >> > > > > will be confusing. I wasn't aware
> > segment.ms=0
> > >>>> ==
> > >>>> > > > > > disabled,
> > >>>> > > > > > > > > but I
> > >>>> > > > > > > > > >> > > think
> > >>>> > > > > > > > > >> > > > > that is also unintuitive.
> > >>>> > > > > > > > > >> > > >
> > >>>> > > > > > > > > >> > > > I think there is an argument for keeping
> > these
> > >>>> two
> > >>>> > > > > > > > configurations
> > >>>> > > > > > > > > >> > > > consistent, since they are so similar. I
> > agree
> > >>>> that
> > >>>> > 0
> > >>>> > > > was an
> > >>>> > > > > > > > > >> > unfortunate
> > >>>> > > > > > > > > >> > > > choice.,
> > >>>> > > > > > > > > >> > > >
> > >>>> > > > > > > > > >> > > > best,
> > >>>> > > > > > > > > >> > > > Colin
> > >>>> > > > > > > > > >> > > >
> > >>>> > > > > > > > > >> > > > >
> > >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM Colin
> > McCabe
> > >>>> <
> > >>>> > > > > > > > > cmccabe@apache.org>
> > >>>> > > > > > > > > >> > > wrote:
> > >>>> > > > > > > > > >> > > > >
> > >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47, xiongqi
> wu
> > >>>> wrote:
> > >>>> > > > > > > > > >> > > > > > > Colin,
> > >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> > >>>> > > > > > > > > >> > > > > > > see my inline reply below.
> > >>>> > > > > > > > > >> > > > > > >
> > >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> > >>>> > > > > > > > > >> > > > > > >
> > >>>> > > > > > > > > >> > > > > > >
> > >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 PM Colin
> > >>>> McCabe <
> > >>>> > > > > > > > > >> cmccabe@apache.org>
> > >>>> > > > > > > > > >> > > > wrote:
> > >>>> > > > > > > > > >> > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > The name seems a bit ambiguous. Our
> > >>>> > compaction
> > >>>> > > > > > > policies
> > >>>> > > > > > > > > are
> > >>>> > > > > > > > > >> > > already
> > >>>> > > > > > > > > >> > > > > > > > time-based, after all. It seems
> like
> > >>>> this
> > >>>> > > > change is
> > >>>> > > > > > > > > focused
> > >>>> > > > > > > > > >> > > around
> > >>>> > > > > > > > > >> > > > > > adding
> > >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms." Perhaps
> > >>>> the KIP
> > >>>> > > > title
> > >>>> > > > > > > should
> > >>>> > > > > > > > > be
> > >>>> > > > > > > > > >> > > > something
> > >>>> > > > > > > > > >> > > > > > > > like "add maximum compaction lag
> > time"?
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will change the
> > >>>> title.
> > >>>> > > > > > > > > >> > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > The active segment is forced to
> roll
> > >>>> when
> > >>>> > > > either "
> > >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> > >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (log.roll.ms and
> > >>>> > > > log.roll.hours)
> > >>>> > > > > > > has
> > >>>> > > > > > > > > >> > reached.
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > If the max.compaction.lag.ms is
> low,
> > >>>> it
> > >>>> > seems
> > >>>> > > > like
> > >>>> > > > > > > > > segments
> > >>>> > > > > > > > > >> > will
> > >>>> > > > > > > > > >> > > > be
> > >>>> > > > > > > > > >> > > > > > > > rolled very frequently. This can
> be a
> > >>>> > source of
> > >>>> > > > > > > problems
> > >>>> > > > > > > > > in
> > >>>> > > > > > > > > >> the
> > >>>> > > > > > > > > >> > > > > > cluster,
> > >>>> > > > > > > > > >> > > > > > > > since creating many different small
> > log
> > >>>> > segments
> > >>>> > > > > > > > consumes
> > >>>> > > > > > > > > a
> > >>>> > > > > > > > > >> > huge
> > >>>> > > > > > > > > >> > > > > > amount of
> > >>>> > > > > > > > > >> > > > > > > > cluster resources. Therefore, I
> would
> > >>>> > suggest
> > >>>> > > > > > adding a
> > >>>> > > > > > > > > >> > > broker-level
> > >>>> > > > > > > > > >> > > > > > > > configuration which allows us to
> set
> > a
> > >>>> > minimum
> > >>>> > > > value
> > >>>> > > > > > > for
> > >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If we let
> > >>>> users set
> > >>>> > it
> > >>>> > > > on a
> > >>>> > > > > > > > > >> per-topic
> > >>>> > > > > > > > > >> > > > basis,
> > >>>> > > > > > > > > >> > > > > > > > someone could set a value of 1 ms
> or
> > >>>> > something,
> > >>>> > > > and
> > >>>> > > > > > > > cause
> > >>>> > > > > > > > > >> > chaos.
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > =========> this applies to
> > segment.ms
> > >>>> as
> > >>>> > well.
> > >>>> > > > > > Today
> > >>>> > > > > > > > > users
> > >>>> > > > > > > > > >> can
> > >>>> > > > > > > > > >> > > > set "
> > >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low value, and
> > >>>> cause a
> > >>>> > > > frequent
> > >>>> > > > > > > > > >> rolling of
> > >>>> > > > > > > > > >> > > > active
> > >>>> > > > > > > > > >> > > > > > > segments.
> > >>>> > > > > > > > > >> > > > > >
> > >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> > >>>> > > > > > > > > >> > > > > >
> > >>>> > > > > > > > > >> > > > > > I agree that this is an existing
> problem
> > >>>> with
> > >>>> > > > > > segment.ms.
> > >>>> > > > > > > > > >> However,
> > >>>> > > > > > > > > >> > > > that
> > >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't fix it.
> As
> > >>>> you
> > >>>> > noted,
> > >>>> > > > > > there
> > >>>> > > > > > > > > will
> > >>>> > > > > > > > > >> be
> > >>>> > > > > > > > > >> > > more
> > >>>> > > > > > > > > >> > > > > > interest in these topic-level retention
> > >>>> > settings as
> > >>>> > > > a
> > >>>> > > > > > > result
> > >>>> > > > > > > > > of
> > >>>> > > > > > > > > >> > GDPR.
> > >>>> > > > > > > > > >> > > > It
> > >>>> > > > > > > > > >> > > > > > seems likely that pre-existing problems
> > >>>> will
> > >>>> > cause
> > >>>> > > > more
> > >>>> > > > > > > > > trouble.
> > >>>> > > > > > > > > >> > > > > >
> > >>>> > > > > > > > > >> > > > > > The fix seems relatively
> straightforward
> > >>>> here --
> > >>>> > > > add a
> > >>>> > > > > > > > > >> broker-level
> > >>>> > > > > > > > > >> > > > > > minimum segment.ms that overrides
> > >>>> per-topic
> > >>>> > > > minimums.
> > >>>> > > > > > We
> > >>>> > > > > > > > can
> > >>>> > > > > > > > > >> also
> > >>>> > > > > > > > > >> > > fail
> > >>>> > > > > > > > > >> > > > > > with a helpful error message when
> someone
> > >>>> > attempts
> > >>>> > > > to
> > >>>> > > > > > set
> > >>>> > > > > > > an
> > >>>> > > > > > > > > >> > invalid
> > >>>> > > > > > > > > >> > > > > > configuration.
> > >>>> > > > > > > > > >> > > > > >
> > >>>> > > > > > > > > >> > > > > > > In my option, the minimum of "
> > >>>> > > > max.compaction.lag.ms"
> > >>>> > > > > > > > should
> > >>>> > > > > > > > > >> be
> > >>>> > > > > > > > > >> > > > > > > based on the minimum of "segment.ms
> ".
> > >>>> Since
> > >>>> > > > today the
> > >>>> > > > > > > > > >> minimum of
> > >>>> > > > > > > > > >> > > > > > segment.ms
> > >>>> > > > > > > > > >> > > > > > > is 1, "max.compaction.lag.ms" also
> > >>>> starts
> > >>>> > with
> > >>>> > > > 1. "0"
> > >>>> > > > > > > > means
> > >>>> > > > > > > > > >> > > > disable. I
> > >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but it is hard
> > to
> > >>>> > define
> > >>>> > > > the
> > >>>> > > > > > > > meaning
> > >>>> > > > > > > > > >> of 0
> > >>>> > > > > > > > > >> > > > because
> > >>>> > > > > > > > > >> > > > > > we
> > >>>> > > > > > > > > >> > > > > > > cannot just roll the active segment
> > >>>> > immediately.
> > >>>> > > > > > > > > >> > > > > >
> > >>>> > > > > > > > > >> > > > > > That's a fair point. We should make 0 =
> > >>>> > disable, to
> > >>>> > > > be
> > >>>> > > > > > > > > >> consistent
> > >>>> > > > > > > > > >> > > with
> > >>>> > > > > > > > > >> > > > the
> > >>>> > > > > > > > > >> > > > > > other settings.
> > >>>> > > > > > > > > >> > > > > >
> > >>>> > > > > > > > > >> > > > > > best,
> > >>>> > > > > > > > > >> > > > > > Colin
> > >>>> > > > > > > > > >> > > > > >
> > >>>> > > > > > > > > >> > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > -- Note that an alternative
> > >>>> configuration
> > >>>> > is to
> > >>>> > > > use
> > >>>> > > > > > -1
> > >>>> > > > > > > > as
> > >>>> > > > > > > > > >> > > > "disabled"
> > >>>> > > > > > > > > >> > > > > > and
> > >>>> > > > > > > > > >> > > > > > > > 0
> > >>>> > > > > > > > > >> > > > > > > > > as "immediate compaction".
> Because
> > >>>> > compaction
> > >>>> > > > lag
> > >>>> > > > > > is
> > >>>> > > > > > > > > still
> > >>>> > > > > > > > > >> > > > determined
> > >>>> > > > > > > > > >> > > > > > > > > based on min.compaction.lag and
> how
> > >>>> long
> > >>>> > to
> > >>>> > > > roll
> > >>>> > > > > > an
> > >>>> > > > > > > > > active
> > >>>> > > > > > > > > >> > > > segment,
> > >>>> > > > > > > > > >> > > > > > > > the
> > >>>> > > > > > > > > >> > > > > > > > > actual lag for compaction is
> > >>>> undetermined
> > >>>> > if
> > >>>> > > > we
> > >>>> > > > > > use
> > >>>> > > > > > > > "0".
> > >>>> > > > > > > > > >> On
> > >>>> > > > > > > > > >> > the
> > >>>> > > > > > > > > >> > > > other
> > >>>> > > > > > > > > >> > > > > > > > > hand, we can already set
> > >>>> > > > > > "min.cleanable.dirty.ratio"
> > >>>> > > > > > > > to
> > >>>> > > > > > > > > >> > achieve
> > >>>> > > > > > > > > >> > > > the
> > >>>> > > > > > > > > >> > > > > > > > same
> > >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose "0" as
> > >>>> "disabled".
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be the invalid
> > >>>> setting.
> > >>>> > > > > > Treating
> > >>>> > > > > > > 0
> > >>>> > > > > > > > > >> > > differently
> > >>>> > > > > > > > > >> > > > > > than
> > >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > =====> see my previous comment, I
> am
> > >>>> not
> > >>>> > > > strongly
> > >>>> > > > > > > > against,
> > >>>> > > > > > > > > >> but
> > >>>> > > > > > > > > >> > 0
> > >>>> > > > > > > > > >> > > is
> > >>>> > > > > > > > > >> > > > > > not a
> > >>>> > > > > > > > > >> > > > > > > valid configuration in my option. So
> I
> > >>>> use
> > >>>> > "0" as
> > >>>> > > > > > > disabled
> > >>>> > > > > > > > > >> state.
> > >>>> > > > > > > > > >> > > > > > >
> > >>>> > > > > > > > > >> > > > > > > best,
> > >>>> > > > > > > > > >> > > > > > > > Colin
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at 15:04,
> > xiongqi
> > >>>> wu
> > >>>> > wrote:
> > >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
> > >>>> > > > > > > > > >> > > > > > > > > KIP:
> > >>>> > > > > > > > > >> > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > >>>> > > > > > > > <
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > >>>> > > > > > > > > >> <
> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > >>>> > > > > > > > <
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>
> > >>>> > > > > > > > > >> > > <
> > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > >>>> > > > > > > > <
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > >>>> > > > > > > > > >> <
> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > >>>> > > > > > > > <
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>>
> > >>>> > > > > > > > > >> > > > > > <
> > >>>> > > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > >>>> > > > > > > > <
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > >>>> > > > > > > > > >> <
> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > >>>> > > > > > > > <
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>
> > >>>> > > > > > > > > >> > > <
> > >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > >>>> > > > > > > > <
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> > >>>> > > > > > > > > >> <
> > >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> > >>>> > > > > > > > <
> > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> > >>>> > > > > > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> > >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>>>
> > >>>> > > > > > > > > >> > > > > > > > >
> > %3A+Time-based+log+compaction+policy
> > >>>> > > > > > > > > >> > > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > > Implementation:
> > >>>> > > > > > > > > >> > > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > >
> > >>>> https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > >>>> > > > > > > > > >> > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> > >>>> > > > > > > > > >> > > > > > <
> > https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> > >>>> > > > > > > > > >> > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>>
> > >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>
> > >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>
> > >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> > >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> > >>>> > > > > > > > > >> > > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > >
> > >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> > >>>> > > > > > > > > >> > > > > > > >
> > >>>> > > > > > > > > >> > > > > >
> > >>>> > > > > > > > > >> > > >
> > >>>> > > > > > > > > >> > >
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> > --
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> > Brett Rann
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> > Senior DevOps Engineer
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> > Zendesk International Ltd
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000
> Australia
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> > >>>> > > > > > > > > >> >
> > >>>> > > > > > > > > >>
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > --
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > Brett Rann
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > Senior DevOps Engineer
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > Zendesk International Ltd
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> > >>>> > > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > --
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Brett Rann
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Senior DevOps Engineer
> > >>>> > > > > > > > >
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Zendesk International Ltd
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
> > >>>> > > > > > > > >
> > >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> > >>>> > > > > > > > >
> > >>>> > > > > > > >
> > >>>> > > > > > >
> > >>>> > > > > >
> > >>>> > > >
> > >>>> >
> > >>>>
> > >>>
> >
>
Re: [VOTE] KIP-354 Time-based log compaction policy
Posted by Joel Koshy <jj...@gmail.com>.
+1 with one suggestion on the proposed metric. You should probably include
the unit. So for e.g., max-compaction-delay-secs.
Joel
On Tue, Nov 6, 2018 at 5:30 PM xiongqi wu <xi...@gmail.com> wrote:
> bump
> Xiongqi (Wesley) Wu
>
>
> On Thu, Sep 27, 2018 at 4:20 PM xiongqi wu <xi...@gmail.com> wrote:
>
> >
> > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for feedback.
> > Can I have more feedback or VOTE on this KIP?
> >
> >
> > Xiongqi (Wesley) Wu
> >
> >
> > On Wed, Sep 19, 2018 at 10:52 AM xiongqi wu <xi...@gmail.com> wrote:
> >
> >> Any other votes or comments?
> >>
> >> Xiongqi (Wesley) Wu
> >>
> >>
> >> On Tue, Sep 11, 2018 at 11:45 AM xiongqi wu <xi...@gmail.com>
> wrote:
> >>
> >>> Yes, more votes and code review.
> >>>
> >>> Xiongqi (Wesley) Wu
> >>>
> >>>
> >>> On Mon, Sep 10, 2018 at 11:37 PM Brett Rann <brann@zendesk.com.invalid
> >
> >>> wrote:
> >>>
> >>>> +1 (non binding) from on 0 then, and on the KIP.
> >>>>
> >>>> Where do we go from here? More votes?
> >>>>
> >>>> On Tue, Sep 11, 2018 at 5:34 AM Colin McCabe <cm...@apache.org>
> >>>> wrote:
> >>>>
> >>>> > On Mon, Sep 10, 2018, at 11:44, xiongqi wu wrote:
> >>>> > > Thank you for comments. I will use '0' for now.
> >>>> > >
> >>>> > > If we create topics through admin client in the future, we might
> >>>> perform
> >>>> > > some useful checks. (but the assumption is all brokers in the same
> >>>> > cluster
> >>>> > > have the same default configurations value, otherwise,it might
> >>>> still be
> >>>> > > tricky to do such cross validation check.)
> >>>> >
> >>>> > This isn't something that we might do in the future-- this is
> >>>> something we
> >>>> > are doing now. We already have Create Topic policies which are
> >>>> enforced by
> >>>> > the broker. Check KIP-108 and KIP-170 for details. This is one of
> the
> >>>> > motivations for getting rid of direct ZK access-- making sure that
> >>>> these
> >>>> > policies are applied.
> >>>> >
> >>>> > I agree that having different configurations on different brokers
> can
> >>>> be
> >>>> > confusing and frustrating . That's why more configurations are being
> >>>> made
> >>>> > dynamic using KIP-226. Dynamic configurations are stored centrally
> in
> >>>> ZK,
> >>>> > so they are the same on all brokers (modulo propagation delays). In
> >>>> any
> >>>> > case, this is a general issue, not specific to "create topics".
> >>>> >
> >>>> > cheers,
> >>>> > Colin
> >>>> >
> >>>> >
> >>>> > >
> >>>> > >
> >>>> > > Xiongqi (Wesley) Wu
> >>>> > >
> >>>> > >
> >>>> > > On Mon, Sep 10, 2018 at 11:15 AM Colin McCabe <cmccabe@apache.org
> >
> >>>> > wrote:
> >>>> > >
> >>>> > > > I don't have a strong opinion. But I think we should probably be
> >>>> > > > consistent with how segment.ms works, and just use 0.
> >>>> > > >
> >>>> > > > best,
> >>>> > > > Colin
> >>>> > > >
> >>>> > > >
> >>>> > > > On Wed, Sep 5, 2018, at 21:19, Brett Rann wrote:
> >>>> > > > > OK thanks for that clarification. I see why you're
> uncomfortable
> >>>> > with 0
> >>>> > > > now.
> >>>> > > > >
> >>>> > > > > I'm not really fussed. I just prefer consistency in
> >>>> configuration
> >>>> > > > options.
> >>>> > > > >
> >>>> > > > > Personally I lean towards treating 0 and 1 similarly in that
> >>>> > scenario,
> >>>> > > > > because it favours the person thinking about setting the
> >>>> > configurations,
> >>>> > > > > and a person doesn't care about a 1ms edge case especially
> when
> >>>> the
> >>>> > > > context
> >>>> > > > > is the true minimum is tied to the log cleaner cadence.
> >>>> > > > >
> >>>> > > > > Introducing 0 to mean "disabled" because there is some
> >>>> uniquness in
> >>>> > > > > segment.ms not being able to be set to 0, reduces
> configuration
> >>>> > > > consistency
> >>>> > > > > in favour of capturing a MS gap in an edge case that nobody
> >>>> would
> >>>> > ever
> >>>> > > > > notice. For someone to understand why everywhere else -1 is
> >>>> used to
> >>>> > > > > disable, but here 0 is used, they would need to learn about
> >>>> > segment.ms
> >>>> > > > > having a 1ms minimum and then after learning would think "who
> >>>> cares
> >>>> > about
> >>>> > > > > 1ms?" in this context. I would anyway :)
> >>>> > > > >
> >>>> > > > > my 2c anyway. Will again defer to majority. Curious which way
> >>>> Colin
> >>>> > falls
> >>>> > > > > now.
> >>>> > > > >
> >>>> > > > > Don't want to spend more time on this though, It's well into
> >>>> > > > bikeshedding
> >>>> > > > > territory now. :)
> >>>> > > > >
> >>>> > > > >
> >>>> > > > >
> >>>> > > > > On Thu, Sep 6, 2018 at 1:31 PM xiongqi wu <
> xiongqiwu@gmail.com>
> >>>> > wrote:
> >>>> > > > >
> >>>> > > > > > I want to honor the minimum value of segment.ms (which is
> >>>> 1ms) to
> >>>> > > > force
> >>>> > > > > > roll an active segment.
> >>>> > > > > > So if we set "max.compaction.lag.ms" any value > 0, the
> >>>> minimum of
> >>>> > > > > > max.compaction.lag.ms and segment.ms will be used to seal
> an
> >>>> > active
> >>>> > > > > > segment.
> >>>> > > > > >
> >>>> > > > > > If we set max.compaction.lag.ms to 0, the current
> >>>> implementation
> >>>> > will
> >>>> > > > > > treat it as disabled.
> >>>> > > > > >
> >>>> > > > > > It is a little bit weird to treat max.compaction.lag=0 the
> >>>> same as
> >>>> > > > > > max.compaction.lag=1.
> >>>> > > > > >
> >>>> > > > > > There might be a reason why we set the minimum of
> segment.ms
> >>>> to 1,
> >>>> > > > and I
> >>>> > > > > > don't want to break this assumption.
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > > Xiongqi (Wesley) Wu
> >>>> > > > > >
> >>>> > > > > >
> >>>> > > > > > On Wed, Sep 5, 2018 at 7:54 PM Brett Rann
> >>>> > <br...@zendesk.com.invalid>
> >>>> > > > > > wrote:
> >>>> > > > > >
> >>>> > > > > > > You're rolling a new segment if the condition is met
> right?
> >>>> So
> >>>> > I'm
> >>>> > > > > > > struggling to understand the relevance of segment.ms
> here.
> >>>> > Maybe an
> >>>> > > > > > > example
> >>>> > > > > > > would help my understanding:
> >>>> > > > > > >
> >>>> > > > > > > segment.ms=9999999
> >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> >>>> > > > > > > max.compaction.lag.ms=1
> >>>> > > > > > >
> >>>> > > > > > > When a duplicate message comes in, after 1ms the topic
> >>>> should be
> >>>> > > > eligible
> >>>> > > > > > > for compaction when the log compaction thread gets around
> to
> >>>> > > > evaluating
> >>>> > > > > > the
> >>>> > > > > > > topic.
> >>>> > > > > > >
> >>>> > > > > > > if we have
> >>>> > > > > > > segment.ms=9999999
> >>>> > > > > > > *min.cleanable.dirty.ratio=1*
> >>>> > > > > > > max.compaction.lag.ms=0
> >>>> > > > > > >
> >>>> > > > > > > When a duplicate message comes in, after 0ms the topic
> >>>> should be
> >>>> > > > eligible
> >>>> > > > > > > for compaction when the log compaction thread gets around
> to
> >>>> > > > evaluating
> >>>> > > > > > the
> >>>> > > > > > > topic.
> >>>> > > > > > >
> >>>> > > > > > > In both of those cases the change would mean a new segment
> >>>> is
> >>>> > rolled
> >>>> > > > so
> >>>> > > > > > the
> >>>> > > > > > > new message would be part of the compaction task. 0 and 1
> >>>> are
> >>>> > > > practically
> >>>> > > > > > > the same meaning since neither is providing an actual
> >>>> guarantee
> >>>> > at
> >>>> > > > such
> >>>> > > > > > low
> >>>> > > > > > > MS settings, but effectively tying it to both the
> frequency
> >>>> of
> >>>> > the
> >>>> > > > log
> >>>> > > > > > > cleaner running and the priority of the given topic being
> >>>> the
> >>>> > highest
> >>>> > > > > > > priority of all topics that are evaluated for cleaning on
> >>>> the
> >>>> > next
> >>>> > > > cycle.
> >>>> > > > > > > You've captured that nuance with careful "skipped" wording
> >>>> in
> >>>> > the KIP
> >>>> > > > > > > here "controls
> >>>> > > > > > > the max time interval a message/segment can be skipped for
> >>>> log
> >>>> > > > > > compaction".
> >>>> > > > > > >
> >>>> > > > > > > How is 0 different to 1, practically? And how is it
> >>>> relating to
> >>>> > > > > > segment.ms
> >>>> > > > > > > ?
> >>>> > > > > > > Is it that you're proposing to have 0 mean "use
> segment.ms
> >>>> > > > instead?" as
> >>>> > > > > > a
> >>>> > > > > > > kind of third option?
> >>>> > > > > > >
> >>>> > > > > > >
> >>>> > > > > > >
> >>>> > > > > > > On Thu, Sep 6, 2018 at 11:34 AM xiongqi wu <
> >>>> xiongqiwu@gmail.com>
> >>>> > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > > To make it clear,
> >>>> > > > > > > > I don't against using -1 as disabled, but we need to
> come
> >>>> up
> >>>> > with
> >>>> > > > the
> >>>> > > > > > > > meaning of "0".
> >>>> > > > > > > > If "0" means immediate compaction, but the actual
> >>>> compaction
> >>>> > lag
> >>>> > > > will
> >>>> > > > > > be
> >>>> > > > > > > > segment.ms.
> >>>> > > > > > > > It has longer lag than setting the value to be half of
> >>>> > segment.ms.
> >>>> > > > > > > > We cannot provide "0" as max compaction lag.
> >>>> > > > > > > >
> >>>> > > > > > > > Here are two options.
> >>>> > > > > > > > Option 1:
> >>>> > > > > > > > Keep 0 as disabled
> >>>> > > > > > > > Option 2:
> >>>> > > > > > > > -1 (disabled), 0 (max compaction lag = segment.ms), and
> >>>> > others.
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > > Xiongqi (Wesley) Wu
> >>>> > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > > > On Wed, Sep 5, 2018 at 5:49 PM Brett Rann
> >>>> > > > <br...@zendesk.com.invalid>
> >>>> > > > > > > > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > > > -1 is consistent as "special" with these settings for
> >>>> > example:
> >>>> > > > > > > > >
> >>>> > > > > > > > > log.retention.bytes
> >>>> > > > > > > > > socket.received.buffer.bytes
> >>>> > > > > > > > > socket.send.buffer.bytes
> >>>> > > > > > > > > queued.max.request.bytes
> >>>> > > > > > > > > retention.bytes
> >>>> > > > > > > > > retention.ms
> >>>> > > > > > > > >
> >>>> > > > > > > > > and acks.
> >>>> > > > > > > > >
> >>>> > > > > > > > > Where it may mean no limit, use OS defaults, max
> (acks),
> >>>> > etc. I
> >>>> > > > don't
> >>>> > > > > > > see
> >>>> > > > > > > > > much convention of 0 meaning those things.
> >>>> > > > > > > > >
> >>>> > > > > > > > > There are some NULLs but it seems convetion there is
> >>>> NULL is
> >>>> > used
> >>>> > > > > > where
> >>>> > > > > > > > > there's another setting in the hierarchy that would be
> >>>> used
> >>>> > > > instead.
> >>>> > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > > > > On Thu, Sep 6, 2018 at 10:42 AM Brett Rann <
> >>>> > brann@zendesk.com>
> >>>> > > > > > wrote:
> >>>> > > > > > > > >
> >>>> > > > > > > > > > If segment.ms can't be set to 0, then we're not
> being
> >>>> > > > consistent
> >>>> > > > > > > > > > by using 0 for this new setting? I throw out -1 for
> >>>> > > > consideration
> >>>> > > > > > > > > > again :)
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > On Thu, Sep 6, 2018 at 10:03 AM xiongqi wu <
> >>>> > > > xiongqiwu@gmail.com>
> >>>> > > > > > > > wrote:
> >>>> > > > > > > > > >
> >>>> > > > > > > > > >> Thanks. I will document after PR is merged.
> >>>> > > > > > > > > >>
> >>>> > > > > > > > > >> BTW, Kafka enforce the minimum of "segment.ms" to
> >>>> 1, we
> >>>> > > > cannot
> >>>> > > > > > set
> >>>> > > > > > > "
> >>>> > > > > > > > > >> segment.ms" to 0.
> >>>> > > > > > > > > >>
> >>>> > > > > > > > > >> I also updated the title of this KIP.
> >>>> > > > > > > > > >>
> >>>> > > > > > > > > >> Xiongqi (Wesley) Wu
> >>>> > > > > > > > > >>
> >>>> > > > > > > > > >>
> >>>> > > > > > > > > >> On Wed, Sep 5, 2018 at 4:34 PM Brett Rann
> >>>> > > > > > <brann@zendesk.com.invalid
> >>>> > > > > > > >
> >>>> > > > > > > > > >> wrote:
> >>>> > > > > > > > > >>
> >>>> > > > > > > > > >> > I withdraw my comments on -1 since i'm in the
> >>>> minority.
> >>>> > :)
> >>>> > > > Can
> >>>> > > > > > we
> >>>> > > > > > > > > >> > make sure 0 gets documented as meaning disabled
> >>>> here:
> >>>> > > > > > > > > >> >
> >>>> https://kafka.apache.org/documentation/#brokerconfigs
> >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> >>>> > > > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>
> >>>> > > > > > > > > >> <
> >>>> https://kafka.apache.org/documentation/#brokerconfigs
> >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>
> >>>> > > > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>
> >>>> > > > > > <https://kafka.apache.org/documentation/#brokerconfigs
> >>>> > <https://kafka.apache.org/documentation/#brokerconfigs>>>> ?
> >>>> > > > > > > > > >> > And while there it would be good if segment.ms
> is
> >>>> > > > documented
> >>>> > > > > > > > > >> > that 0 is disabled too. (there's some hierarchy
> of
> >>>> > configs
> >>>> > > > for
> >>>> > > > > > > that
> >>>> > > > > > > > > too
> >>>> > > > > > > > > >> > if its not set and null for others means
> disabled!)
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> > On Thu, Sep 6, 2018 at 4:44 AM xiongqi wu <
> >>>> > > > xiongqiwu@gmail.com>
> >>>> > > > > > > > > wrote:
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> > > If we use 0 to indicate immediate compaction,
> the
> >>>> > > > compaction
> >>>> > > > > > lag
> >>>> > > > > > > > is
> >>>> > > > > > > > > >> > > determined by segment.ms in worst case. If
> >>>> segment.ms
> >>>> > is
> >>>> > > > 24
> >>>> > > > > > > > hours,
> >>>> > > > > > > > > >> > > "immediate compaction" is a weaker guarantee
> than
> >>>> > setting
> >>>> > > > any
> >>>> > > > > > > > value
> >>>> > > > > > > > > >> less
> >>>> > > > > > > > > >> > > than 24 hours. By the definition of "max
> >>>> compaction
> >>>> > lag",
> >>>> > > > we
> >>>> > > > > > > > cannot
> >>>> > > > > > > > > >> have
> >>>> > > > > > > > > >> > > zero lag. So I use 0 to indicate "disable".
> >>>> > > > > > > > > >> > >
> >>>> > > > > > > > > >> > >
> >>>> > > > > > > > > >> > >
> >>>> > > > > > > > > >> > > Xiongqi (Wesley) Wu
> >>>> > > > > > > > > >> > >
> >>>> > > > > > > > > >> > >
> >>>> > > > > > > > > >> > > On Wed, Sep 5, 2018 at 8:34 AM Colin McCabe <
> >>>> > > > > > cmccabe@apache.org
> >>>> > > > > > > >
> >>>> > > > > > > > > >> wrote:
> >>>> > > > > > > > > >> > >
> >>>> > > > > > > > > >> > > > On Tue, Sep 4, 2018, at 22:11, Brett Rann
> >>>> wrote:
> >>>> > > > > > > > > >> > > > > > That's a fair point. We should make 0 =
> >>>> > disable, to
> >>>> > > > be
> >>>> > > > > > > > > >> consistent
> >>>> > > > > > > > > >> > > with
> >>>> > > > > > > > > >> > > > > the other settings.
> >>>> > > > > > > > > >> > > > >
> >>>> > > > > > > > > >> > > > > -1 is used elsewhere for disable and when
> >>>> seeing
> >>>> > it
> >>>> > > > in a
> >>>> > > > > > > > config
> >>>> > > > > > > > > >> it's
> >>>> > > > > > > > > >> > > > clear
> >>>> > > > > > > > > >> > > > > that it's a special meaning. 0 doesn't have
> >>>> to
> >>>> > mean
> >>>> > > > > > instant,
> >>>> > > > > > > > it
> >>>> > > > > > > > > >> just
> >>>> > > > > > > > > >> > > > means
> >>>> > > > > > > > > >> > > > > as quickly as possible. I don't think 0 is
> >>>> > intuitive
> >>>> > > > for
> >>>> > > > > > > > > disabled
> >>>> > > > > > > > > >> and
> >>>> > > > > > > > > >> > > it
> >>>> > > > > > > > > >> > > > > will be confusing. I wasn't aware
> segment.ms=0
> >>>> ==
> >>>> > > > > > disabled,
> >>>> > > > > > > > > but I
> >>>> > > > > > > > > >> > > think
> >>>> > > > > > > > > >> > > > > that is also unintuitive.
> >>>> > > > > > > > > >> > > >
> >>>> > > > > > > > > >> > > > I think there is an argument for keeping
> these
> >>>> two
> >>>> > > > > > > > configurations
> >>>> > > > > > > > > >> > > > consistent, since they are so similar. I
> agree
> >>>> that
> >>>> > 0
> >>>> > > > was an
> >>>> > > > > > > > > >> > unfortunate
> >>>> > > > > > > > > >> > > > choice.,
> >>>> > > > > > > > > >> > > >
> >>>> > > > > > > > > >> > > > best,
> >>>> > > > > > > > > >> > > > Colin
> >>>> > > > > > > > > >> > > >
> >>>> > > > > > > > > >> > > > >
> >>>> > > > > > > > > >> > > > > On Wed, Sep 5, 2018 at 11:38 AM Colin
> McCabe
> >>>> <
> >>>> > > > > > > > > cmccabe@apache.org>
> >>>> > > > > > > > > >> > > wrote:
> >>>> > > > > > > > > >> > > > >
> >>>> > > > > > > > > >> > > > > > On Tue, Sep 4, 2018, at 17:47, xiongqi wu
> >>>> wrote:
> >>>> > > > > > > > > >> > > > > > > Colin,
> >>>> > > > > > > > > >> > > > > > > Thank you for comments.
> >>>> > > > > > > > > >> > > > > > > see my inline reply below.
> >>>> > > > > > > > > >> > > > > > >
> >>>> > > > > > > > > >> > > > > > > Xiongqi (Wesley) Wu
> >>>> > > > > > > > > >> > > > > > >
> >>>> > > > > > > > > >> > > > > > >
> >>>> > > > > > > > > >> > > > > > > On Tue, Sep 4, 2018 at 5:24 PM Colin
> >>>> McCabe <
> >>>> > > > > > > > > >> cmccabe@apache.org>
> >>>> > > > > > > > > >> > > > wrote:
> >>>> > > > > > > > > >> > > > > > >
> >>>> > > > > > > > > >> > > > > > > > Hi Xiongqi,
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > Thanks for this KIP.
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > The name seems a bit ambiguous. Our
> >>>> > compaction
> >>>> > > > > > > policies
> >>>> > > > > > > > > are
> >>>> > > > > > > > > >> > > already
> >>>> > > > > > > > > >> > > > > > > > time-based, after all. It seems like
> >>>> this
> >>>> > > > change is
> >>>> > > > > > > > > focused
> >>>> > > > > > > > > >> > > around
> >>>> > > > > > > > > >> > > > > > adding
> >>>> > > > > > > > > >> > > > > > > > a “max.compaction.lag.ms." Perhaps
> >>>> the KIP
> >>>> > > > title
> >>>> > > > > > > should
> >>>> > > > > > > > > be
> >>>> > > > > > > > > >> > > > something
> >>>> > > > > > > > > >> > > > > > > > like "add maximum compaction lag
> time"?
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > ==========> sure. I will change the
> >>>> title.
> >>>> > > > > > > > > >> > > > > > >
> >>>> > > > > > > > > >> > > > > > > > The active segment is forced to roll
> >>>> when
> >>>> > > > either "
> >>>> > > > > > > > > >> > > > > > max.compaction.lag.ms"
> >>>> > > > > > > > > >> > > > > > > > > or "segment.ms" (log.roll.ms and
> >>>> > > > log.roll.hours)
> >>>> > > > > > > has
> >>>> > > > > > > > > >> > reached.
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > If the max.compaction.lag.ms is low,
> >>>> it
> >>>> > seems
> >>>> > > > like
> >>>> > > > > > > > > segments
> >>>> > > > > > > > > >> > will
> >>>> > > > > > > > > >> > > > be
> >>>> > > > > > > > > >> > > > > > > > rolled very frequently. This can be a
> >>>> > source of
> >>>> > > > > > > problems
> >>>> > > > > > > > > in
> >>>> > > > > > > > > >> the
> >>>> > > > > > > > > >> > > > > > cluster,
> >>>> > > > > > > > > >> > > > > > > > since creating many different small
> log
> >>>> > segments
> >>>> > > > > > > > consumes
> >>>> > > > > > > > > a
> >>>> > > > > > > > > >> > huge
> >>>> > > > > > > > > >> > > > > > amount of
> >>>> > > > > > > > > >> > > > > > > > cluster resources. Therefore, I would
> >>>> > suggest
> >>>> > > > > > adding a
> >>>> > > > > > > > > >> > > broker-level
> >>>> > > > > > > > > >> > > > > > > > configuration which allows us to set
> a
> >>>> > minimum
> >>>> > > > value
> >>>> > > > > > > for
> >>>> > > > > > > > > >> > > > > > > > max.compaction.lag.ms. If we let
> >>>> users set
> >>>> > it
> >>>> > > > on a
> >>>> > > > > > > > > >> per-topic
> >>>> > > > > > > > > >> > > > basis,
> >>>> > > > > > > > > >> > > > > > > > someone could set a value of 1 ms or
> >>>> > something,
> >>>> > > > and
> >>>> > > > > > > > cause
> >>>> > > > > > > > > >> > chaos.
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > =========> this applies to
> segment.ms
> >>>> as
> >>>> > well.
> >>>> > > > > > Today
> >>>> > > > > > > > > users
> >>>> > > > > > > > > >> can
> >>>> > > > > > > > > >> > > > set "
> >>>> > > > > > > > > >> > > > > > > segment.ms" to a very low value, and
> >>>> cause a
> >>>> > > > frequent
> >>>> > > > > > > > > >> rolling of
> >>>> > > > > > > > > >> > > > active
> >>>> > > > > > > > > >> > > > > > > segments.
> >>>> > > > > > > > > >> > > > > >
> >>>> > > > > > > > > >> > > > > > Hi Xiongqi,
> >>>> > > > > > > > > >> > > > > >
> >>>> > > > > > > > > >> > > > > > I agree that this is an existing problem
> >>>> with
> >>>> > > > > > segment.ms.
> >>>> > > > > > > > > >> However,
> >>>> > > > > > > > > >> > > > that
> >>>> > > > > > > > > >> > > > > > doesn't mean that we shouldn't fix it. As
> >>>> you
> >>>> > noted,
> >>>> > > > > > there
> >>>> > > > > > > > > will
> >>>> > > > > > > > > >> be
> >>>> > > > > > > > > >> > > more
> >>>> > > > > > > > > >> > > > > > interest in these topic-level retention
> >>>> > settings as
> >>>> > > > a
> >>>> > > > > > > result
> >>>> > > > > > > > > of
> >>>> > > > > > > > > >> > GDPR.
> >>>> > > > > > > > > >> > > > It
> >>>> > > > > > > > > >> > > > > > seems likely that pre-existing problems
> >>>> will
> >>>> > cause
> >>>> > > > more
> >>>> > > > > > > > > trouble.
> >>>> > > > > > > > > >> > > > > >
> >>>> > > > > > > > > >> > > > > > The fix seems relatively straightforward
> >>>> here --
> >>>> > > > add a
> >>>> > > > > > > > > >> broker-level
> >>>> > > > > > > > > >> > > > > > minimum segment.ms that overrides
> >>>> per-topic
> >>>> > > > minimums.
> >>>> > > > > > We
> >>>> > > > > > > > can
> >>>> > > > > > > > > >> also
> >>>> > > > > > > > > >> > > fail
> >>>> > > > > > > > > >> > > > > > with a helpful error message when someone
> >>>> > attempts
> >>>> > > > to
> >>>> > > > > > set
> >>>> > > > > > > an
> >>>> > > > > > > > > >> > invalid
> >>>> > > > > > > > > >> > > > > > configuration.
> >>>> > > > > > > > > >> > > > > >
> >>>> > > > > > > > > >> > > > > > > In my option, the minimum of "
> >>>> > > > max.compaction.lag.ms"
> >>>> > > > > > > > should
> >>>> > > > > > > > > >> be
> >>>> > > > > > > > > >> > > > > > > based on the minimum of "segment.ms".
> >>>> Since
> >>>> > > > today the
> >>>> > > > > > > > > >> minimum of
> >>>> > > > > > > > > >> > > > > > segment.ms
> >>>> > > > > > > > > >> > > > > > > is 1, "max.compaction.lag.ms" also
> >>>> starts
> >>>> > with
> >>>> > > > 1. "0"
> >>>> > > > > > > > means
> >>>> > > > > > > > > >> > > > disable. I
> >>>> > > > > > > > > >> > > > > > > can use -1 as disable, but it is hard
> to
> >>>> > define
> >>>> > > > the
> >>>> > > > > > > > meaning
> >>>> > > > > > > > > >> of 0
> >>>> > > > > > > > > >> > > > because
> >>>> > > > > > > > > >> > > > > > we
> >>>> > > > > > > > > >> > > > > > > cannot just roll the active segment
> >>>> > immediately.
> >>>> > > > > > > > > >> > > > > >
> >>>> > > > > > > > > >> > > > > > That's a fair point. We should make 0 =
> >>>> > disable, to
> >>>> > > > be
> >>>> > > > > > > > > >> consistent
> >>>> > > > > > > > > >> > > with
> >>>> > > > > > > > > >> > > > the
> >>>> > > > > > > > > >> > > > > > other settings.
> >>>> > > > > > > > > >> > > > > >
> >>>> > > > > > > > > >> > > > > > best,
> >>>> > > > > > > > > >> > > > > > Colin
> >>>> > > > > > > > > >> > > > > >
> >>>> > > > > > > > > >> > > > > > >
> >>>> > > > > > > > > >> > > > > > > > -- Note that an alternative
> >>>> configuration
> >>>> > is to
> >>>> > > > use
> >>>> > > > > > -1
> >>>> > > > > > > > as
> >>>> > > > > > > > > >> > > > "disabled"
> >>>> > > > > > > > > >> > > > > > and
> >>>> > > > > > > > > >> > > > > > > > 0
> >>>> > > > > > > > > >> > > > > > > > > as "immediate compaction". Because
> >>>> > compaction
> >>>> > > > lag
> >>>> > > > > > is
> >>>> > > > > > > > > still
> >>>> > > > > > > > > >> > > > determined
> >>>> > > > > > > > > >> > > > > > > > > based on min.compaction.lag and how
> >>>> long
> >>>> > to
> >>>> > > > roll
> >>>> > > > > > an
> >>>> > > > > > > > > active
> >>>> > > > > > > > > >> > > > segment,
> >>>> > > > > > > > > >> > > > > > > > the
> >>>> > > > > > > > > >> > > > > > > > > actual lag for compaction is
> >>>> undetermined
> >>>> > if
> >>>> > > > we
> >>>> > > > > > use
> >>>> > > > > > > > "0".
> >>>> > > > > > > > > >> On
> >>>> > > > > > > > > >> > the
> >>>> > > > > > > > > >> > > > other
> >>>> > > > > > > > > >> > > > > > > > > hand, we can already set
> >>>> > > > > > "min.cleanable.dirty.ratio"
> >>>> > > > > > > > to
> >>>> > > > > > > > > >> > achieve
> >>>> > > > > > > > > >> > > > the
> >>>> > > > > > > > > >> > > > > > > > same
> >>>> > > > > > > > > >> > > > > > > > > goal. So here we choose "0" as
> >>>> "disabled".
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > I would prefer -1 to be the invalid
> >>>> setting.
> >>>> > > > > > Treating
> >>>> > > > > > > 0
> >>>> > > > > > > > > >> > > differently
> >>>> > > > > > > > > >> > > > > > than
> >>>> > > > > > > > > >> > > > > > > > 1 seems strange to me.
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > =====> see my previous comment, I am
> >>>> not
> >>>> > > > strongly
> >>>> > > > > > > > against,
> >>>> > > > > > > > > >> but
> >>>> > > > > > > > > >> > 0
> >>>> > > > > > > > > >> > > is
> >>>> > > > > > > > > >> > > > > > not a
> >>>> > > > > > > > > >> > > > > > > valid configuration in my option. So I
> >>>> use
> >>>> > "0" as
> >>>> > > > > > > disabled
> >>>> > > > > > > > > >> state.
> >>>> > > > > > > > > >> > > > > > >
> >>>> > > > > > > > > >> > > > > > > best,
> >>>> > > > > > > > > >> > > > > > > > Colin
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > On Tue, Sep 4, 2018, at 15:04,
> xiongqi
> >>>> wu
> >>>> > wrote:
> >>>> > > > > > > > > >> > > > > > > > > Let's VOTE for this KIP.
> >>>> > > > > > > > > >> > > > > > > > > KIP:
> >>>> > > > > > > > > >> > > > > > > > >
> >>>> > > > > > > > >
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> >>>> > > > > > > > <
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> >>>> > > > > > > > > >> <
> >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> >>>> > > > > > > > <
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>
> >>>> > > > > > > > > >> > > <
> >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> >>>> > > > > > > > <
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> >>>> > > > > > > > > >> <
> >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> >>>> > > > > > > > <
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>>
> >>>> > > > > > > > > >> > > > > > <
> >>>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> >>>> > > > > > > > <
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> >>>> > > > > > > > > >> <
> >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> >>>> > > > > > > > <
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>
> >>>> > > > > > > > > >> > > <
> >>>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> >>>> > > > > > > > <
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>
> >>>> > > > > > > > > >> <
> >>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>
> >>>> > > > > > > > <
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>
> >>>> > > > > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354
> >>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-354>>>>>>
> >>>> > > > > > > > > >> > > > > > > > >
> %3A+Time-based+log+compaction+policy
> >>>> > > > > > > > > >> > > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > > Implementation:
> >>>> > > > > > > > > >> > > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > >
> >>>> https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>
> >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>>
> >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>
> >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> >>>> > > > > > > > > >> > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>
> >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>>
> >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>
> >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>>>>
> >>>> > > > > > > > > >> > > > > > <
> https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>
> >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>>
> >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>
> >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>>>
> >>>> > > > > > > > > >> > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>
> >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>>
> >>>> > > > > > > > > >> <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>
> >>>> > > > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>
> >>>> > > > > > <https://github.com/apache/kafka/pull/5611
> >>>> > <https://github.com/apache/kafka/pull/5611>>>>>>
> >>>> > > > > > > > > >> > > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > >
> >>>> > > > > > > > > >> > > > > > > > > Xiongqi (Wesley) Wu
> >>>> > > > > > > > > >> > > > > > > >
> >>>> > > > > > > > > >> > > > > >
> >>>> > > > > > > > > >> > > >
> >>>> > > > > > > > > >> > >
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> > --
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> > Brett Rann
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> > Senior DevOps Engineer
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> > Zendesk International Ltd
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> > 395 Collins Street, Melbourne VIC 3000 Australia
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >> > Mobile: +61 (0) 418 826 017
> >>>> > > > > > > > > >> >
> >>>> > > > > > > > > >>
> >>>> > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > --
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > Brett Rann
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > Senior DevOps Engineer
> >>>> > > > > > > > > >
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > Zendesk International Ltd
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
> >>>> > > > > > > > > >
> >>>> > > > > > > > > > Mobile: +61 (0) 418 826 017
> >>>> > > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > > > > --
> >>>> > > > > > > > >
> >>>> > > > > > > > > Brett Rann
> >>>> > > > > > > > >
> >>>> > > > > > > > > Senior DevOps Engineer
> >>>> > > > > > > > >
> >>>> > > > > > > > >
> >>>> > > > > > > > > Zendesk International Ltd
> >>>> > > > > > > > >
> >>>> > > > > > > > > 395 Collins Street, Melbourne VIC 3000 Australia
> >>>> > > > > > > > >
> >>>> > > > > > > > > Mobile: +61 (0) 418 826 017
> >>>> > > > > > > > >
> >>>> > > > > > > >
> >>>> > > > > > >
> >>>> > > > > >
> >>>> > > >
> >>>> >
> >>>>
> >>>
>