You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Badai Aqrandista <ba...@confluent.io.INVALID> on 2021/07/06 21:23:04 UTC
[DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes
Hi all
I have just created KIP-760
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes).
I created this KIP because I have seen so many Kafka brokers crash due
to small segment.ms and/or segment.bytes.
Please let me know what you think.
--
Thanks,
Badai
Re: [DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes
Posted by Kamal Chandraprakash <ka...@gmail.com>.
Badai,
Thanks for the KIP! This is useful to avoid crashes in production scenarios.
If someone still wants to quickly roll the log segments in test env, then
they can
do it by configuring the index.interval.bytes and segment.index.bytes to
low value
to force the log roll.
--
Kamal
On Thu, Jul 8, 2021 at 4:30 AM Badai Aqrandista <ba...@confluent.io.invalid>
wrote:
> Ryanne/James
>
> Yes, the proposed values are kinda arbitrary. I just want to prevent
> users from setting segment.ms to 1 ms or segment.bytes to 1 byte,
> which will blow the file handles count and crash the broker.
>
> Because of that and James' input, I have updated the KIP to add broker
> configuration to set the minimum value. Every environment will need
> different minimum values. And they may need to be modified
> temporarily, like James' use case.
>
> Thank you for reviewing the KIP.
>
> Regards
> Badai
>
> On Thu, Jul 8, 2021 at 12:09 AM Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> > Badai, the KIP makes sense to me, but could you maybe add some evidence
> to
> > support your proposed values? I'm sure they come from experience but I
> > don't think they are obvious and so may appear arbitrary.
> >
> > Ryanne
> >
> >
> > On Tue, Jul 6, 2021, 4:23 PM Badai Aqrandista <badai@confluent.io.invalid
> >
> > wrote:
> >
> > > Hi all
> > >
> > > I have just created KIP-760
> > > (
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes
> > > ).
> > >
> > > I created this KIP because I have seen so many Kafka brokers crash due
> > > to small segment.ms and/or segment.bytes.
> > >
> > > Please let me know what you think.
> > >
> > > --
> > > Thanks,
> > > Badai
> > >
>
>
>
> --
> Thanks,
> Badai
>
Re: [DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes
Posted by Badai Aqrandista <ba...@confluent.io.INVALID>.
Ryanne/James
Yes, the proposed values are kinda arbitrary. I just want to prevent
users from setting segment.ms to 1 ms or segment.bytes to 1 byte,
which will blow the file handles count and crash the broker.
Because of that and James' input, I have updated the KIP to add broker
configuration to set the minimum value. Every environment will need
different minimum values. And they may need to be modified
temporarily, like James' use case.
Thank you for reviewing the KIP.
Regards
Badai
On Thu, Jul 8, 2021 at 12:09 AM Ryanne Dolan <ry...@gmail.com> wrote:
>
> Badai, the KIP makes sense to me, but could you maybe add some evidence to
> support your proposed values? I'm sure they come from experience but I
> don't think they are obvious and so may appear arbitrary.
>
> Ryanne
>
>
> On Tue, Jul 6, 2021, 4:23 PM Badai Aqrandista <ba...@confluent.io.invalid>
> wrote:
>
> > Hi all
> >
> > I have just created KIP-760
> > (
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes
> > ).
> >
> > I created this KIP because I have seen so many Kafka brokers crash due
> > to small segment.ms and/or segment.bytes.
> >
> > Please let me know what you think.
> >
> > --
> > Thanks,
> > Badai
> >
--
Thanks,
Badai
Re: [DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes
Posted by Ryanne Dolan <ry...@gmail.com>.
Badai, the KIP makes sense to me, but could you maybe add some evidence to
support your proposed values? I'm sure they come from experience but I
don't think they are obvious and so may appear arbitrary.
Ryanne
On Tue, Jul 6, 2021, 4:23 PM Badai Aqrandista <ba...@confluent.io.invalid>
wrote:
> Hi all
>
> I have just created KIP-760
> (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes
> ).
>
> I created this KIP because I have seen so many Kafka brokers crash due
> to small segment.ms and/or segment.bytes.
>
> Please let me know what you think.
>
> --
> Thanks,
> Badai
>
Re: [DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes
Posted by Badai Aqrandista <ba...@confluent.io.INVALID>.
James
Thank you for replying. I originally thought about adding two new
broker dynamic config to allow administrator to determine the
acceptable minimum:
min.topic.segment.ms
min.topic.segment.bytes
I put this in the "rejected alternatives" section because I think
adding two more configs to the broker would be too much trouble. Or is
it not?
Based on your use case, which is a valid use case, adding these
dynamic config would still allow your use case to work, with an
additional step of lowering the dynamic config temporarily.
What do you think?
Regards
Badai
On Wed, Jul 7, 2021 at 2:06 PM James Cheng <wu...@gmail.com> wrote:
>
> Badai,
>
> Thanks for the KIP.
>
> We sometimes want to force compaction on a topic. This might be because there is a bad record in the topic, and we want to force it to get deleted. The way we do this is, we set segment.ms to a small value and write a record, in order to force a segment roll. And we also set min.cleanable.dirty.ratio=0, in order to trigger compaction. It's rare that we need to do it, but it happens sometimes. This change would make it more difficult to do that. With this KIP, we would have to write up to 1MB of data before causing the segment roll, or wait an hour.
>
> Although come to think of it, if my goal is to trigger compaction, then I can just write my tombstone a couple thousand times. So maybe this KIP just makes it slightly more tedious, but doesn't make it impossible.
>
> Another use case is when we want to truncate a topic, so we set a small segment size and set retention to almost zero, which will allow Kafka to delete what is in the topic. For that, though, we could also use kafka-delete-records.sh, so this KIP would not have impact on that particular use case.
>
> -James
>
> > On Jul 6, 2021, at 2:23 PM, Badai Aqrandista <ba...@confluent.io.INVALID> wrote:
> >
> > Hi all
> >
> > I have just created KIP-760
> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes).
> >
> > I created this KIP because I have seen so many Kafka brokers crash due
> > to small segment.ms and/or segment.bytes.
> >
> > Please let me know what you think.
> >
> > --
> > Thanks,
> > Badai
>
--
Thanks,
Badai
Re: [DISCUSS] KIP-760: Increase minimum value of segment.ms and
segment.bytes
Posted by James Cheng <wu...@gmail.com>.
Badai,
Thanks for the KIP.
We sometimes want to force compaction on a topic. This might be because there is a bad record in the topic, and we want to force it to get deleted. The way we do this is, we set segment.ms to a small value and write a record, in order to force a segment roll. And we also set min.cleanable.dirty.ratio=0, in order to trigger compaction. It's rare that we need to do it, but it happens sometimes. This change would make it more difficult to do that. With this KIP, we would have to write up to 1MB of data before causing the segment roll, or wait an hour.
Although come to think of it, if my goal is to trigger compaction, then I can just write my tombstone a couple thousand times. So maybe this KIP just makes it slightly more tedious, but doesn't make it impossible.
Another use case is when we want to truncate a topic, so we set a small segment size and set retention to almost zero, which will allow Kafka to delete what is in the topic. For that, though, we could also use kafka-delete-records.sh, so this KIP would not have impact on that particular use case.
-James
> On Jul 6, 2021, at 2:23 PM, Badai Aqrandista <ba...@confluent.io.INVALID> wrote:
>
> Hi all
>
> I have just created KIP-760
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-760%3A+Increase+minimum+value+of+segment.ms+and+segment.bytes).
>
> I created this KIP because I have seen so many Kafka brokers crash due
> to small segment.ms and/or segment.bytes.
>
> Please let me know what you think.
>
> --
> Thanks,
> Badai