You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Chia-Ping Tsai <ch...@apache.org> on 2026/02/03 19:14:50 UTC
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
hi Jun
I agree that we should apply reasonable strategies for different `breaking changes`. This makes ensuring that the initialization process sees the 'latest' configs even more critical. It is otherwise hard to guarantee that the broker fails at the appropriate time when a `fail-fast` policy is intended
Best,
Chia-Ping
On 2026/01/30 22:13:00 Jun Rao via dev wrote:
> Hi, Chia-Ping,
>
> Thanks for following up on this.
>
> To me, there are two categories of changes. The first type involves changes
> that always benefit the user. An example is segment.bytes since there is no
> good reason for a user to ever set it to less than 1MB. In this case, one
> option is to automatically clamp the value to the new lower bound without
> failing. This is probably best for users because they get better
> configuration and compatibility. This is essentially the approach that we
> followed in KIP-1161 for handling duplicates in the List config. The second
> category includes changes where disallowed values may impact a user's
> intention. An example is segment.ms (change not implemented yet, but
> targeted for 5.0). A user may intentionally set it to a really small value
> to control the retention time. In this case, it's probably better to fail
> the broker with an invalid config error so that the users know about it.
>
> Jun
>
>
> On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > hi all
> >
> > There was a long discussion about compatibility after increasing the lower
> > bound (https://github.com/apache/kafka/pull/20334#discussion_r2538011530).
> >
> > In summary, increasing the lower bound results in the following issues:
> >
> > 1) Static configurations created before KIP-1030 cause the broker to fail.
> > For example, setting log.segment.bytes=1000 will now encounter a validation
> > error due to the new lower bound when starting the updated broker.
> >
> > 2) Dynamic configurations created before KIP-1030 cannot be applied due to
> > validation errors.
> >
> > Personally, I believe users should be explicitly aware of breaking
> > changes. Therefore, I suggest forcing the broker to fail when initializing
> > the metadata publisher if the dynamic configuration cannot be applied.
> >
> > A softer approach is to use warnings instead of fatal errors. The broker
> > would proceed smoothly, but users might be unaware of the breaking changes,
> > and the broker would run with unexpected configurations.
> >
> > Any feedback is welcome.
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/11/18 10:13:41 Divij Vaidya wrote:
> > > Hey folks
> > >
> > > With 4.0, we have an opportunity to reset the default values and add
> > > constraints in the configurations based on our learnings since 3.0.
> > >
> > > Here's a KIP which modifies defaults for some properties and modifies the
> > > constraints for a few others.
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > >
> > >
> > > Looking forward for your feedback.
> > >
> > > (Previous discussion thread on this topic -
> > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp )
> > >
> >
>
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
Posted by Chia-Ping Tsai <ch...@apache.org>.
Ah, I see your point now. The controller has no visibility into the static configurations of other nodes.
On 2026/02/17 19:28:08 Jun Rao via dev wrote:
> Hi, Chia-Ping,
>
> Thanks for the reply.
>
> I am not sure about the MV approach. For example, a user could set
> `segment.bytes` through a static config file. How will MV work in that case?
>
> Jun
>
> On Sun, Feb 15, 2026 at 12:17 AM Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > Hi, Jun
> >
> > I prefer to ensure users are aware of what happens with their
> > configuration. Clamping seems to hide the actual behaviour from users,
> > which may lead to potential issues.
> >
> > Here is a crazy idea: what if the validation is bound to the MV?
> >
> > For example, `segment.bytes` could have different lower bounds for 4.0 and
> > 4.1. This allows users to safely keep the smaller value while upgrading the
> > binary files. However, they must update the configurations to meet the new
> > requirement before bumping the MV. Otherwise, they will receive an error
> > when calling `updateFeatures`
> >
> > This approach not only offers a smooth upgrade path bug also ensures the
> > final cluster state adheres to the strict validation rules once the MV is
> > updated.
> >
> > WDYT?
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/02/09 17:08:19 Jun Rao via dev wrote:
> > > Hi, Chia-Ping,
> > >
> > > Thanks for the reply. Do you agree that it's better to use the clamping
> > > approach for segment.bytes ?
> > >
> > > Jun
> > >
> > > On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > >
> > > > hi Jun
> > > >
> > > > I agree that we should apply reasonable strategies for different
> > `breaking
> > > > changes`. This makes ensuring that the initialization process sees the
> > > > 'latest' configs even more critical. It is otherwise hard to guarantee
> > that
> > > > the broker fails at the appropriate time when a `fail-fast` policy is
> > > > intended
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2026/01/30 22:13:00 Jun Rao via dev wrote:
> > > > > Hi, Chia-Ping,
> > > > >
> > > > > Thanks for following up on this.
> > > > >
> > > > > To me, there are two categories of changes. The first type involves
> > > > changes
> > > > > that always benefit the user. An example is segment.bytes since
> > there is
> > > > no
> > > > > good reason for a user to ever set it to less than 1MB. In this
> > case, one
> > > > > option is to automatically clamp the value to the new lower bound
> > without
> > > > > failing. This is probably best for users because they get better
> > > > > configuration and compatibility. This is essentially the approach
> > that we
> > > > > followed in KIP-1161 for handling duplicates in the List config. The
> > > > second
> > > > > category includes changes where disallowed values may impact a user's
> > > > > intention. An example is segment.ms (change not implemented yet, but
> > > > > targeted for 5.0). A user may intentionally set it to a really small
> > > > value
> > > > > to control the retention time. In this case, it's probably better to
> > fail
> > > > > the broker with an invalid config error so that the users know about
> > it.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <chia7712@apache.org
> > >
> > > > wrote:
> > > > >
> > > > > > hi all
> > > > > >
> > > > > > There was a long discussion about compatibility after increasing
> > the
> > > > lower
> > > > > > bound (
> > > > https://github.com/apache/kafka/pull/20334#discussion_r2538011530).
> > > > > >
> > > > > > In summary, increasing the lower bound results in the following
> > issues:
> > > > > >
> > > > > > 1) Static configurations created before KIP-1030 cause the broker
> > to
> > > > fail.
> > > > > > For example, setting log.segment.bytes=1000 will now encounter a
> > > > validation
> > > > > > error due to the new lower bound when starting the updated broker.
> > > > > >
> > > > > > 2) Dynamic configurations created before KIP-1030 cannot be applied
> > > > due to
> > > > > > validation errors.
> > > > > >
> > > > > > Personally, I believe users should be explicitly aware of breaking
> > > > > > changes. Therefore, I suggest forcing the broker to fail when
> > > > initializing
> > > > > > the metadata publisher if the dynamic configuration cannot be
> > applied.
> > > > > >
> > > > > > A softer approach is to use warnings instead of fatal errors. The
> > > > broker
> > > > > > would proceed smoothly, but users might be unaware of the breaking
> > > > changes,
> > > > > > and the broker would run with unexpected configurations.
> > > > > >
> > > > > > Any feedback is welcome.
> > > > > >
> > > > > > Best,
> > > > > > Chia-Ping
> > > > > >
> > > > > > On 2024/11/18 10:13:41 Divij Vaidya wrote:
> > > > > > > Hey folks
> > > > > > >
> > > > > > > With 4.0, we have an opportunity to reset the default values and
> > add
> > > > > > > constraints in the configurations based on our learnings since
> > 3.0.
> > > > > > >
> > > > > > > Here's a KIP which modifies defaults for some properties and
> > > > modifies the
> > > > > > > constraints for a few others.
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > > > > > >
> > > > > > >
> > > > > > > Looking forward for your feedback.
> > > > > > >
> > > > > > > (Previous discussion thread on this topic -
> > > > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp
> > )
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
Posted by Chia-Ping Tsai <ch...@gmail.com>.
If we implement the topic-level validation first, we will run into the same backward compatibility issues. I will start a KIP shortly to address the validation constraints holistically
> Jun Rao via dev <de...@kafka.apache.org> 於 2026年2月28日 凌晨2:27 寫道：
>
> Hi, Chia-Ping,
>
> Thanks for the proposal. It could work. Do you want to flesh that out as a
> separate KIP?
>
> What's your suggestion for https://issues.apache.org/jira/browse/KAFKA-20107?
> Should we implement it before or after your proposal is in place?
>
> Thanks,
>
> Jun
>
>
>
>> On Thu, Feb 19, 2026 at 12:03 AM Chia-Ping Tsai <ch...@apache.org> wrote:
>>
>> A rough idea is that we could add a tagged field to the broker
>> registration to contain the static properties that have validators in their
>> config definition. The goal is to empower the controller to do more
>> validation before approving the upgrade
>>
>>> On 2026/02/17 19:28:08 Jun Rao via dev wrote:
>>> Hi, Chia-Ping,
>>>
>>> Thanks for the reply.
>>>
>>> I am not sure about the MV approach. For example, a user could set
>>> `segment.bytes` through a static config file. How will MV work in that
>> case?
>>>
>>> Jun
>>>
>>> On Sun, Feb 15, 2026 at 12:17 AM Chia-Ping Tsai <ch...@apache.org>
>> wrote:
>>>
>>>> Hi, Jun
>>>>
>>>> I prefer to ensure users are aware of what happens with their
>>>> configuration. Clamping seems to hide the actual behaviour from users,
>>>> which may lead to potential issues.
>>>>
>>>> Here is a crazy idea: what if the validation is bound to the MV?
>>>>
>>>> For example, `segment.bytes` could have different lower bounds for 4.0
>> and
>>>> 4.1. This allows users to safely keep the smaller value while
>> upgrading the
>>>> binary files. However, they must update the configurations to meet the
>> new
>>>> requirement before bumping the MV. Otherwise, they will receive an
>> error
>>>> when calling `updateFeatures`
>>>>
>>>> This approach not only offers a smooth upgrade path bug also ensures
>> the
>>>> final cluster state adheres to the strict validation rules once the MV
>> is
>>>> updated.
>>>>
>>>> WDYT?
>>>>
>>>> Best,
>>>> Chia-Ping
>>>>
>>>> On 2026/02/09 17:08:19 Jun Rao via dev wrote:
>>>>> Hi, Chia-Ping,
>>>>>
>>>>> Thanks for the reply. Do you agree that it's better to use the
>> clamping
>>>>> approach for segment.bytes ?
>>>>>
>>>>> Jun
>>>>>
>>>>> On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <ch...@apache.org>
>>>> wrote:
>>>>>
>>>>>> hi Jun
>>>>>>
>>>>>> I agree that we should apply reasonable strategies for different
>>>> `breaking
>>>>>> changes`. This makes ensuring that the initialization process sees
>> the
>>>>>> 'latest' configs even more critical. It is otherwise hard to
>> guarantee
>>>> that
>>>>>> the broker fails at the appropriate time when a `fail-fast` policy
>> is
>>>>>> intended
>>>>>>
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>>
>>>>>> On 2026/01/30 22:13:00 Jun Rao via dev wrote:
>>>>>>> Hi, Chia-Ping,
>>>>>>>
>>>>>>> Thanks for following up on this.
>>>>>>>
>>>>>>> To me, there are two categories of changes. The first type
>> involves
>>>>>> changes
>>>>>>> that always benefit the user. An example is segment.bytes since
>>>> there is
>>>>>> no
>>>>>>> good reason for a user to ever set it to less than 1MB. In this
>>>> case, one
>>>>>>> option is to automatically clamp the value to the new lower bound
>>>> without
>>>>>>> failing. This is probably best for users because they get better
>>>>>>> configuration and compatibility. This is essentially the approach
>>>> that we
>>>>>>> followed in KIP-1161 for handling duplicates in the List config.
>> The
>>>>>> second
>>>>>>> category includes changes where disallowed values may impact a
>> user's
>>>>>>> intention. An example is segment.ms (change not implemented
>> yet, but
>>>>>>> targeted for 5.0). A user may intentionally set it to a really
>> small
>>>>>> value
>>>>>>> to control the retention time. In this case, it's probably
>> better to
>>>> fail
>>>>>>> the broker with an invalid config error so that the users know
>> about
>>>> it.
>>>>>>>
>>>>>>> Jun
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <
>> chia7712@apache.org
>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>>> hi all
>>>>>>>>
>>>>>>>> There was a long discussion about compatibility after
>> increasing
>>>> the
>>>>>> lower
>>>>>>>> bound (
>>>>>> https://github.com/apache/kafka/pull/20334#discussion_r2538011530
>> ).
>>>>>>>>
>>>>>>>> In summary, increasing the lower bound results in the following
>>>> issues:
>>>>>>>>
>>>>>>>> 1) Static configurations created before KIP-1030 cause the
>> broker
>>>> to
>>>>>> fail.
>>>>>>>> For example, setting log.segment.bytes=1000 will now encounter
>> a
>>>>>> validation
>>>>>>>> error due to the new lower bound when starting the updated
>> broker.
>>>>>>>>
>>>>>>>> 2) Dynamic configurations created before KIP-1030 cannot be
>> applied
>>>>>> due to
>>>>>>>> validation errors.
>>>>>>>>
>>>>>>>> Personally, I believe users should be explicitly aware of
>> breaking
>>>>>>>> changes. Therefore, I suggest forcing the broker to fail when
>>>>>> initializing
>>>>>>>> the metadata publisher if the dynamic configuration cannot be
>>>> applied.
>>>>>>>>
>>>>>>>> A softer approach is to use warnings instead of fatal errors.
>> The
>>>>>> broker
>>>>>>>> would proceed smoothly, but users might be unaware of the
>> breaking
>>>>>> changes,
>>>>>>>> and the broker would run with unexpected configurations.
>>>>>>>>
>>>>>>>> Any feedback is welcome.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>>
>>>>>>>> On 2024/11/18 10:13:41 Divij Vaidya wrote:
>>>>>>>>> Hey folks
>>>>>>>>>
>>>>>>>>> With 4.0, we have an opportunity to reset the default values
>> and
>>>> add
>>>>>>>>> constraints in the configurations based on our learnings
>> since
>>>> 3.0.
>>>>>>>>>
>>>>>>>>> Here's a KIP which modifies defaults for some properties and
>>>>>> modifies the
>>>>>>>>> constraints for a few others.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Looking forward for your feedback.
>>>>>>>>>
>>>>>>>>> (Previous discussion thread on this topic -
>>>>>>>>>
>> https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp
>>>> )
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping,
Thanks for the proposal. It could work. Do you want to flesh that out as a
separate KIP?
What's your suggestion for https://issues.apache.org/jira/browse/KAFKA-20107?
Should we implement it before or after your proposal is in place?
Thanks,
Jun
On Thu, Feb 19, 2026 at 12:03 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> A rough idea is that we could add a tagged field to the broker
> registration to contain the static properties that have validators in their
> config definition. The goal is to empower the controller to do more
> validation before approving the upgrade
>
> On 2026/02/17 19:28:08 Jun Rao via dev wrote:
> > Hi, Chia-Ping,
> >
> > Thanks for the reply.
> >
> > I am not sure about the MV approach. For example, a user could set
> > `segment.bytes` through a static config file. How will MV work in that
> case?
> >
> > Jun
> >
> > On Sun, Feb 15, 2026 at 12:17 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >
> > > Hi, Jun
> > >
> > > I prefer to ensure users are aware of what happens with their
> > > configuration. Clamping seems to hide the actual behaviour from users,
> > > which may lead to potential issues.
> > >
> > > Here is a crazy idea: what if the validation is bound to the MV?
> > >
> > > For example, `segment.bytes` could have different lower bounds for 4.0
> and
> > > 4.1. This allows users to safely keep the smaller value while
> upgrading the
> > > binary files. However, they must update the configurations to meet the
> new
> > > requirement before bumping the MV. Otherwise, they will receive an
> error
> > > when calling `updateFeatures`
> > >
> > > This approach not only offers a smooth upgrade path bug also ensures
> the
> > > final cluster state adheres to the strict validation rules once the MV
> is
> > > updated.
> > >
> > > WDYT?
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2026/02/09 17:08:19 Jun Rao via dev wrote:
> > > > Hi, Chia-Ping,
> > > >
> > > > Thanks for the reply. Do you agree that it's better to use the
> clamping
> > > > approach for segment.bytes ?
> > > >
> > > > Jun
> > > >
> > > > On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > > >
> > > > > hi Jun
> > > > >
> > > > > I agree that we should apply reasonable strategies for different
> > > `breaking
> > > > > changes`. This makes ensuring that the initialization process sees
> the
> > > > > 'latest' configs even more critical. It is otherwise hard to
> guarantee
> > > that
> > > > > the broker fails at the appropriate time when a `fail-fast` policy
> is
> > > > > intended
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > On 2026/01/30 22:13:00 Jun Rao via dev wrote:
> > > > > > Hi, Chia-Ping,
> > > > > >
> > > > > > Thanks for following up on this.
> > > > > >
> > > > > > To me, there are two categories of changes. The first type
> involves
> > > > > changes
> > > > > > that always benefit the user. An example is segment.bytes since
> > > there is
> > > > > no
> > > > > > good reason for a user to ever set it to less than 1MB. In this
> > > case, one
> > > > > > option is to automatically clamp the value to the new lower bound
> > > without
> > > > > > failing. This is probably best for users because they get better
> > > > > > configuration and compatibility. This is essentially the approach
> > > that we
> > > > > > followed in KIP-1161 for handling duplicates in the List config.
> The
> > > > > second
> > > > > > category includes changes where disallowed values may impact a
> user's
> > > > > > intention. An example is segment.ms (change not implemented
> yet, but
> > > > > > targeted for 5.0). A user may intentionally set it to a really
> small
> > > > > value
> > > > > > to control the retention time. In this case, it's probably
> better to
> > > fail
> > > > > > the broker with an invalid config error so that the users know
> about
> > > it.
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <
> chia7712@apache.org
> > > >
> > > > > wrote:
> > > > > >
> > > > > > > hi all
> > > > > > >
> > > > > > > There was a long discussion about compatibility after
> increasing
> > > the
> > > > > lower
> > > > > > > bound (
> > > > > https://github.com/apache/kafka/pull/20334#discussion_r2538011530
> ).
> > > > > > >
> > > > > > > In summary, increasing the lower bound results in the following
> > > issues:
> > > > > > >
> > > > > > > 1) Static configurations created before KIP-1030 cause the
> broker
> > > to
> > > > > fail.
> > > > > > > For example, setting log.segment.bytes=1000 will now encounter
> a
> > > > > validation
> > > > > > > error due to the new lower bound when starting the updated
> broker.
> > > > > > >
> > > > > > > 2) Dynamic configurations created before KIP-1030 cannot be
> applied
> > > > > due to
> > > > > > > validation errors.
> > > > > > >
> > > > > > > Personally, I believe users should be explicitly aware of
> breaking
> > > > > > > changes. Therefore, I suggest forcing the broker to fail when
> > > > > initializing
> > > > > > > the metadata publisher if the dynamic configuration cannot be
> > > applied.
> > > > > > >
> > > > > > > A softer approach is to use warnings instead of fatal errors.
> The
> > > > > broker
> > > > > > > would proceed smoothly, but users might be unaware of the
> breaking
> > > > > changes,
> > > > > > > and the broker would run with unexpected configurations.
> > > > > > >
> > > > > > > Any feedback is welcome.
> > > > > > >
> > > > > > > Best,
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > > On 2024/11/18 10:13:41 Divij Vaidya wrote:
> > > > > > > > Hey folks
> > > > > > > >
> > > > > > > > With 4.0, we have an opportunity to reset the default values
> and
> > > add
> > > > > > > > constraints in the configurations based on our learnings
> since
> > > 3.0.
> > > > > > > >
> > > > > > > > Here's a KIP which modifies defaults for some properties and
> > > > > modifies the
> > > > > > > > constraints for a few others.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > > > > > > >
> > > > > > > >
> > > > > > > > Looking forward for your feedback.
> > > > > > > >
> > > > > > > > (Previous discussion thread on this topic -
> > > > > > > >
> https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp
> > > )
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
Posted by Chia-Ping Tsai <ch...@apache.org>.
A rough idea is that we could add a tagged field to the broker registration to contain the static properties that have validators in their config definition. The goal is to empower the controller to do more validation before approving the upgrade
On 2026/02/17 19:28:08 Jun Rao via dev wrote:
> Hi, Chia-Ping,
>
> Thanks for the reply.
>
> I am not sure about the MV approach. For example, a user could set
> `segment.bytes` through a static config file. How will MV work in that case?
>
> Jun
>
> On Sun, Feb 15, 2026 at 12:17 AM Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > Hi, Jun
> >
> > I prefer to ensure users are aware of what happens with their
> > configuration. Clamping seems to hide the actual behaviour from users,
> > which may lead to potential issues.
> >
> > Here is a crazy idea: what if the validation is bound to the MV?
> >
> > For example, `segment.bytes` could have different lower bounds for 4.0 and
> > 4.1. This allows users to safely keep the smaller value while upgrading the
> > binary files. However, they must update the configurations to meet the new
> > requirement before bumping the MV. Otherwise, they will receive an error
> > when calling `updateFeatures`
> >
> > This approach not only offers a smooth upgrade path bug also ensures the
> > final cluster state adheres to the strict validation rules once the MV is
> > updated.
> >
> > WDYT?
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/02/09 17:08:19 Jun Rao via dev wrote:
> > > Hi, Chia-Ping,
> > >
> > > Thanks for the reply. Do you agree that it's better to use the clamping
> > > approach for segment.bytes ?
> > >
> > > Jun
> > >
> > > On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > >
> > > > hi Jun
> > > >
> > > > I agree that we should apply reasonable strategies for different
> > `breaking
> > > > changes`. This makes ensuring that the initialization process sees the
> > > > 'latest' configs even more critical. It is otherwise hard to guarantee
> > that
> > > > the broker fails at the appropriate time when a `fail-fast` policy is
> > > > intended
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2026/01/30 22:13:00 Jun Rao via dev wrote:
> > > > > Hi, Chia-Ping,
> > > > >
> > > > > Thanks for following up on this.
> > > > >
> > > > > To me, there are two categories of changes. The first type involves
> > > > changes
> > > > > that always benefit the user. An example is segment.bytes since
> > there is
> > > > no
> > > > > good reason for a user to ever set it to less than 1MB. In this
> > case, one
> > > > > option is to automatically clamp the value to the new lower bound
> > without
> > > > > failing. This is probably best for users because they get better
> > > > > configuration and compatibility. This is essentially the approach
> > that we
> > > > > followed in KIP-1161 for handling duplicates in the List config. The
> > > > second
> > > > > category includes changes where disallowed values may impact a user's
> > > > > intention. An example is segment.ms (change not implemented yet, but
> > > > > targeted for 5.0). A user may intentionally set it to a really small
> > > > value
> > > > > to control the retention time. In this case, it's probably better to
> > fail
> > > > > the broker with an invalid config error so that the users know about
> > it.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <chia7712@apache.org
> > >
> > > > wrote:
> > > > >
> > > > > > hi all
> > > > > >
> > > > > > There was a long discussion about compatibility after increasing
> > the
> > > > lower
> > > > > > bound (
> > > > https://github.com/apache/kafka/pull/20334#discussion_r2538011530).
> > > > > >
> > > > > > In summary, increasing the lower bound results in the following
> > issues:
> > > > > >
> > > > > > 1) Static configurations created before KIP-1030 cause the broker
> > to
> > > > fail.
> > > > > > For example, setting log.segment.bytes=1000 will now encounter a
> > > > validation
> > > > > > error due to the new lower bound when starting the updated broker.
> > > > > >
> > > > > > 2) Dynamic configurations created before KIP-1030 cannot be applied
> > > > due to
> > > > > > validation errors.
> > > > > >
> > > > > > Personally, I believe users should be explicitly aware of breaking
> > > > > > changes. Therefore, I suggest forcing the broker to fail when
> > > > initializing
> > > > > > the metadata publisher if the dynamic configuration cannot be
> > applied.
> > > > > >
> > > > > > A softer approach is to use warnings instead of fatal errors. The
> > > > broker
> > > > > > would proceed smoothly, but users might be unaware of the breaking
> > > > changes,
> > > > > > and the broker would run with unexpected configurations.
> > > > > >
> > > > > > Any feedback is welcome.
> > > > > >
> > > > > > Best,
> > > > > > Chia-Ping
> > > > > >
> > > > > > On 2024/11/18 10:13:41 Divij Vaidya wrote:
> > > > > > > Hey folks
> > > > > > >
> > > > > > > With 4.0, we have an opportunity to reset the default values and
> > add
> > > > > > > constraints in the configurations based on our learnings since
> > 3.0.
> > > > > > >
> > > > > > > Here's a KIP which modifies defaults for some properties and
> > > > modifies the
> > > > > > > constraints for a few others.
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > > > > > >
> > > > > > >
> > > > > > > Looking forward for your feedback.
> > > > > > >
> > > > > > > (Previous discussion thread on this topic -
> > > > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp
> > )
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
Posted by Chia-Ping Tsai <ch...@apache.org>.
Hi Jun,
Your question is spot on.
If the MV approach is implemented, a configuration like segment.bytes could have multiple lower bounds. For instance: an absolute minimum of 1MB, a 2MB bound for MV 4.3, and a 3MB bound for MV 4.7.
During broker startup, if the MV is momentarily unknown, the absolute minimum (1MB) will be applied for validation.
If users have segment.bytes=1.5MB in their static files and are just upgrading the binaries, it will work fine. Since the MV hasn't been bumped yet, the active lower bound remains 1MB.
Later, if users try to update the MV from 4.2 to 4.3, the validation will use the new 2MB lower bound, causing the MV upgrade to fail. To proceed, users must update the configs dynamically to at least 2MB (which overrides the static config) before they can successfully bump the MV.
Best,
Chia-Ping
On 2026/02/17 19:28:08 Jun Rao via dev wrote:
> Hi, Chia-Ping,
>
> Thanks for the reply.
>
> I am not sure about the MV approach. For example, a user could set
> `segment.bytes` through a static config file. How will MV work in that case?
>
> Jun
>
> On Sun, Feb 15, 2026 at 12:17 AM Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > Hi, Jun
> >
> > I prefer to ensure users are aware of what happens with their
> > configuration. Clamping seems to hide the actual behaviour from users,
> > which may lead to potential issues.
> >
> > Here is a crazy idea: what if the validation is bound to the MV?
> >
> > For example, `segment.bytes` could have different lower bounds for 4.0 and
> > 4.1. This allows users to safely keep the smaller value while upgrading the
> > binary files. However, they must update the configurations to meet the new
> > requirement before bumping the MV. Otherwise, they will receive an error
> > when calling `updateFeatures`
> >
> > This approach not only offers a smooth upgrade path bug also ensures the
> > final cluster state adheres to the strict validation rules once the MV is
> > updated.
> >
> > WDYT?
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/02/09 17:08:19 Jun Rao via dev wrote:
> > > Hi, Chia-Ping,
> > >
> > > Thanks for the reply. Do you agree that it's better to use the clamping
> > > approach for segment.bytes ?
> > >
> > > Jun
> > >
> > > On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > >
> > > > hi Jun
> > > >
> > > > I agree that we should apply reasonable strategies for different
> > `breaking
> > > > changes`. This makes ensuring that the initialization process sees the
> > > > 'latest' configs even more critical. It is otherwise hard to guarantee
> > that
> > > > the broker fails at the appropriate time when a `fail-fast` policy is
> > > > intended
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2026/01/30 22:13:00 Jun Rao via dev wrote:
> > > > > Hi, Chia-Ping,
> > > > >
> > > > > Thanks for following up on this.
> > > > >
> > > > > To me, there are two categories of changes. The first type involves
> > > > changes
> > > > > that always benefit the user. An example is segment.bytes since
> > there is
> > > > no
> > > > > good reason for a user to ever set it to less than 1MB. In this
> > case, one
> > > > > option is to automatically clamp the value to the new lower bound
> > without
> > > > > failing. This is probably best for users because they get better
> > > > > configuration and compatibility. This is essentially the approach
> > that we
> > > > > followed in KIP-1161 for handling duplicates in the List config. The
> > > > second
> > > > > category includes changes where disallowed values may impact a user's
> > > > > intention. An example is segment.ms (change not implemented yet, but
> > > > > targeted for 5.0). A user may intentionally set it to a really small
> > > > value
> > > > > to control the retention time. In this case, it's probably better to
> > fail
> > > > > the broker with an invalid config error so that the users know about
> > it.
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <chia7712@apache.org
> > >
> > > > wrote:
> > > > >
> > > > > > hi all
> > > > > >
> > > > > > There was a long discussion about compatibility after increasing
> > the
> > > > lower
> > > > > > bound (
> > > > https://github.com/apache/kafka/pull/20334#discussion_r2538011530).
> > > > > >
> > > > > > In summary, increasing the lower bound results in the following
> > issues:
> > > > > >
> > > > > > 1) Static configurations created before KIP-1030 cause the broker
> > to
> > > > fail.
> > > > > > For example, setting log.segment.bytes=1000 will now encounter a
> > > > validation
> > > > > > error due to the new lower bound when starting the updated broker.
> > > > > >
> > > > > > 2) Dynamic configurations created before KIP-1030 cannot be applied
> > > > due to
> > > > > > validation errors.
> > > > > >
> > > > > > Personally, I believe users should be explicitly aware of breaking
> > > > > > changes. Therefore, I suggest forcing the broker to fail when
> > > > initializing
> > > > > > the metadata publisher if the dynamic configuration cannot be
> > applied.
> > > > > >
> > > > > > A softer approach is to use warnings instead of fatal errors. The
> > > > broker
> > > > > > would proceed smoothly, but users might be unaware of the breaking
> > > > changes,
> > > > > > and the broker would run with unexpected configurations.
> > > > > >
> > > > > > Any feedback is welcome.
> > > > > >
> > > > > > Best,
> > > > > > Chia-Ping
> > > > > >
> > > > > > On 2024/11/18 10:13:41 Divij Vaidya wrote:
> > > > > > > Hey folks
> > > > > > >
> > > > > > > With 4.0, we have an opportunity to reset the default values and
> > add
> > > > > > > constraints in the configurations based on our learnings since
> > 3.0.
> > > > > > >
> > > > > > > Here's a KIP which modifies defaults for some properties and
> > > > modifies the
> > > > > > > constraints for a few others.
> > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > > > > > >
> > > > > > >
> > > > > > > Looking forward for your feedback.
> > > > > > >
> > > > > > > (Previous discussion thread on this topic -
> > > > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp
> > )
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping,
Thanks for the reply.
I am not sure about the MV approach. For example, a user could set
`segment.bytes` through a static config file. How will MV work in that case?
Jun
On Sun, Feb 15, 2026 at 12:17 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> Hi, Jun
>
> I prefer to ensure users are aware of what happens with their
> configuration. Clamping seems to hide the actual behaviour from users,
> which may lead to potential issues.
>
> Here is a crazy idea: what if the validation is bound to the MV?
>
> For example, `segment.bytes` could have different lower bounds for 4.0 and
> 4.1. This allows users to safely keep the smaller value while upgrading the
> binary files. However, they must update the configurations to meet the new
> requirement before bumping the MV. Otherwise, they will receive an error
> when calling `updateFeatures`
>
> This approach not only offers a smooth upgrade path bug also ensures the
> final cluster state adheres to the strict validation rules once the MV is
> updated.
>
> WDYT?
>
> Best,
> Chia-Ping
>
> On 2026/02/09 17:08:19 Jun Rao via dev wrote:
> > Hi, Chia-Ping,
> >
> > Thanks for the reply. Do you agree that it's better to use the clamping
> > approach for segment.bytes ?
> >
> > Jun
> >
> > On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >
> > > hi Jun
> > >
> > > I agree that we should apply reasonable strategies for different
> `breaking
> > > changes`. This makes ensuring that the initialization process sees the
> > > 'latest' configs even more critical. It is otherwise hard to guarantee
> that
> > > the broker fails at the appropriate time when a `fail-fast` policy is
> > > intended
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2026/01/30 22:13:00 Jun Rao via dev wrote:
> > > > Hi, Chia-Ping,
> > > >
> > > > Thanks for following up on this.
> > > >
> > > > To me, there are two categories of changes. The first type involves
> > > changes
> > > > that always benefit the user. An example is segment.bytes since
> there is
> > > no
> > > > good reason for a user to ever set it to less than 1MB. In this
> case, one
> > > > option is to automatically clamp the value to the new lower bound
> without
> > > > failing. This is probably best for users because they get better
> > > > configuration and compatibility. This is essentially the approach
> that we
> > > > followed in KIP-1161 for handling duplicates in the List config. The
> > > second
> > > > category includes changes where disallowed values may impact a user's
> > > > intention. An example is segment.ms (change not implemented yet, but
> > > > targeted for 5.0). A user may intentionally set it to a really small
> > > value
> > > > to control the retention time. In this case, it's probably better to
> fail
> > > > the broker with an invalid config error so that the users know about
> it.
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <chia7712@apache.org
> >
> > > wrote:
> > > >
> > > > > hi all
> > > > >
> > > > > There was a long discussion about compatibility after increasing
> the
> > > lower
> > > > > bound (
> > > https://github.com/apache/kafka/pull/20334#discussion_r2538011530).
> > > > >
> > > > > In summary, increasing the lower bound results in the following
> issues:
> > > > >
> > > > > 1) Static configurations created before KIP-1030 cause the broker
> to
> > > fail.
> > > > > For example, setting log.segment.bytes=1000 will now encounter a
> > > validation
> > > > > error due to the new lower bound when starting the updated broker.
> > > > >
> > > > > 2) Dynamic configurations created before KIP-1030 cannot be applied
> > > due to
> > > > > validation errors.
> > > > >
> > > > > Personally, I believe users should be explicitly aware of breaking
> > > > > changes. Therefore, I suggest forcing the broker to fail when
> > > initializing
> > > > > the metadata publisher if the dynamic configuration cannot be
> applied.
> > > > >
> > > > > A softer approach is to use warnings instead of fatal errors. The
> > > broker
> > > > > would proceed smoothly, but users might be unaware of the breaking
> > > changes,
> > > > > and the broker would run with unexpected configurations.
> > > > >
> > > > > Any feedback is welcome.
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > On 2024/11/18 10:13:41 Divij Vaidya wrote:
> > > > > > Hey folks
> > > > > >
> > > > > > With 4.0, we have an opportunity to reset the default values and
> add
> > > > > > constraints in the configurations based on our learnings since
> 3.0.
> > > > > >
> > > > > > Here's a KIP which modifies defaults for some properties and
> > > modifies the
> > > > > > constraints for a few others.
> > > > > >
> > > > > >
> > > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > > > > >
> > > > > >
> > > > > > Looking forward for your feedback.
> > > > > >
> > > > > > (Previous discussion thread on this topic -
> > > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp
> )
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
Posted by Chia-Ping Tsai <ch...@apache.org>.
Hi, Jun
I prefer to ensure users are aware of what happens with their configuration. Clamping seems to hide the actual behaviour from users, which may lead to potential issues.
Here is a crazy idea: what if the validation is bound to the MV?
For example, `segment.bytes` could have different lower bounds for 4.0 and 4.1. This allows users to safely keep the smaller value while upgrading the binary files. However, they must update the configurations to meet the new requirement before bumping the MV. Otherwise, they will receive an error when calling `updateFeatures`
This approach not only offers a smooth upgrade path bug also ensures the final cluster state adheres to the strict validation rules once the MV is updated.
WDYT?
Best,
Chia-Ping
On 2026/02/09 17:08:19 Jun Rao via dev wrote:
> Hi, Chia-Ping,
>
> Thanks for the reply. Do you agree that it's better to use the clamping
> approach for segment.bytes ?
>
> Jun
>
> On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > hi Jun
> >
> > I agree that we should apply reasonable strategies for different `breaking
> > changes`. This makes ensuring that the initialization process sees the
> > 'latest' configs even more critical. It is otherwise hard to guarantee that
> > the broker fails at the appropriate time when a `fail-fast` policy is
> > intended
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/01/30 22:13:00 Jun Rao via dev wrote:
> > > Hi, Chia-Ping,
> > >
> > > Thanks for following up on this.
> > >
> > > To me, there are two categories of changes. The first type involves
> > changes
> > > that always benefit the user. An example is segment.bytes since there is
> > no
> > > good reason for a user to ever set it to less than 1MB. In this case, one
> > > option is to automatically clamp the value to the new lower bound without
> > > failing. This is probably best for users because they get better
> > > configuration and compatibility. This is essentially the approach that we
> > > followed in KIP-1161 for handling duplicates in the List config. The
> > second
> > > category includes changes where disallowed values may impact a user's
> > > intention. An example is segment.ms (change not implemented yet, but
> > > targeted for 5.0). A user may intentionally set it to a really small
> > value
> > > to control the retention time. In this case, it's probably better to fail
> > > the broker with an invalid config error so that the users know about it.
> > >
> > > Jun
> > >
> > >
> > > On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > >
> > > > hi all
> > > >
> > > > There was a long discussion about compatibility after increasing the
> > lower
> > > > bound (
> > https://github.com/apache/kafka/pull/20334#discussion_r2538011530).
> > > >
> > > > In summary, increasing the lower bound results in the following issues:
> > > >
> > > > 1) Static configurations created before KIP-1030 cause the broker to
> > fail.
> > > > For example, setting log.segment.bytes=1000 will now encounter a
> > validation
> > > > error due to the new lower bound when starting the updated broker.
> > > >
> > > > 2) Dynamic configurations created before KIP-1030 cannot be applied
> > due to
> > > > validation errors.
> > > >
> > > > Personally, I believe users should be explicitly aware of breaking
> > > > changes. Therefore, I suggest forcing the broker to fail when
> > initializing
> > > > the metadata publisher if the dynamic configuration cannot be applied.
> > > >
> > > > A softer approach is to use warnings instead of fatal errors. The
> > broker
> > > > would proceed smoothly, but users might be unaware of the breaking
> > changes,
> > > > and the broker would run with unexpected configurations.
> > > >
> > > > Any feedback is welcome.
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2024/11/18 10:13:41 Divij Vaidya wrote:
> > > > > Hey folks
> > > > >
> > > > > With 4.0, we have an opportunity to reset the default values and add
> > > > > constraints in the configurations based on our learnings since 3.0.
> > > > >
> > > > > Here's a KIP which modifies defaults for some properties and
> > modifies the
> > > > > constraints for a few others.
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > > > >
> > > > >
> > > > > Looking forward for your feedback.
> > > > >
> > > > > (Previous discussion thread on this topic -
> > > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp )
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-1030: Change constraints and default values for various configurations
Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping,
Thanks for the reply. Do you agree that it's better to use the clamping
approach for segment.bytes ?
Jun
On Tue, Feb 3, 2026 at 11:15 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> hi Jun
>
> I agree that we should apply reasonable strategies for different `breaking
> changes`. This makes ensuring that the initialization process sees the
> 'latest' configs even more critical. It is otherwise hard to guarantee that
> the broker fails at the appropriate time when a `fail-fast` policy is
> intended
>
> Best,
> Chia-Ping
>
> On 2026/01/30 22:13:00 Jun Rao via dev wrote:
> > Hi, Chia-Ping,
> >
> > Thanks for following up on this.
> >
> > To me, there are two categories of changes. The first type involves
> changes
> > that always benefit the user. An example is segment.bytes since there is
> no
> > good reason for a user to ever set it to less than 1MB. In this case, one
> > option is to automatically clamp the value to the new lower bound without
> > failing. This is probably best for users because they get better
> > configuration and compatibility. This is essentially the approach that we
> > followed in KIP-1161 for handling duplicates in the List config. The
> second
> > category includes changes where disallowed values may impact a user's
> > intention. An example is segment.ms (change not implemented yet, but
> > targeted for 5.0). A user may intentionally set it to a really small
> value
> > to control the retention time. In this case, it's probably better to fail
> > the broker with an invalid config error so that the users know about it.
> >
> > Jun
> >
> >
> > On Wed, Nov 19, 2025 at 11:18 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >
> > > hi all
> > >
> > > There was a long discussion about compatibility after increasing the
> lower
> > > bound (
> https://github.com/apache/kafka/pull/20334#discussion_r2538011530).
> > >
> > > In summary, increasing the lower bound results in the following issues:
> > >
> > > 1) Static configurations created before KIP-1030 cause the broker to
> fail.
> > > For example, setting log.segment.bytes=1000 will now encounter a
> validation
> > > error due to the new lower bound when starting the updated broker.
> > >
> > > 2) Dynamic configurations created before KIP-1030 cannot be applied
> due to
> > > validation errors.
> > >
> > > Personally, I believe users should be explicitly aware of breaking
> > > changes. Therefore, I suggest forcing the broker to fail when
> initializing
> > > the metadata publisher if the dynamic configuration cannot be applied.
> > >
> > > A softer approach is to use warnings instead of fatal errors. The
> broker
> > > would proceed smoothly, but users might be unaware of the breaking
> changes,
> > > and the broker would run with unexpected configurations.
> > >
> > > Any feedback is welcome.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2024/11/18 10:13:41 Divij Vaidya wrote:
> > > > Hey folks
> > > >
> > > > With 4.0, we have an opportunity to reset the default values and add
> > > > constraints in the configurations based on our learnings since 3.0.
> > > >
> > > > Here's a KIP which modifies defaults for some properties and
> modifies the
> > > > constraints for a few others.
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations
> > > >
> > > >
> > > > Looking forward for your feedback.
> > > >
> > > > (Previous discussion thread on this topic -
> > > > https://lists.apache.org/thread/3dx9mdmsqf8pko9xdmhks80k96g650zp )
> > > >
> > >
> >
>