Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

Since ValidList is a public API, we need to maintain backward compatibility. Therefore, the isEmptyAllowed flag is necessary. 
We can update the signature of isNonEmpty() to remove the boolean parameter.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply.
> 
> Do we really need isEmptyAllowed? It's awkward since the method name is
> inNonEmpty. Also, it's not clear when it's set to true.
> 
> Thanks,
> 
> Jun
> 
> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hi Jun,
>> 
>> Thank you for pointing that out.
>> 
>> I have revised the KIP as follows:
>> 
>> “In this case, the change does not introduce new invalid configurations or
>> prevent any currently valid setup.
>> The main behavioral difference is that we now explicitly throw a
>> ConfigException  during the storage format validation phase
>> instead of relying on the current behavior.”
>> 
>> This reflects the correct behavior.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
>>> 
>>> Hi, Jiunn-Yang,
>>> 
>>> "The main behavioral difference introduced by this change is that a
>>> ConfigException  will now be thrown during the storage format validation
>>> phase, rather than during server startup."
>>> 
>>> It seems that currently formating the storage already fails if
>>> group.coordinator.rebalance.protocols and process.roles  are empty. It
>> just
>>> gets a different exception.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hi Jun, chia,
>>>> 
>>>> Thanks for your feedback.
>>>> 
>>>> I add a new section for this change:
>>>> 
>>>> - Validation for group.coordinator.rebalance.protocols and process.roles
>>>> will be
>>>> moved from the server startup phase to the storage format phase.
>>>> - For cleanup.policy, setting an empty list will now be considered
>> invalid
>>>> and will
>>>> result in a ConfigException during storage format phase.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
>>>>> 
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
>>>> process.roles
>>>>> configurations can't be empty right now. With this KIP, the user will
>>>>> probably get a different (and more accurate) exception. It will be
>> useful
>>>>> to document that.
>>>>> 
>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a user
>>>> leaves
>>>>> cleanup.policy empty, it's more likely to be a mistake than an
>> intention.
>>>>> If we automatically treat it as None, the user will never realize the
>>>>> mistake. Throwing an exception will let the user realize there is a
>>>> mistake.
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <ch...@gmail.com>
>>>> wrote:
>>>>> 
>>>>>> hi Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the KIP, and I understand that maintaining compatibility is
>>>>>> important.
>>>>>> 
>>>>>> However, according to the documentation, we don't allow an empty value
>>>> for
>>>>>> the cleanup.policy. Hence, should we consider throwing an exception
>> for
>>>> an
>>>>>> empty value in version 4.x? This could streamline the code and avoid
>> an
>>>>>> extra deprecation cycle.
>>>>>> 
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>> 
>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> Regarding "The none policy will not delete or compact any segments",
>> we
>>>>>>> should be more accurate. We won't delete segments based on
>>>>>>> log.retention.bytes/log.retention.ms, but we should continue to
>> delete
>>>>>>> segments based on log.local.retention.bytes/log.retention.ms.
>>>> Otherwise,
>>>>>>> we
>>>>>>> risk running out of local disk space when remote storage is enabled.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io> wrote:
>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> Q2. What about existing empty values for
>>>>>>>> group.coordinator.rebalance.protocols and process.roles during
>>>> upgrade?
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hello Jun,
>>>>>>>>> 
>>>>>>>>> Thanks for review this KIP.
>>>>>>>>> 
>>>>>>>>> Q1 & Q3:
>>>>>>>>> I’ve updated the method name accordingly and revised the
>>>>>> cleanup.policy
>>>>>>>>> documentation
>>>>>>>>> to clarify that the none policy cannot be used with any other
>> policy.
>>>>>>>>> 
>>>>>>>>> Q2:
>>>>>>>>> For users currently using an empty cleanup.policy, the approach is
>> to
>>>>>>>>> apply the none policy
>>>>>>>>> during the preProcessParsedConfig step. Additionally, a warning
>>>>>> message
>>>>>>>>> will be emitted to inform users
>>>>>>>>> of the upcoming change.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52 寫道：
>>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>> 
>>>>>>>>>> 1. It's fine to introduce a new value None for cleanup.policy. But
>>>>>> now
>>>>>>>>> not
>>>>>>>>>> all value combinations are valid. For example, None can't be used
>>>>>> with
>>>>>>>>>> Delete or Compact. It would be useful to document that.
>>>>>>>>>> 2. What's the behavior during upgrade when an existing config has
>> an
>>>>>>>>> empty
>>>>>>>>>> list.
>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check does. How
>>>>>>> about
>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello everyone,
>>>>>>>>>>> 
>>>>>>>>>>> I would like to start a discussion on KIP-1161: cleanup.policy
>>>>>>>>> shouldn't
>>>>>>>>>>> be empty
>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>> 
>>>>>>>>>>> This proposal aims to improve the cleanup.policy configuration.
>>>>>>>>> Currently,
>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>> Therefore, there are two proposed improvements:
>>>>>>>>>>> 1. Update ValidList to validate whether an empty list is allowed.
>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Mickael,

Thanks for this nice catch.

> One weirdness of cleanup.policy is that you can set it to
> "delete,delete,delete,compact,compact" for example. Setting delete or
> compact multiple times has no functional impact.


Currently, only the process.roles config checks for duplicate values in the list. For this type of configuration, 
there shouldn’t be a valid use case where the same value needs to be specified multiple times. Therefore, 
the check for duplicate entries could be deferred to the storage format validation stage instead. 

As for whether this behavior needs to be backward compatible — personally, I think such a configuration is 
invalid to begin with. So we might consider dropping support for it in the next release.

Best Regards,
Jiunn-Yang

> Mickael Maison <mi...@gmail.com> 於 2025年5月13日 凌晨12:47 寫道：
> 
> Hi,
> 
> Thanks for the KIP.
> 
> One weirdness of cleanup.policy is that you can set it to
> "delete,delete,delete,compact,compact" for example. Setting delete or
> compact multiple times has no functional impact.
> I don't think it's something that has to be addressed, but while
> you're looking at the configuration's validation I thought I'd bring
> it up.
> 
> Thanks,
> Mickael
> 
> 
> On Mon, May 12, 2025 at 5:18 PM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>> Hi Luke, Chia,
>> 
>> Thanks for your information.
>> 
>>>> FYI, there are indeed use cases that need to keep all history data without
>>>> a cleanup policy.
>> 
>> I agree that we should maintain backward compatibility. Therefore, I will update the KIP as follows:
>> - If the user sets an empty cleanup.policy, we will log a warning message and default it to 'none’.
>> - Starting from Kafka 5.0, setting an empty cleanup.policy will no longer be allowed.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月12日 上午11:56 寫道：
>>> 
>>> hi Luke
>>> 
>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0 we throw exception directly to make it fail fast
>>> 
>>> Jiunn:   WDYT?
>>> 
>>> Best,
>>> Chia-Ping
>>> 
>>>> Luke Chen <sh...@gmail.com> 於 2025年5月12日 上午10:52 寫道：
>>>> 
>>>> Hi Jiunn-Yang,
>>>> 
>>>> Thanks for the KIP.
>>>> 
>>>> Comments:
>>>> 1. "it is acceptable because an empty cleanup.policy is considered invalid
>>>> in Kafka. Therefore, this trade-off is justified."
>>>> Could you explain more about this? Why is this trade-off justified?
>>>> If we think the empty cleanup.policy is invalid in kafka, why do we provide
>>>> an additional "none" option to users in this KIP?
>>>> FYI, there are indeed use cases that need to keep all history data without
>>>> a cleanup policy.
>>>> 
>>>> 2. Following (1), I agree we should fail fast for empty
>>>> "group.coordinator.rebalance.protocols" and "process.roles" configs since
>>>> they are invalid.
>>>> But about "cleanup.policy", I don't see a persuasive reason why we should
>>>> break backward compatibility to change it.
>>>> "This provides a clear and direct way to configure Kafka to retain all log
>>>> segments without any automatic cleanup."
>>>> This is the motivation we mentioned in the KIP, but to me, backward
>>>> compatibility is much more important than "a clear and direct way to config
>>>> kafka".
>>>> Do we really have to change the "cleanup.policy" config?
>>>> 
>>>> Thanks.
>>>> Luke
>>>> 
>>>> 
>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid> wrote:
>>>>> 
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> Thanks for the updated KIP. It looks good to me.
>>>>> 
>>>>> Jun
>>>>> 
>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>> Hi Jun, chia
>>>>>> 
>>>>>> Thanks for all the comments. They have all been addressed in the updated
>>>>>> KIP.
>>>>>> 
>>>>>> I've removed all deprecated-related sections. Additionally, an exception
>>>>>> will now be thrown if a
>>>>>> developer passes an empty validStrings list to the inNonEmpty method.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
>>>>>>> 
>>>>>>> hi Jiunn-Yang
>>>>>>> 
>>>>>>> The `inNonEmpty` is used to prevent users pass empty config, so should
>>>>> it
>>>>>>> be non-empty too?
>>>>>>> 
>>>>>>> For example, `inNonEmpty()` should throw exception directly.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the reply. There are still references to the deprecated
>>>>>> method.
>>>>>>>> 
>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jun, chia,
>>>>>>>>> 
>>>>>>>>> Thanks for all the comments. They have all been addressed in the
>>>>>> updated
>>>>>>>>> KIP.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
>>>>>>>>>> 
>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan", could you
>>>>> add
>>>>>>>>>> description to remind readers that "clean.policy=" should be
>>>>> replaced
>>>>>>>> by
>>>>>>>>>> "clean.policy=none" if they do want to disable delete and compact.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Chia-Ping
>>>>>>>>>> 
>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
>>>>>>>>>> 
>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you update
>>>>> the
>>>>>>>>> KIP
>>>>>>>>>>> completely? There are still references to deprecation.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>> 
>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in method, as
>>>>> there
>>>>>>>> may
>>>>>>>>>>> be
>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have both
>>>>> methods:
>>>>>>>> in
>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>>>>> 
>>>>>>>>>>>>> Just a minor comment. It would be useful to document that during
>>>>>>>>>>> upgrade,
>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to start.
>>>>>>>>>>>> 
>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Just a minor comment. It would be useful to document that during
>>>>>>>>>>> upgrade,
>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to start.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain backward
>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove the
>>>>> boolean
>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the method
>>>>>>>> name
>>>>>>>>>>> is
>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> “In this case, the change does not introduce new invalid
>>>>>>>>>>>> configurations
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>> The main behavioral difference is that we now explicitly
>>>>> throw a
>>>>>>>>>>>>>>>> ConfigException  during the storage format validation phase
>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by this change is
>>>>>>>> that
>>>>>>>>> a
>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the storage format
>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> It seems that currently formating the storage already fails
>>>>> if
>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
>>>>>>>>> empty.
>>>>>>>>>>>> It
>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>> moved from the server startup phase to the storage format
>>>>>>>> phase.
>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
>>>>>>>>> considered
>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>> result in a ConfigException during storage format phase.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols
>>>>> and
>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With this KIP, the
>>>>>>>> user
>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate) exception. It
>>>>>>>> will
>>>>>>>>>>> be
>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping.
>>>>> If
>>>>>> a
>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than
>>>>>> an
>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user will never
>>>>>>>>> realize
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user realize
>>>>>> there
>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we don't allow an
>>>>>>>>> empty
>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline the code
>>>>>>>> and
>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact any
>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete segments based
>>>>> on
>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
>>>>>>>> continue
>>>>>>>>>>> to
>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote storage
>>>>> is
>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>>>>> jun@confluent.io>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
>>>>>>>>> during
>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7133700@gmail.com
>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised
>>>>> the
>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
>>>>>>>>> other
>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
>>>>>>>>>>> approach
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
>>>>> 凌晨4:52
>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None
>>>>>> can't
>>>>>>>>> be
>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document
>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
>>>>>>>>>>> config
>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty
>>>>> check
>>>>>>>>>>> does.
>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>>>>> s7133700@gmail.com
>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list
>>>>>> is
>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Mickael Maison <mi...@gmail.com>]

Hi,

Thanks for the KIP.

One weirdness of cleanup.policy is that you can set it to
"delete,delete,delete,compact,compact" for example. Setting delete or
compact multiple times has no functional impact.
I don't think it's something that has to be addressed, but while
you're looking at the configuration's validation I thought I'd bring
it up.

Thanks,
Mickael


On Mon, May 12, 2025 at 5:18 PM 黃竣陽 <s7...@gmail.com> wrote:
>
> Hi Luke, Chia,
>
> Thanks for your information.
>
> >> FYI, there are indeed use cases that need to keep all history data without
> >> a cleanup policy.
>
> I agree that we should maintain backward compatibility. Therefore, I will update the KIP as follows:
> - If the user sets an empty cleanup.policy, we will log a warning message and default it to 'none’.
> - Starting from Kafka 5.0, setting an empty cleanup.policy will no longer be allowed.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月12日 上午11:56 寫道：
> >
> > hi Luke
> >
> > We can print warnings for empty cleanup.policy in 4.x and in 5.0 we throw exception directly to make it fail fast
> >
> > Jiunn:   WDYT?
> >
> > Best,
> > Chia-Ping
> >
> >> Luke Chen <sh...@gmail.com> 於 2025年5月12日 上午10:52 寫道：
> >>
> >> Hi Jiunn-Yang,
> >>
> >> Thanks for the KIP.
> >>
> >> Comments:
> >> 1. "it is acceptable because an empty cleanup.policy is considered invalid
> >> in Kafka. Therefore, this trade-off is justified."
> >> Could you explain more about this? Why is this trade-off justified?
> >> If we think the empty cleanup.policy is invalid in kafka, why do we provide
> >> an additional "none" option to users in this KIP?
> >> FYI, there are indeed use cases that need to keep all history data without
> >> a cleanup policy.
> >>
> >> 2. Following (1), I agree we should fail fast for empty
> >> "group.coordinator.rebalance.protocols" and "process.roles" configs since
> >> they are invalid.
> >> But about "cleanup.policy", I don't see a persuasive reason why we should
> >> break backward compatibility to change it.
> >> "This provides a clear and direct way to configure Kafka to retain all log
> >> segments without any automatic cleanup."
> >> This is the motivation we mentioned in the KIP, but to me, backward
> >> compatibility is much more important than "a clear and direct way to config
> >> kafka".
> >> Do we really have to change the "cleanup.policy" config?
> >>
> >> Thanks.
> >> Luke
> >>
> >>
> >>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid> wrote:
> >>>
> >>> Hi, Jiunn-Yang,
> >>>
> >>> Thanks for the updated KIP. It looks good to me.
> >>>
> >>> Jun
> >>>
> >>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>
> >>>> Hi Jun, chia
> >>>>
> >>>> Thanks for all the comments. They have all been addressed in the updated
> >>>> KIP.
> >>>>
> >>>> I've removed all deprecated-related sections. Additionally, an exception
> >>>> will now be thrown if a
> >>>> developer passes an empty validStrings list to the inNonEmpty method.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> >>>>>
> >>>>> hi Jiunn-Yang
> >>>>>
> >>>>> The `inNonEmpty` is used to prevent users pass empty config, so should
> >>> it
> >>>>> be non-empty too?
> >>>>>
> >>>>> For example, `inNonEmpty()` should throw exception directly.
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> >>>>>
> >>>>>> Hi, Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the reply. There are still references to the deprecated
> >>>> method.
> >>>>>>
> >>>>>> "stop relying on the deprecated methods"
> >>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi Jun, chia,
> >>>>>>>
> >>>>>>> Thanks for all the comments. They have all been addressed in the
> >>>> updated
> >>>>>>> KIP.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> >>>>>>>>
> >>>>>>>> hi Jiunn-Yang
> >>>>>>>>
> >>>>>>>> in the "Compatibility, Deprecation, and Migration Plan", could you
> >>> add
> >>>>>>>> description to remind readers that "clean.policy=" should be
> >>> replaced
> >>>>>> by
> >>>>>>>> "clean.policy=none" if they do want to disable delete and compact.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>>
> >>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> >>>>>>>>
> >>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>
> >>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you update
> >>> the
> >>>>>>> KIP
> >>>>>>>>> completely? There are still references to deprecation.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Jun
> >>>>>>>>>
> >>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Jun,
> >>>>>>>>>>
> >>>>>>>>>> I don’t think we should deprecate the ValidList#in method, as
> >>> there
> >>>>>> may
> >>>>>>>>> be
> >>>>>>>>>> future scenarios where we want
> >>>>>>>>>> to allow list configs to be empty. It’s useful to have both
> >>> methods:
> >>>>>> in
> >>>>>>>>>> (which allows empty lists)
> >>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> >>>>>>>>>>
> >>>>>>>>>>> Just a minor comment. It would be useful to document that during
> >>>>>>>>> upgrade,
> >>>>>>>>>>> if cleanup.policy is empty, the broker will fail to start.
> >>>>>>>>>>
> >>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>
> >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> >>>>>>>>>>>
> >>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>
> >>>>>>>>>>> Just a minor comment. It would be useful to document that during
> >>>>>>>>> upgrade,
> >>>>>>>>>>> if cleanup.policy is empty, the broker will fail to start.
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Since ValidList is a public API, we need to maintain backward
> >>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
> >>>>>>>>>>>> We can update the signature of isNonEmpty() to remove the
> >>> boolean
> >>>>>>>>>>>> parameter.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the method
> >>>>>> name
> >>>>>>>>> is
> >>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
> >>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> “In this case, the change does not introduce new invalid
> >>>>>>>>>> configurations
> >>>>>>>>>>>> or
> >>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>> The main behavioral difference is that we now explicitly
> >>> throw a
> >>>>>>>>>>>>>> ConfigException  during the storage format validation phase
> >>>>>>>>>>>>>> instead of relying on the current behavior.”
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> "The main behavioral difference introduced by this change is
> >>>>>> that
> >>>>>>> a
> >>>>>>>>>>>>>>> ConfigException  will now be thrown during the storage format
> >>>>>>>>>>>> validation
> >>>>>>>>>>>>>>> phase, rather than during server startup."
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It seems that currently formating the storage already fails
> >>> if
> >>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
> >>>>>>> empty.
> >>>>>>>>>> It
> >>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
> >>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>> moved from the server startup phase to the storage format
> >>>>>> phase.
> >>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
> >>>>>>> considered
> >>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>> result in a ConfigException during storage format phase.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols
> >>> and
> >>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>> configurations can't be empty right now. With this KIP, the
> >>>>>> user
> >>>>>>>>>> will
> >>>>>>>>>>>>>>>>> probably get a different (and more accurate) exception. It
> >>>>>> will
> >>>>>>>>> be
> >>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping.
> >>> If
> >>>> a
> >>>>>>>>> user
> >>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than
> >>>> an
> >>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>> If we automatically treat it as None, the user will never
> >>>>>>> realize
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user realize
> >>>> there
> >>>>>>>>> is a
> >>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> >>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
> >>>>>>>>>> compatibility
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> However, according to the documentation, we don't allow an
> >>>>>>> empty
> >>>>>>>>>>>> value
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
> >>>>>>>>>> exception
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline the code
> >>>>>> and
> >>>>>>>>>> avoid
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
> >>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact any
> >>>>>>>>>>>> segments",
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete segments based
> >>> on
> >>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
> >>>>>> continue
> >>>>>>>>> to
> >>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> >>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote storage
> >>> is
> >>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> >>> jun@confluent.io>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
> >>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> >>>>>>> during
> >>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7133700@gmail.com
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised
> >>> the
> >>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
> >>>>>>> other
> >>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
> >>>>>>>>> approach
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
> >>>>>>>>> warning
> >>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
> >>> 凌晨4:52
> >>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> >>>>>>>>> cleanup.policy.
> >>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None
> >>>> can't
> >>>>>>> be
> >>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document
> >>> that.
> >>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
> >>>>>>>>> config
> >>>>>>>>>>>> has
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty
> >>> check
> >>>>>>>>> does.
> >>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> >>> s7133700@gmail.com
> >>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> >>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> >>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> >>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> >>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list
> >>>> is
> >>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi Luke, Chia,

Thanks for your information.

>> FYI, there are indeed use cases that need to keep all history data without
>> a cleanup policy.

I agree that we should maintain backward compatibility. Therefore, I will update the KIP as follows: 
- If the user sets an empty cleanup.policy, we will log a warning message and default it to 'none’. 
- Starting from Kafka 5.0, setting an empty cleanup.policy will no longer be allowed.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月12日 上午11:56 寫道：
> 
> hi Luke
> 
> We can print warnings for empty cleanup.policy in 4.x and in 5.0 we throw exception directly to make it fail fast
> 
> Jiunn:   WDYT?
> 
> Best,
> Chia-Ping
> 
>> Luke Chen <sh...@gmail.com> 於 2025年5月12日 上午10:52 寫道：
>> 
>> Hi Jiunn-Yang,
>> 
>> Thanks for the KIP.
>> 
>> Comments:
>> 1. "it is acceptable because an empty cleanup.policy is considered invalid
>> in Kafka. Therefore, this trade-off is justified."
>> Could you explain more about this? Why is this trade-off justified?
>> If we think the empty cleanup.policy is invalid in kafka, why do we provide
>> an additional "none" option to users in this KIP?
>> FYI, there are indeed use cases that need to keep all history data without
>> a cleanup policy.
>> 
>> 2. Following (1), I agree we should fail fast for empty
>> "group.coordinator.rebalance.protocols" and "process.roles" configs since
>> they are invalid.
>> But about "cleanup.policy", I don't see a persuasive reason why we should
>> break backward compatibility to change it.
>> "This provides a clear and direct way to configure Kafka to retain all log
>> segments without any automatic cleanup."
>> This is the motivation we mentioned in the KIP, but to me, backward
>> compatibility is much more important than "a clear and direct way to config
>> kafka".
>> Do we really have to change the "cleanup.policy" config?
>> 
>> Thanks.
>> Luke
>> 
>> 
>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid> wrote:
>>> 
>>> Hi, Jiunn-Yang,
>>> 
>>> Thanks for the updated KIP. It looks good to me.
>>> 
>>> Jun
>>> 
>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>> Hi Jun, chia
>>>> 
>>>> Thanks for all the comments. They have all been addressed in the updated
>>>> KIP.
>>>> 
>>>> I've removed all deprecated-related sections. Additionally, an exception
>>>> will now be thrown if a
>>>> developer passes an empty validStrings list to the inNonEmpty method.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
>>>>> 
>>>>> hi Jiunn-Yang
>>>>> 
>>>>> The `inNonEmpty` is used to prevent users pass empty config, so should
>>> it
>>>>> be non-empty too?
>>>>> 
>>>>> For example, `inNonEmpty()` should throw exception directly.
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply. There are still references to the deprecated
>>>> method.
>>>>>> 
>>>>>> "stop relying on the deprecated methods"
>>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi Jun, chia,
>>>>>>> 
>>>>>>> Thanks for all the comments. They have all been addressed in the
>>>> updated
>>>>>>> KIP.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
>>>>>>>> 
>>>>>>>> hi Jiunn-Yang
>>>>>>>> 
>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan", could you
>>> add
>>>>>>>> description to remind readers that "clean.policy=" should be
>>> replaced
>>>>>> by
>>>>>>>> "clean.policy=none" if they do want to disable delete and compact.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you update
>>> the
>>>>>>> KIP
>>>>>>>>> completely? There are still references to deprecation.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Jun,
>>>>>>>>>> 
>>>>>>>>>> I don’t think we should deprecate the ValidList#in method, as
>>> there
>>>>>> may
>>>>>>>>> be
>>>>>>>>>> future scenarios where we want
>>>>>>>>>> to allow list configs to be empty. It’s useful to have both
>>> methods:
>>>>>> in
>>>>>>>>>> (which allows empty lists)
>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>>> 
>>>>>>>>>>> Just a minor comment. It would be useful to document that during
>>>>>>>>> upgrade,
>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to start.
>>>>>>>>>> 
>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>> 
>>>>>>>>>>> Just a minor comment. It would be useful to document that during
>>>>>>>>> upgrade,
>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to start.
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>> 
>>>>>>>>>>>> Since ValidList is a public API, we need to maintain backward
>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove the
>>> boolean
>>>>>>>>>>>> parameter.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the method
>>>>>> name
>>>>>>>>> is
>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> “In this case, the change does not introduce new invalid
>>>>>>>>>> configurations
>>>>>>>>>>>> or
>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>> The main behavioral difference is that we now explicitly
>>> throw a
>>>>>>>>>>>>>> ConfigException  during the storage format validation phase
>>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> "The main behavioral difference introduced by this change is
>>>>>> that
>>>>>>> a
>>>>>>>>>>>>>>> ConfigException  will now be thrown during the storage format
>>>>>>>>>>>> validation
>>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> It seems that currently formating the storage already fails
>>> if
>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
>>>>>>> empty.
>>>>>>>>>> It
>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com>
>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>> moved from the server startup phase to the storage format
>>>>>> phase.
>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
>>>>>>> considered
>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>> result in a ConfigException during storage format phase.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols
>>> and
>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>> configurations can't be empty right now. With this KIP, the
>>>>>> user
>>>>>>>>>> will
>>>>>>>>>>>>>>>>> probably get a different (and more accurate) exception. It
>>>>>> will
>>>>>>>>> be
>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping.
>>> If
>>>> a
>>>>>>>>> user
>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than
>>>> an
>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user will never
>>>>>>> realize
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user realize
>>>> there
>>>>>>>>> is a
>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
>>>>>>>>>> compatibility
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> However, according to the documentation, we don't allow an
>>>>>>> empty
>>>>>>>>>>>> value
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
>>>>>>>>>> exception
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline the code
>>>>>> and
>>>>>>>>>> avoid
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact any
>>>>>>>>>>>> segments",
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete segments based
>>> on
>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
>>>>>> continue
>>>>>>>>> to
>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote storage
>>> is
>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>>> jun@confluent.io>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
>>>>>>> during
>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7133700@gmail.com
>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised
>>> the
>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
>>>>>>> other
>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
>>>>>>>>> approach
>>>>>>>>>>>> is
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
>>>>>>>>> warning
>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
>>> 凌晨4:52
>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
>>>>>>>>> cleanup.policy.
>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None
>>>> can't
>>>>>>> be
>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document
>>> that.
>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
>>>>>>>>> config
>>>>>>>>>>>> has
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty
>>> check
>>>>>>>>> does.
>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>>> s7133700@gmail.com
>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list
>>>> is
>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

hi Luke

We can print warnings for empty cleanup.policy in 4.x and in 5.0 we throw exception directly to make it fail fast

Jiunn:   WDYT?

Best,
Chia-Ping

> Luke Chen <sh...@gmail.com> 於 2025年5月12日 上午10:52 寫道：
> 
> Hi Jiunn-Yang,
> 
> Thanks for the KIP.
> 
> Comments:
> 1. "it is acceptable because an empty cleanup.policy is considered invalid
> in Kafka. Therefore, this trade-off is justified."
> Could you explain more about this? Why is this trade-off justified?
> If we think the empty cleanup.policy is invalid in kafka, why do we provide
> an additional "none" option to users in this KIP?
> FYI, there are indeed use cases that need to keep all history data without
> a cleanup policy.
> 
> 2. Following (1), I agree we should fail fast for empty
> "group.coordinator.rebalance.protocols" and "process.roles" configs since
> they are invalid.
> But about "cleanup.policy", I don't see a persuasive reason why we should
> break backward compatibility to change it.
> "This provides a clear and direct way to configure Kafka to retain all log
> segments without any automatic cleanup."
> This is the motivation we mentioned in the KIP, but to me, backward
> compatibility is much more important than "a clear and direct way to config
> kafka".
> Do we really have to change the "cleanup.policy" config?
> 
> Thanks.
> Luke
> 
> 
>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid> wrote:
>> 
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the updated KIP. It looks good to me.
>> 
>> Jun
>> 
>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>> Hi Jun, chia
>>> 
>>> Thanks for all the comments. They have all been addressed in the updated
>>> KIP.
>>> 
>>> I've removed all deprecated-related sections. Additionally, an exception
>>> will now be thrown if a
>>> developer passes an empty validStrings list to the inNonEmpty method.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
>>>> 
>>>> hi Jiunn-Yang
>>>> 
>>>> The `inNonEmpty` is used to prevent users pass empty config, so should
>> it
>>>> be non-empty too?
>>>> 
>>>> For example, `inNonEmpty()` should throw exception directly.
>>>> 
>>>> Best,
>>>> Chia-Ping
>>>> 
>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
>>>> 
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> Thanks for the reply. There are still references to the deprecated
>>> method.
>>>>> 
>>>>> "stop relying on the deprecated methods"
>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hi Jun, chia,
>>>>>> 
>>>>>> Thanks for all the comments. They have all been addressed in the
>>> updated
>>>>>> KIP.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
>>>>>>> 
>>>>>>> hi Jiunn-Yang
>>>>>>> 
>>>>>>> in the "Compatibility, Deprecation, and Migration Plan", could you
>> add
>>>>>>> description to remind readers that "clean.policy=" should be
>> replaced
>>>>> by
>>>>>>> "clean.policy=none" if they do want to disable delete and compact.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you update
>> the
>>>>>> KIP
>>>>>>>> completely? There are still references to deprecation.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jun,
>>>>>>>>> 
>>>>>>>>> I don’t think we should deprecate the ValidList#in method, as
>> there
>>>>> may
>>>>>>>> be
>>>>>>>>> future scenarios where we want
>>>>>>>>> to allow list configs to be empty. It’s useful to have both
>> methods:
>>>>> in
>>>>>>>>> (which allows empty lists)
>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>> 
>>>>>>>>>> Just a minor comment. It would be useful to document that during
>>>>>>>> upgrade,
>>>>>>>>>> if cleanup.policy is empty, the broker will fail to start.
>>>>>>>>> 
>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
>>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>> 
>>>>>>>>>> Just a minor comment. It would be useful to document that during
>>>>>>>> upgrade,
>>>>>>>>>> if cleanup.policy is empty, the broker will fail to start.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello Jun,
>>>>>>>>>>> 
>>>>>>>>>>> Since ValidList is a public API, we need to maintain backward
>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
>>>>>>>>>>> We can update the signature of isNonEmpty() to remove the
>> boolean
>>>>>>>>>>> parameter.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>> 
>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the method
>>>>> name
>>>>>>>> is
>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> “In this case, the change does not introduce new invalid
>>>>>>>>> configurations
>>>>>>>>>>> or
>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>> The main behavioral difference is that we now explicitly
>> throw a
>>>>>>>>>>>>> ConfigException  during the storage format validation phase
>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> "The main behavioral difference introduced by this change is
>>>>> that
>>>>>> a
>>>>>>>>>>>>>> ConfigException  will now be thrown during the storage format
>>>>>>>>>>> validation
>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> It seems that currently formating the storage already fails
>> if
>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
>>>>>> empty.
>>>>>>>>> It
>>>>>>>>>>>>> just
>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>> moved from the server startup phase to the storage format
>>>>> phase.
>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
>>>>>> considered
>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>> result in a ConfigException during storage format phase.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols
>> and
>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>> configurations can't be empty right now. With this KIP, the
>>>>> user
>>>>>>>>> will
>>>>>>>>>>>>>>>> probably get a different (and more accurate) exception. It
>>>>> will
>>>>>>>> be
>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping.
>> If
>>> a
>>>>>>>> user
>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than
>>> an
>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>> If we automatically treat it as None, the user will never
>>>>>> realize
>>>>>>>>> the
>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user realize
>>> there
>>>>>>>> is a
>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>>>>>>>> chia7712@gmail.com
>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
>>>>>>>>> compatibility
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> However, according to the documentation, we don't allow an
>>>>>> empty
>>>>>>>>>>> value
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
>>>>>>>>> exception
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline the code
>>>>> and
>>>>>>>>> avoid
>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact any
>>>>>>>>>>> segments",
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete segments based
>> on
>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
>>>>> continue
>>>>>>>> to
>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>>>>> log.retention.ms.
>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote storage
>> is
>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>> jun@confluent.io>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
>>>>>> during
>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7133700@gmail.com
>>> 
>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised
>> the
>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
>>>>>> other
>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
>>>>>>>> approach
>>>>>>>>>>> is
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
>>>>>>>> warning
>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
>> 凌晨4:52
>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
>>>>>>>> cleanup.policy.
>>>>>>>>>>> But
>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None
>>> can't
>>>>>> be
>>>>>>>>>>> used
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document
>> that.
>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
>>>>>>>> config
>>>>>>>>>>> has
>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty
>> check
>>>>>>>> does.
>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>> s7133700@gmail.com
>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list
>>> is
>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Edoardo Comar <ed...@gmail.com>]

FYI while trying with an AlterConfigPolicy to stop the cleanup.policy
being emptied through a AlterConfigOp.OpType.SUBTRACT,
I stumbled in this behavior
that doesn't allow to block an empty policy in Zookeeper versions of Kafka
https://issues.apache.org/jira/browse/KAFKA-19026

On Tue, 13 May 2025 at 07:36, Luke Chen <sh...@gmail.com> wrote:
>
> Hi Jun,
>
> > Regarding the motivation, currently, we never documented that an empty
> cleanup.policy implies infinite retention. In fact, if one leaves
> cleanup.policy empty, it actually breaks remote storage since it will stop
> the cleaning based on local retention size and time too. So, leaving
> cleanup.policy empty is probably more like a user mistake than an
> intentional choice. Based on this assumption, the idea behind the KIP is to
> fail an empty cleanup.policy so that the user is aware of this likely
> mistake and provide a new option None for users wanting infinite retention
> to opt in explicitly.
>
> While we never documented the empty cleanup.policy implies infinite
> retention, we also never documented (or failed) the empty cleanup.policy is
> an invalid configuration. My thought is that this behavior has been existed
> for years (maybe after "cleanup.policy" was introduced?), there could be
> users relying on the empty cleanup.policy for a long time. It is not good
> to break the backward compatibility in a minor release version (ex:
> v4.1.0). Even though we think we are fixing a long existing bug, it could
> be a surprise to users.
>
>
> > We could also consider supporting an empty cleanup.policy by fixing the
> issue in remote storage. But then the user may never realize this if it's a
> mistake.
>
> Good point! I agree we need to fix it!
>
>
> Hi Chia-Ping,
>
> > We can print warnings for empty cleanup.policy in 4.x and in 5.0 we throw
> exception directly to make it fail fast
>
> This suggestion looks good to me!
>
> Thank you.
> Luke
>
> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <ch...@apache.org> wrote:
>
> > hi all,
> >
> > Given Luke mentioned the existence of use cases, it is worth considering
> > the compatibility issue. In fact, I had previously encountered this use
> > case but advised customers to utilize retention configurations instead.
> >
> > > We could also consider supporting an empty cleanup.policy by fixing the
> > > issue in remote storage. But then the user may never realize this if
> > it's a
> > > mistake.
> >
> > Good catch! The "empty" or "none" wouldn't make sense for remote storage.
> > I've opened KAFKA-19273 to ensure topics with tiered storage use a valid
> > delete policy.
> >
> > Best,
> > Chia-Ping
> >
> >
> > On 2025/05/12 19:06:10 Jun Rao wrote:
> > > Hi, Luke,
> > >
> > > Regarding the motivation, currently, we never documented that an empty
> > > cleanup.policy implies infinite retention. In fact, if one leaves
> > > cleanup.policy empty, it actually breaks remote storage since it will
> > stop
> > > the cleaning based on local retention size and time too. So, leaving
> > > cleanup.policy empty is probably more like a user mistake than an
> > > intentional choice. Based on this assumption, the idea behind the KIP is
> > to
> > > fail an empty cleanup.policy so that the user is aware of this likely
> > > mistake and provide a new option None for users wanting infinite
> > retention
> > > to opt in explicitly.
> > >
> > > We could also consider supporting an empty cleanup.policy by fixing the
> > > issue in remote storage. But then the user may never realize this if
> > it's a
> > > mistake.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi Jiunn-Yang,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > Comments:
> > > > 1. "it is acceptable because an empty cleanup.policy is considered
> > invalid
> > > > in Kafka. Therefore, this trade-off is justified."
> > > > Could you explain more about this? Why is this trade-off justified?
> > > > If we think the empty cleanup.policy is invalid in kafka, why do we
> > provide
> > > > an additional "none" option to users in this KIP?
> > > > FYI, there are indeed use cases that need to keep all history data
> > without
> > > > a cleanup policy.
> > > >
> > > > 2. Following (1), I agree we should fail fast for empty
> > > > "group.coordinator.rebalance.protocols" and "process.roles" configs
> > since
> > > > they are invalid.
> > > > But about "cleanup.policy", I don't see a persuasive reason why we
> > should
> > > > break backward compatibility to change it.
> > > > "This provides a clear and direct way to configure Kafka to retain all
> > log
> > > > segments without any automatic cleanup."
> > > > This is the motivation we mentioned in the KIP, but to me, backward
> > > > compatibility is much more important than "a clear and direct way to
> > config
> > > > kafka".
> > > > Do we really have to change the "cleanup.policy" config?
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > > On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > > >
> > > > > Hi, Jiunn-Yang,
> > > > >
> > > > > Thanks for the updated KIP. It looks good to me.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >
> > > > > > Hi Jun, chia
> > > > > >
> > > > > > Thanks for all the comments. They have all been addressed in the
> > > > updated
> > > > > > KIP.
> > > > > >
> > > > > > I've removed all deprecated-related sections. Additionally, an
> > > > exception
> > > > > > will now be thrown if a
> > > > > > developer passes an empty validStrings list to the inNonEmpty
> > method.
> > > > > >
> > > > > > Best Regards,
> > > > > > Jiunn-Yang
> > > > > >
> > > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> > > > > > >
> > > > > > > hi Jiunn-Yang
> > > > > > >
> > > > > > > The `inNonEmpty` is used to prevent users pass empty config, so
> > > > should
> > > > > it
> > > > > > > be non-empty too?
> > > > > > >
> > > > > > > For example, `inNonEmpty()` should throw exception directly.
> > > > > > >
> > > > > > > Best,
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> > > > > > >
> > > > > > >> Hi, Jiunn-Yang,
> > > > > > >>
> > > > > > >> Thanks for the reply. There are still references to the
> > deprecated
> > > > > > method.
> > > > > > >>
> > > > > > >> "stop relying on the deprecated methods"
> > > > > > >> "Finally, the deprecated method will be removed in Kafka 5.0"
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >> Jun
> > > > > > >>
> > > > > > >> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > > > >>
> > > > > > >>> Hi Jun, chia,
> > > > > > >>>
> > > > > > >>> Thanks for all the comments. They have all been addressed in
> > the
> > > > > > updated
> > > > > > >>> KIP.
> > > > > > >>>
> > > > > > >>> Best Regards,
> > > > > > >>> Jiunn-Yang
> > > > > > >>>
> > > > > > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> > > > > > >>>>
> > > > > > >>>> hi Jiunn-Yang
> > > > > > >>>>
> > > > > > >>>> in the "Compatibility, Deprecation, and Migration Plan",
> > could you
> > > > > add
> > > > > > >>>> description to remind readers that "clean.policy=" should be
> > > > > replaced
> > > > > > >> by
> > > > > > >>>> "clean.policy=none" if they do want to disable delete and
> > compact.
> > > > > > >>>>
> > > > > > >>>> Best,
> > > > > > >>>> Chia-Ping
> > > > > > >>>>
> > > > > > >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> > > > > > >>>>
> > > > > > >>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>
> > > > > > >>>>> It's fine not to deprecate ValidList#in for now. Could you
> > update
> > > > > the
> > > > > > >>> KIP
> > > > > > >>>>> completely? There are still references to deprecation.
> > > > > > >>>>>
> > > > > > >>>>> Thanks,
> > > > > > >>>>>
> > > > > > >>>>> Jun
> > > > > > >>>>>
> > > > > > >>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Hi Jun,
> > > > > > >>>>>>
> > > > > > >>>>>> I don’t think we should deprecate the ValidList#in method,
> > as
> > > > > there
> > > > > > >> may
> > > > > > >>>>> be
> > > > > > >>>>>> future scenarios where we want
> > > > > > >>>>>> to allow list configs to be empty. It’s useful to have both
> > > > > methods:
> > > > > > >> in
> > > > > > >>>>>> (which allows empty lists)
> > > > > > >>>>>> and inNonEmpty (which enforces non-empty lists).
> > > > > > >>>>>>
> > > > > > >>>>>>> Just a minor comment. It would be useful to document that
> > > > during
> > > > > > >>>>> upgrade,
> > > > > > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > > > > > >>>>>>
> > > > > > >>>>>> I’ve updated the KIP accordingly.
> > > > > > >>>>>>
> > > > > > >>>>>> Best Regards,
> > > > > > >>>>>> Jiunn-Yang
> > > > > > >>>>>>
> > > > > > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> > > > > > >>>>>>>
> > > > > > >>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks for the reply. Sounds good.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Just a minor comment. It would be useful to document that
> > > > during
> > > > > > >>>>> upgrade,
> > > > > > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Jun
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Hello Jun,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Since ValidList is a public API, we need to maintain
> > backward
> > > > > > >>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> > > > necessary.
> > > > > > >>>>>>>> We can update the signature of isNonEmpty() to remove the
> > > > > boolean
> > > > > > >>>>>>>> parameter.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Best Regards,
> > > > > > >>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
> > 寫道：
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the
> > > > method
> > > > > > >> name
> > > > > > >>>>> is
> > > > > > >>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Jun
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
> > > > > wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Hi Jun,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Thank you for pointing that out.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> I have revised the KIP as follows:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> “In this case, the change does not introduce new invalid
> > > > > > >>>>>> configurations
> > > > > > >>>>>>>> or
> > > > > > >>>>>>>>>> prevent any currently valid setup.
> > > > > > >>>>>>>>>> The main behavioral difference is that we now explicitly
> > > > > throw a
> > > > > > >>>>>>>>>> ConfigException  during the storage format validation
> > phase
> > > > > > >>>>>>>>>> instead of relying on the current behavior.”
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> This reflects the correct behavior.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
> > 凌晨1:17 寫道：
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> "The main behavioral difference introduced by this
> > change
> > > > is
> > > > > > >> that
> > > > > > >>> a
> > > > > > >>>>>>>>>>> ConfigException  will now be thrown during the storage
> > > > format
> > > > > > >>>>>>>> validation
> > > > > > >>>>>>>>>>> phase, rather than during server startup."
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> It seems that currently formating the storage already
> > fails
> > > > > if
> > > > > > >>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> > > > are
> > > > > > >>> empty.
> > > > > > >>>>>> It
> > > > > > >>>>>>>>>> just
> > > > > > >>>>>>>>>>> gets a different exception.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> > s7133700@gmail.com>
> > > > > > wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Hi Jun, chia,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thanks for your feedback.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I add a new section for this change:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> - Validation for
> > group.coordinator.rebalance.protocols and
> > > > > > >>>>>>>> process.roles
> > > > > > >>>>>>>>>>>> will be
> > > > > > >>>>>>>>>>>> moved from the server startup phase to the storage
> > format
> > > > > > >> phase.
> > > > > > >>>>>>>>>>>> - For cleanup.policy, setting an empty list will now
> > be
> > > > > > >>> considered
> > > > > > >>>>>>>>>> invalid
> > > > > > >>>>>>>>>>>> and will
> > > > > > >>>>>>>>>>>> result in a ConfigException during storage format
> > phase.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
> > 凌晨4:44
> > > > 寫道：
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Q2. It's true that
> > group.coordinator.rebalance.protocols
> > > > > and
> > > > > > >>>>>>>>>>>> process.roles
> > > > > > >>>>>>>>>>>>> configurations can't be empty right now. With this
> > KIP,
> > > > the
> > > > > > >> user
> > > > > > >>>>>> will
> > > > > > >>>>>>>>>>>>> probably get a different (and more accurate)
> > exception.
> > > > It
> > > > > > >> will
> > > > > > >>>>> be
> > > > > > >>>>>>>>>> useful
> > > > > > >>>>>>>>>>>>> to document that.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
> > Chia-Ping.
> > > > > If
> > > > > > a
> > > > > > >>>>> user
> > > > > > >>>>>>>>>>>> leaves
> > > > > > >>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
> > mistake
> > > > than
> > > > > > an
> > > > > > >>>>>>>>>> intention.
> > > > > > >>>>>>>>>>>>> If we automatically treat it as None, the user will
> > never
> > > > > > >>> realize
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>>>> mistake. Throwing an exception will let the user
> > realize
> > > > > > there
> > > > > > >>>>> is a
> > > > > > >>>>>>>>>>>> mistake.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> > > > > > >>>>> chia7712@gmail.com
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> hi Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> > maintaining
> > > > > > >>>>>> compatibility
> > > > > > >>>>>>>> is
> > > > > > >>>>>>>>>>>>>> important.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> However, according to the documentation, we don't
> > allow
> > > > an
> > > > > > >>> empty
> > > > > > >>>>>>>> value
> > > > > > >>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
> > throwing
> > > > an
> > > > > > >>>>>> exception
> > > > > > >>>>>>>>>> for
> > > > > > >>>>>>>>>>>> an
> > > > > > >>>>>>>>>>>>>> empty value in version 4.x? This could streamline
> > the
> > > > code
> > > > > > >> and
> > > > > > >>>>>> avoid
> > > > > > >>>>>>>>>> an
> > > > > > >>>>>>>>>>>>>> extra deprecation cycle.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
> > > > > > 上午6:33寫道：
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Regarding "The none policy will not delete or
> > compact
> > > > any
> > > > > > >>>>>>>> segments",
> > > > > > >>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>> should be more accurate. We won't delete segments
> > based
> > > > > on
> > > > > > >>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
> > should
> > > > > > >> continue
> > > > > > >>>>> to
> > > > > > >>>>>>>>>> delete
> > > > > > >>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> > > > > > >> log.retention.ms.
> > > > > > >>>>>>>>>>>> Otherwise,
> > > > > > >>>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>> risk running out of local disk space when remote
> > > > storage
> > > > > is
> > > > > > >>>>>>>> enabled.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> > > > > jun@confluent.io>
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > > > > > >>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > > > process.roles
> > > > > > >>> during
> > > > > > >>>>>>>>>>>> upgrade?
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> > > > s7133700@gmail.com
> > > > > >
> > > > > > >>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Hello Jun,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Q1 & Q3:
> > > > > > >>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
> > revised
> > > > > the
> > > > > > >>>>>>>>>>>>>> cleanup.policy
> > > > > > >>>>>>>>>>>>>>>>> documentation
> > > > > > >>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
> > with
> > > > any
> > > > > > >>> other
> > > > > > >>>>>>>>>> policy.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Q2:
> > > > > > >>>>>>>>>>>>>>>>> For users currently using an empty
> > cleanup.policy,
> > > > the
> > > > > > >>>>> approach
> > > > > > >>>>>>>> is
> > > > > > >>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> apply the none policy
> > > > > > >>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> > > > Additionally, a
> > > > > > >>>>> warning
> > > > > > >>>>>>>>>>>>>> message
> > > > > > >>>>>>>>>>>>>>>>> will be emitted to inform users
> > > > > > >>>>>>>>>>>>>>>>> of the upcoming change.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
> > > > > 凌晨4:52
> > > > > > >> 寫道：
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> > > > > > >>>>> cleanup.policy.
> > > > > > >>>>>>>> But
> > > > > > >>>>>>>>>>>>>> now
> > > > > > >>>>>>>>>>>>>>>>> not
> > > > > > >>>>>>>>>>>>>>>>>> all value combinations are valid. For example,
> > None
> > > > > > can't
> > > > > > >>> be
> > > > > > >>>>>>>> used
> > > > > > >>>>>>>>>>>>>> with
> > > > > > >>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> > document
> > > > > that.
> > > > > > >>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
> > > > existing
> > > > > > >>>>> config
> > > > > > >>>>>>>> has
> > > > > > >>>>>>>>>> an
> > > > > > >>>>>>>>>>>>>>>>> empty
> > > > > > >>>>>>>>>>>>>>>>>> list.
> > > > > > >>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
> > empty
> > > > > check
> > > > > > >>>>> does.
> > > > > > >>>>>>>> How
> > > > > > >>>>>>>>>>>>>>> about
> > > > > > >>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> > > > > s7133700@gmail.com
> > > > > > >
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> > > > > > >>>>>> cleanup.policy
> > > > > > >>>>>>>>>>>>>>>>> shouldn't
> > > > > > >>>>>>>>>>>>>>>>>>> be empty
> > > > > > >>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> > cleanup.policy
> > > > > > >>>>>> configuration.
> > > > > > >>>>>>>>>>>>>>>>> Currently,
> > > > > > >>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > > > > > >>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> > > > > > >>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
> > empty
> > > > list
> > > > > > is
> > > > > > >>>>>>>> allowed.
> > > > > > >>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> > cleanup.policy.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello all, 

I have updated the KIP with following changes

1. Before, This KIP would throw an exception when users specified duplicate values 
in LIST-type configurations. However, to maintain backward compatibility, if users provide 
duplicate entries for configurations that do not support duplicates, those duplicates will now be
ignored and a warning will be logged instead.

2. The following LIST-type configurations were updated accordingly:
- BrokerSecurityConfigs.SSL_CIPHER_SUITES_CONFIG
- BrokerSecurityConfigs.SASL_ENABLED_MECHANISMS_CONFIG
- BrokerSecurityConfigs.SASL_KERBEROS_PRINCIPAL_TO_LOCAL_RULES_CONFIG
- BrokerSecurityConfigs.SASL_OAUTHBEARER_EXPECTED_AUDIENCE_CONFIG
- DefaultConfigPropertyFilter.CONFIG_PROPERTIES_EXCLUDE_CONFIG
- DefaultTopicFilter.TOPICS_CONFIG
- DefaultTopicFilter.TOPICS_EXCLUDE_CONFIG

Best Regards,
Jiunn-Yang

> 黃竣陽 <s7...@gmail.com> 於 2025年8月23日 下午1:32 寫道：
> 
> Hello all, 
> 
> I have updated the KIP with following changes
> 
> 1. HeaderFrom.headers should allow duplicate values, so I have removed it from the KIP. 
> FYI, the reference test case[1].
> 
> 2. The log.dir configuration already allows multiple values under the existing logic [2]. 
> Therefore, we propose updating its type from STRING to LIST, changing the default 
> value from "/tmp/kafka-logs" to List.of("/tmp/kafka-logs"). In addition, we update its 
> validator from null to anyNonDuplicateValues(isNullAllowed = true, isEmptyAllowed = false).
> 
> [1]:  https://github.com/apache/kafka/blob/eba9839776c07e860c9aa7fe2d028e4f65031d5b/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/HeaderFromTest.java#L251
> 
> [2]: https://github.com/apache/kafka/blob/eba9839776c07e860c9aa7fe2d028e4f65031d5b/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L82
> 
> Best Regards,
> Jiunn-Yang
> 
>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月22日 凌晨3:06 寫道：
>> 
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the reply. The changes look good to me and we can follow up on
>> the 0.0.0.0 issue separately in KIP-1202.
>> 
>> Jun
>> 
>> On Thu, Aug 21, 2025 at 6:55 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hello Jun, chia,
>>> 
>>>> (By the way, the table in KIP-1202 has an incorrect value — null is
>>> acceptable for both cases.)
>>> 
>>> KIP-1202 covers the scenario where the listeners configuration is set to
>>> the special
>>> value PLAINTEXT://0.0.0.0:9092. In this case, if advertised.listeners is
>>> configured as null,
>>> an IllegalArgumentException will be thrown: requirement failed:
>>> advertised.listeners cannot
>>> use the non-routable meta-address 0.0.0.0. Use a routable IP address.
>>> 
>>>> If we want to address `advertised.listeners` in this KIP, it would be
>>> better to also have the controller reject an `empty` value
>>> 
>>> Yes, I fully agree that we should disallow an empty list for this
>>> configuration, since specifying an
>>> empty list is odd, it would mean that the node does not advertise any
>>> listener addresses,
>>> making it unreachable by clients. I will updated KIP according by this.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年8月21日 晚上8:48 寫道：
>>>> 
>>>> 2. It seems that advertised.listeners should default to null and can't be
>>>> 
>>>> That is an inconsistency between the broker and controller, and it will
>>> be addressed by KIP-1202. `null` is valid for both, while `empty` is valid
>>> only for the controller. As Jun mentioned, the broker encounters an error
>>> in this case.
>>>> (By the way, the table in KIP-1202 has an incorrect value — null is
>>> acceptable for both cases.)
>>>> 
>>>> If we want to address `advertised.listeners` in this KIP, it would be
>>> better to also have the controller reject an `empty` value
>>>> 
>>>> Best,
>>>> Chia-Ping
>>>> 
>>>> On 2025/08/20 18:01:35 Jun Rao wrote:
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> Thanks for the update.
>>>>> 
>>>>> 1. Sounds good.
>>>>> 
>>>>> 2. It seems that advertised.listeners should default to null and can't
>>> be
>>>>> empty. Currently, if it's set to empty, it fails with the following.
>>>>> 
>>>>> java.lang.IllegalArgumentException: requirement failed: There must be at
>>>>> least one broker advertised listener. Perhaps all listeners appear in
>>>>> controller.listener.names?
>>>>> at scala.Predef$.require(Predef.scala:337)
>>> ~[scala-library-2.13.16.jar:?]
>>>>> at
>>>>> 
>>> kafka.server.KafkaConfig.validateAdvertisedBrokerListenersNonEmptyForBroker$1(KafkaConfig.scala:545)
>>>>> ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Wed, Aug 20, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hello Jun,
>>>>>> 
>>>>>> I have updated the KIP with the following changes.
>>>>>> 
>>>>>> 1. The type of controller.listener.names should be changed from string
>>> to
>>>>>> list.
>>>>>> Its default value should be updated from null to NO_DEFAULT_VALUE, and
>>> its
>>>>>> validator should be updated to anyNonDuplicateValues(isNullAllowed =
>>>>>> false, isEmptyAllowed = false).
>>>>>> 
>>>>>> 2. The type of advertised.listeners should be changed from string to
>>> list.
>>>>>> As for its validator,
>>>>>> I think we can continue the discussion in KIP-1202.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月14日 凌晨12:37 寫道：
>>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the updated KIP. Looks good to me.
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> On Wed, Aug 13, 2025 at 3:08 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hello Jun,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> I have updated the KIP according there comments.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
>>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply. A few more comments.
>>>>>>>>> 
>>>>>>>>> JR50. It seems that you changed the default value for
>>> config.providers
>>>>>>>>> incorrectly. The change is meant for bootstrap.servers.
>>>>>>>>> 
>>>>>>>>> JR51. Could you document the current behavior if bootstrap.servers
>>> is
>>>>>>>> empty
>>>>>>>>> in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
>>>>>>>>> 
>>>>>>>>> JR52. Could you document the justification for changing the default
>>>>>> value
>>>>>>>>> for bootstrap.servers in WorkerConfig?
>>>>>>>>> 
>>>>>>>>> JR53. It seems that WorkerConfig is not public facing. Only classes
>>> in
>>>>>>>>> connect api are public. So, there is no need to document the
>>>>>> deprecation
>>>>>>>>> of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi chia,
>>>>>>>>>> 
>>>>>>>>>> I have updated the KIP with these changes.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> hi Jiunn
>>>>>>>>>>> 
>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>> 3. WorkerConfig – Change the default value of bootstrap.servers
>>> from
>>>>>>>>>> "localhost:9092" to NO_DEFAULT_VALUE
>>>>>>>>>>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> I agree that the default value of “localhost:9092” is strange.
>>>>>> However,
>>>>>>>>>> it is still a breaking change, so please highlight this change in
>>> the
>>>>>>>> KIP.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello all, 

I have updated the KIP with following changes

1. HeaderFrom.headers should allow duplicate values, so I have removed it from the KIP. 
FYI, the reference test case[1].

2. The log.dir configuration already allows multiple values under the existing logic [2]. 
Therefore, we propose updating its type from STRING to LIST, changing the default 
value from "/tmp/kafka-logs" to List.of("/tmp/kafka-logs"). In addition, we update its 
validator from null to anyNonDuplicateValues(isNullAllowed = true, isEmptyAllowed = false).

[1]:  https://github.com/apache/kafka/blob/eba9839776c07e860c9aa7fe2d028e4f65031d5b/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/HeaderFromTest.java#L251

[2]: https://github.com/apache/kafka/blob/eba9839776c07e860c9aa7fe2d028e4f65031d5b/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L82

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月22日 凌晨3:06 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply. The changes look good to me and we can follow up on
> the 0.0.0.0 issue separately in KIP-1202.
> 
> Jun
> 
> On Thu, Aug 21, 2025 at 6:55 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun, chia,
>> 
>>> (By the way, the table in KIP-1202 has an incorrect value — null is
>> acceptable for both cases.)
>> 
>> KIP-1202 covers the scenario where the listeners configuration is set to
>> the special
>> value PLAINTEXT://0.0.0.0:9092. In this case, if advertised.listeners is
>> configured as null,
>> an IllegalArgumentException will be thrown: requirement failed:
>> advertised.listeners cannot
>> use the non-routable meta-address 0.0.0.0. Use a routable IP address.
>> 
>>> If we want to address `advertised.listeners` in this KIP, it would be
>> better to also have the controller reject an `empty` value
>> 
>> Yes, I fully agree that we should disallow an empty list for this
>> configuration, since specifying an
>> empty list is odd, it would mean that the node does not advertise any
>> listener addresses,
>> making it unreachable by clients. I will updated KIP according by this.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年8月21日 晚上8:48 寫道：
>>> 
>>> 2. It seems that advertised.listeners should default to null and can't be
>>> 
>>> That is an inconsistency between the broker and controller, and it will
>> be addressed by KIP-1202. `null` is valid for both, while `empty` is valid
>> only for the controller. As Jun mentioned, the broker encounters an error
>> in this case.
>>> (By the way, the table in KIP-1202 has an incorrect value — null is
>> acceptable for both cases.)
>>> 
>>> If we want to address `advertised.listeners` in this KIP, it would be
>> better to also have the controller reject an `empty` value
>>> 
>>> Best,
>>> Chia-Ping
>>> 
>>> On 2025/08/20 18:01:35 Jun Rao wrote:
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Thanks for the update.
>>>> 
>>>> 1. Sounds good.
>>>> 
>>>> 2. It seems that advertised.listeners should default to null and can't
>> be
>>>> empty. Currently, if it's set to empty, it fails with the following.
>>>> 
>>>> java.lang.IllegalArgumentException: requirement failed: There must be at
>>>> least one broker advertised listener. Perhaps all listeners appear in
>>>> controller.listener.names?
>>>> at scala.Predef$.require(Predef.scala:337)
>> ~[scala-library-2.13.16.jar:?]
>>>> at
>>>> 
>> kafka.server.KafkaConfig.validateAdvertisedBrokerListenersNonEmptyForBroker$1(KafkaConfig.scala:545)
>>>> ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
>>>> 
>>>> Jun
>>>> 
>>>> On Wed, Aug 20, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hello Jun,
>>>>> 
>>>>> I have updated the KIP with the following changes.
>>>>> 
>>>>> 1. The type of controller.listener.names should be changed from string
>> to
>>>>> list.
>>>>> Its default value should be updated from null to NO_DEFAULT_VALUE, and
>> its
>>>>> validator should be updated to anyNonDuplicateValues(isNullAllowed =
>>>>> false, isEmptyAllowed = false).
>>>>> 
>>>>> 2. The type of advertised.listeners should be changed from string to
>> list.
>>>>> As for its validator,
>>>>> I think we can continue the discussion in KIP-1202.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月14日 凌晨12:37 寫道：
>>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the updated KIP. Looks good to me.
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Wed, Aug 13, 2025 at 3:08 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hello Jun,
>>>>>>> 
>>>>>>> Thanks for the reply.
>>>>>>> 
>>>>>>> I have updated the KIP according there comments.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
>>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the reply. A few more comments.
>>>>>>>> 
>>>>>>>> JR50. It seems that you changed the default value for
>> config.providers
>>>>>>>> incorrectly. The change is meant for bootstrap.servers.
>>>>>>>> 
>>>>>>>> JR51. Could you document the current behavior if bootstrap.servers
>> is
>>>>>>> empty
>>>>>>>> in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
>>>>>>>> 
>>>>>>>> JR52. Could you document the justification for changing the default
>>>>> value
>>>>>>>> for bootstrap.servers in WorkerConfig?
>>>>>>>> 
>>>>>>>> JR53. It seems that WorkerConfig is not public facing. Only classes
>> in
>>>>>>>> connect api are public. So, there is no need to document the
>>>>> deprecation
>>>>>>>> of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hi chia,
>>>>>>>>> 
>>>>>>>>> I have updated the KIP with these changes.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
>>>>>>>>>> 
>>>>>>>>>> hi Jiunn
>>>>>>>>>> 
>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> 3. WorkerConfig – Change the default value of bootstrap.servers
>> from
>>>>>>>>> "localhost:9092" to NO_DEFAULT_VALUE
>>>>>>>>>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I agree that the default value of “localhost:9092” is strange.
>>>>> However,
>>>>>>>>> it is still a breaking change, so please highlight this change in
>> the
>>>>>>> KIP.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply. The changes look good to me and we can follow up on
the 0.0.0.0 issue separately in KIP-1202.

Jun

On Thu, Aug 21, 2025 at 6:55 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun, chia,
>
> > (By the way, the table in KIP-1202 has an incorrect value — null is
> acceptable for both cases.)
>
> KIP-1202 covers the scenario where the listeners configuration is set to
> the special
> value PLAINTEXT://0.0.0.0:9092. In this case, if advertised.listeners is
> configured as null,
> an IllegalArgumentException will be thrown: requirement failed:
> advertised.listeners cannot
> use the non-routable meta-address 0.0.0.0. Use a routable IP address.
>
> > If we want to address `advertised.listeners` in this KIP, it would be
> better to also have the controller reject an `empty` value
>
> Yes, I fully agree that we should disallow an empty list for this
> configuration, since specifying an
> empty list is odd, it would mean that the node does not advertise any
> listener addresses,
> making it unreachable by clients. I will updated KIP according by this.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@apache.org> 於 2025年8月21日 晚上8:48 寫道：
> >
> > 2. It seems that advertised.listeners should default to null and can't be
> >
> > That is an inconsistency between the broker and controller, and it will
> be addressed by KIP-1202. `null` is valid for both, while `empty` is valid
> only for the controller. As Jun mentioned, the broker encounters an error
> in this case.
> > (By the way, the table in KIP-1202 has an incorrect value — null is
> acceptable for both cases.)
> >
> > If we want to address `advertised.listeners` in this KIP, it would be
> better to also have the controller reject an `empty` value
> >
> > Best,
> > Chia-Ping
> >
> > On 2025/08/20 18:01:35 Jun Rao wrote:
> >> Hi, Jiunn-Yang,
> >>
> >> Thanks for the update.
> >>
> >> 1. Sounds good.
> >>
> >> 2. It seems that advertised.listeners should default to null and can't
> be
> >> empty. Currently, if it's set to empty, it fails with the following.
> >>
> >> java.lang.IllegalArgumentException: requirement failed: There must be at
> >> least one broker advertised listener. Perhaps all listeners appear in
> >> controller.listener.names?
> >> at scala.Predef$.require(Predef.scala:337)
> ~[scala-library-2.13.16.jar:?]
> >> at
> >>
> kafka.server.KafkaConfig.validateAdvertisedBrokerListenersNonEmptyForBroker$1(KafkaConfig.scala:545)
> >> ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
> >>
> >> Jun
> >>
> >> On Wed, Aug 20, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hello Jun,
> >>>
> >>> I have updated the KIP with the following changes.
> >>>
> >>> 1. The type of controller.listener.names should be changed from string
> to
> >>> list.
> >>> Its default value should be updated from null to NO_DEFAULT_VALUE, and
> its
> >>> validator should be updated to anyNonDuplicateValues(isNullAllowed =
> >>> false, isEmptyAllowed = false).
> >>>
> >>> 2. The type of advertised.listeners should be changed from string to
> list.
> >>> As for its validator,
> >>> I think we can continue the discussion in KIP-1202.
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月14日 凌晨12:37 寫道：
> >>>>
> >>>> Hi, Jiunn-Yang,
> >>>>
> >>>> Thanks for the updated KIP. Looks good to me.
> >>>>
> >>>> Jun
> >>>>
> >>>> On Wed, Aug 13, 2025 at 3:08 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>
> >>>>> Hello Jun,
> >>>>>
> >>>>> Thanks for the reply.
> >>>>>
> >>>>> I have updated the KIP according there comments.
> >>>>>
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
> >>>>>>
> >>>>>> Hi, Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the reply. A few more comments.
> >>>>>>
> >>>>>> JR50. It seems that you changed the default value for
> config.providers
> >>>>>> incorrectly. The change is meant for bootstrap.servers.
> >>>>>>
> >>>>>> JR51. Could you document the current behavior if bootstrap.servers
> is
> >>>>> empty
> >>>>>> in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
> >>>>>>
> >>>>>> JR52. Could you document the justification for changing the default
> >>> value
> >>>>>> for bootstrap.servers in WorkerConfig?
> >>>>>>
> >>>>>> JR53. It seems that WorkerConfig is not public facing. Only classes
> in
> >>>>>> connect api are public. So, there is no need to document the
> >>> deprecation
> >>>>>> of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi chia,
> >>>>>>>
> >>>>>>> I have updated the KIP with these changes.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
> >>>>>>>>
> >>>>>>>> hi Jiunn
> >>>>>>>>
> >>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
> >>>>>>>>>
> >>>>>>>>> 3. WorkerConfig – Change the default value of bootstrap.servers
> from
> >>>>>>> "localhost:9092" to NO_DEFAULT_VALUE
> >>>>>>>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I agree that the default value of “localhost:9092” is strange.
> >>> However,
> >>>>>>> it is still a breaking change, so please highlight this change in
> the
> >>>>> KIP.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun, chia,

> (By the way, the table in KIP-1202 has an incorrect value — null is acceptable for both cases.)

KIP-1202 covers the scenario where the listeners configuration is set to the special 
value PLAINTEXT://0.0.0.0:9092. In this case, if advertised.listeners is configured as null, 
an IllegalArgumentException will be thrown: requirement failed: advertised.listeners cannot 
use the non-routable meta-address 0.0.0.0. Use a routable IP address.

> If we want to address `advertised.listeners` in this KIP, it would be better to also have the controller reject an `empty` value

Yes, I fully agree that we should disallow an empty list for this configuration, since specifying an 
empty list is odd, it would mean that the node does not advertise any listener addresses, 
making it unreachable by clients. I will updated KIP according by this.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@apache.org> 於 2025年8月21日 晚上8:48 寫道：
> 
> 2. It seems that advertised.listeners should default to null and can't be
> 
> That is an inconsistency between the broker and controller, and it will be addressed by KIP-1202. `null` is valid for both, while `empty` is valid only for the controller. As Jun mentioned, the broker encounters an error in this case.
> (By the way, the table in KIP-1202 has an incorrect value — null is acceptable for both cases.)
> 
> If we want to address `advertised.listeners` in this KIP, it would be better to also have the controller reject an `empty` value
> 
> Best,
> Chia-Ping
> 
> On 2025/08/20 18:01:35 Jun Rao wrote:
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the update.
>> 
>> 1. Sounds good.
>> 
>> 2. It seems that advertised.listeners should default to null and can't be
>> empty. Currently, if it's set to empty, it fails with the following.
>> 
>> java.lang.IllegalArgumentException: requirement failed: There must be at
>> least one broker advertised listener. Perhaps all listeners appear in
>> controller.listener.names?
>> at scala.Predef$.require(Predef.scala:337) ~[scala-library-2.13.16.jar:?]
>> at
>> kafka.server.KafkaConfig.validateAdvertisedBrokerListenersNonEmptyForBroker$1(KafkaConfig.scala:545)
>> ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
>> 
>> Jun
>> 
>> On Wed, Aug 20, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hello Jun,
>>> 
>>> I have updated the KIP with the following changes.
>>> 
>>> 1. The type of controller.listener.names should be changed from string to
>>> list.
>>> Its default value should be updated from null to NO_DEFAULT_VALUE, and its
>>> validator should be updated to anyNonDuplicateValues(isNullAllowed =
>>> false, isEmptyAllowed = false).
>>> 
>>> 2. The type of advertised.listeners should be changed from string to list.
>>> As for its validator,
>>> I think we can continue the discussion in KIP-1202.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月14日 凌晨12:37 寫道：
>>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Thanks for the updated KIP. Looks good to me.
>>>> 
>>>> Jun
>>>> 
>>>> On Wed, Aug 13, 2025 at 3:08 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hello Jun,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> I have updated the KIP according there comments.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
>>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply. A few more comments.
>>>>>> 
>>>>>> JR50. It seems that you changed the default value for config.providers
>>>>>> incorrectly. The change is meant for bootstrap.servers.
>>>>>> 
>>>>>> JR51. Could you document the current behavior if bootstrap.servers is
>>>>> empty
>>>>>> in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
>>>>>> 
>>>>>> JR52. Could you document the justification for changing the default
>>> value
>>>>>> for bootstrap.servers in WorkerConfig?
>>>>>> 
>>>>>> JR53. It seems that WorkerConfig is not public facing. Only classes in
>>>>>> connect api are public. So, there is no need to document the
>>> deprecation
>>>>>> of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> 
>>>>>> On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi chia,
>>>>>>> 
>>>>>>> I have updated the KIP with these changes.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
>>>>>>>> 
>>>>>>>> hi Jiunn
>>>>>>>> 
>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
>>>>>>>>> 
>>>>>>>>> 3. WorkerConfig – Change the default value of bootstrap.servers from
>>>>>>> "localhost:9092" to NO_DEFAULT_VALUE
>>>>>>>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I agree that the default value of “localhost:9092” is strange.
>>> However,
>>>>>>> it is still a breaking change, so please highlight this change in the
>>>>> KIP.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@apache.org>]

2. It seems that advertised.listeners should default to null and can't be

That is an inconsistency between the broker and controller, and it will be addressed by KIP-1202. `null` is valid for both, while `empty` is valid only for the controller. As Jun mentioned, the broker encounters an error in this case.
(By the way, the table in KIP-1202 has an incorrect value — null is acceptable for both cases.)

If we want to address `advertised.listeners` in this KIP, it would be better to also have the controller reject an `empty` value

Best,
Chia-Ping

On 2025/08/20 18:01:35 Jun Rao wrote:
> Hi, Jiunn-Yang,
> 
> Thanks for the update.
> 
> 1. Sounds good.
> 
> 2. It seems that advertised.listeners should default to null and can't be
> empty. Currently, if it's set to empty, it fails with the following.
> 
> java.lang.IllegalArgumentException: requirement failed: There must be at
> least one broker advertised listener. Perhaps all listeners appear in
> controller.listener.names?
> at scala.Predef$.require(Predef.scala:337) ~[scala-library-2.13.16.jar:?]
> at
> kafka.server.KafkaConfig.validateAdvertisedBrokerListenersNonEmptyForBroker$1(KafkaConfig.scala:545)
> ~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]
> 
> Jun
> 
> On Wed, Aug 20, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
> > Hello Jun,
> >
> > I have updated the KIP with the following changes.
> >
> > 1. The type of controller.listener.names should be changed from string to
> > list.
> > Its default value should be updated from null to NO_DEFAULT_VALUE, and its
> > validator should be updated to anyNonDuplicateValues(isNullAllowed =
> > false, isEmptyAllowed = false).
> >
> > 2. The type of advertised.listeners should be changed from string to list.
> > As for its validator,
> > I think we can continue the discussion in KIP-1202.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月14日 凌晨12:37 寫道：
> > >
> > > Hi, Jiunn-Yang,
> > >
> > > Thanks for the updated KIP. Looks good to me.
> > >
> > > Jun
> > >
> > > On Wed, Aug 13, 2025 at 3:08 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >
> > >> Hello Jun,
> > >>
> > >> Thanks for the reply.
> > >>
> > >> I have updated the KIP according there comments.
> > >>
> > >> Best Regards,
> > >> Jiunn-Yang
> > >>
> > >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
> > >>>
> > >>> Hi, Jiunn-Yang,
> > >>>
> > >>> Thanks for the reply. A few more comments.
> > >>>
> > >>> JR50. It seems that you changed the default value for config.providers
> > >>> incorrectly. The change is meant for bootstrap.servers.
> > >>>
> > >>> JR51. Could you document the current behavior if bootstrap.servers is
> > >> empty
> > >>> in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
> > >>>
> > >>> JR52. Could you document the justification for changing the default
> > value
> > >>> for bootstrap.servers in WorkerConfig?
> > >>>
> > >>> JR53. It seems that WorkerConfig is not public facing. Only classes in
> > >>> connect api are public. So, there is no need to document the
> > deprecation
> > >>> of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
> > >>>
> > >>> Jun
> > >>>
> > >>>
> > >>> On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>
> > >>>> Hi chia,
> > >>>>
> > >>>> I have updated the KIP with these changes.
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
> > >>>>>
> > >>>>> hi Jiunn
> > >>>>>
> > >>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
> > >>>>>>
> > >>>>>> 3. WorkerConfig – Change the default value of bootstrap.servers from
> > >>>> "localhost:9092" to NO_DEFAULT_VALUE
> > >>>>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
> > >>>>>
> > >>>>>
> > >>>>> I agree that the default value of “localhost:9092” is strange.
> > However,
> > >>>> it is still a breaking change, so please highlight this change in the
> > >> KIP.
> > >>>>>
> > >>>>> Best,
> > >>>>> Chia-Ping
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the update.

1. Sounds good.

2. It seems that advertised.listeners should default to null and can't be
empty. Currently, if it's set to empty, it fails with the following.

java.lang.IllegalArgumentException: requirement failed: There must be at
least one broker advertised listener. Perhaps all listeners appear in
controller.listener.names?
at scala.Predef$.require(Predef.scala:337) ~[scala-library-2.13.16.jar:?]
at
kafka.server.KafkaConfig.validateAdvertisedBrokerListenersNonEmptyForBroker$1(KafkaConfig.scala:545)
~[kafka_2.13-4.2.0-SNAPSHOT.jar:?]

Jun

On Wed, Aug 20, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> I have updated the KIP with the following changes.
>
> 1. The type of controller.listener.names should be changed from string to
> list.
> Its default value should be updated from null to NO_DEFAULT_VALUE, and its
> validator should be updated to anyNonDuplicateValues(isNullAllowed =
> false, isEmptyAllowed = false).
>
> 2. The type of advertised.listeners should be changed from string to list.
> As for its validator,
> I think we can continue the discussion in KIP-1202.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月14日 凌晨12:37 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the updated KIP. Looks good to me.
> >
> > Jun
> >
> > On Wed, Aug 13, 2025 at 3:08 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello Jun,
> >>
> >> Thanks for the reply.
> >>
> >> I have updated the KIP according there comments.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
> >>>
> >>> Hi, Jiunn-Yang,
> >>>
> >>> Thanks for the reply. A few more comments.
> >>>
> >>> JR50. It seems that you changed the default value for config.providers
> >>> incorrectly. The change is meant for bootstrap.servers.
> >>>
> >>> JR51. Could you document the current behavior if bootstrap.servers is
> >> empty
> >>> in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
> >>>
> >>> JR52. Could you document the justification for changing the default
> value
> >>> for bootstrap.servers in WorkerConfig?
> >>>
> >>> JR53. It seems that WorkerConfig is not public facing. Only classes in
> >>> connect api are public. So, there is no need to document the
> deprecation
> >>> of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>
> >>>> Hi chia,
> >>>>
> >>>> I have updated the KIP with these changes.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
> >>>>>
> >>>>> hi Jiunn
> >>>>>
> >>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
> >>>>>>
> >>>>>> 3. WorkerConfig – Change the default value of bootstrap.servers from
> >>>> "localhost:9092" to NO_DEFAULT_VALUE
> >>>>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
> >>>>>
> >>>>>
> >>>>> I agree that the default value of “localhost:9092” is strange.
> However,
> >>>> it is still a breaking change, so please highlight this change in the
> >> KIP.
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

I have updated the KIP with the following changes.

1. The type of controller.listener.names should be changed from string to list. 
Its default value should be updated from null to NO_DEFAULT_VALUE, and its 
validator should be updated to anyNonDuplicateValues(isNullAllowed = false, isEmptyAllowed = false).

2. The type of advertised.listeners should be changed from string to list. As for its validator, 
I think we can continue the discussion in KIP-1202.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月14日 凌晨12:37 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the updated KIP. Looks good to me.
> 
> Jun
> 
> On Wed, Aug 13, 2025 at 3:08 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun,
>> 
>> Thanks for the reply.
>> 
>> I have updated the KIP according there comments.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
>>> 
>>> Hi, Jiunn-Yang,
>>> 
>>> Thanks for the reply. A few more comments.
>>> 
>>> JR50. It seems that you changed the default value for config.providers
>>> incorrectly. The change is meant for bootstrap.servers.
>>> 
>>> JR51. Could you document the current behavior if bootstrap.servers is
>> empty
>>> in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
>>> 
>>> JR52. Could you document the justification for changing the default value
>>> for bootstrap.servers in WorkerConfig?
>>> 
>>> JR53. It seems that WorkerConfig is not public facing. Only classes in
>>> connect api are public. So, there is no need to document the deprecation
>>> of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
>>> 
>>> Jun
>>> 
>>> 
>>> On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hi chia,
>>>> 
>>>> I have updated the KIP with these changes.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
>>>>> 
>>>>> hi Jiunn
>>>>> 
>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
>>>>>> 
>>>>>> 3. WorkerConfig – Change the default value of bootstrap.servers from
>>>> "localhost:9092" to NO_DEFAULT_VALUE
>>>>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
>>>>> 
>>>>> 
>>>>> I agree that the default value of “localhost:9092” is strange. However,
>>>> it is still a breaking change, so please highlight this change in the
>> KIP.
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the updated KIP. Looks good to me.

Jun

On Wed, Aug 13, 2025 at 3:08 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> Thanks for the reply.
>
> I have updated the KIP according there comments.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the reply. A few more comments.
> >
> > JR50. It seems that you changed the default value for config.providers
> > incorrectly. The change is meant for bootstrap.servers.
> >
> > JR51. Could you document the current behavior if bootstrap.servers is
> empty
> > in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
> >
> > JR52. Could you document the justification for changing the default value
> > for bootstrap.servers in WorkerConfig?
> >
> > JR53. It seems that WorkerConfig is not public facing. Only classes in
> > connect api are public. So, there is no need to document the deprecation
> > of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
> >
> > Jun
> >
> >
> > On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hi chia,
> >>
> >> I have updated the KIP with these changes.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
> >>>
> >>> hi Jiunn
> >>>
> >>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
> >>>>
> >>>> 3. WorkerConfig – Change the default value of bootstrap.servers from
> >> "localhost:9092" to NO_DEFAULT_VALUE
> >>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
> >>>
> >>>
> >>> I agree that the default value of “localhost:9092” is strange. However,
> >> it is still a breaking change, so please highlight this change in the
> KIP.
> >>>
> >>> Best,
> >>> Chia-Ping
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

Thanks for the reply.

I have updated the KIP according there comments.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月13日 凌晨1:57 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply. A few more comments.
> 
> JR50. It seems that you changed the default value for config.providers
> incorrectly. The change is meant for bootstrap.servers.
> 
> JR51. Could you document the current behavior if bootstrap.servers is empty
> in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?
> 
> JR52. Could you document the justification for changing the default value
> for bootstrap.servers in WorkerConfig?
> 
> JR53. It seems that WorkerConfig is not public facing. Only classes in
> connect api are public. So, there is no need to document the deprecation
> of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.
> 
> Jun
> 
> 
> On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hi chia,
>> 
>> I have updated the KIP with these changes.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
>>> 
>>> hi Jiunn
>>> 
>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
>>>> 
>>>> 3. WorkerConfig – Change the default value of bootstrap.servers from
>> "localhost:9092" to NO_DEFAULT_VALUE
>>>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
>>> 
>>> 
>>> I agree that the default value of “localhost:9092” is strange. However,
>> it is still a breaking change, so please highlight this change in the KIP.
>>> 
>>> Best,
>>> Chia-Ping
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply. A few more comments.

JR50. It seems that you changed the default value for config.providers
incorrectly. The change is meant for bootstrap.servers.

JR51. Could you document the current behavior if bootstrap.servers is empty
in ConsumerConfig, WorkerConfig, ProducerConfig and StreamsConfig?

JR52. Could you document the justification for changing the default value
for bootstrap.servers in WorkerConfig?

JR53. It seems that WorkerConfig is not public facing. Only classes in
connect api are public. So, there is no need to document the deprecation
of BOOTSTRAP_SERVERS_DEFAULT in WorkerConfig.

Jun


On Tue, Aug 12, 2025 at 8:14 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hi chia,
>
> I have updated the KIP with these changes.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
> >
> > hi Jiunn
> >
> >> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
> >>
> >> 3. WorkerConfig – Change the default value of bootstrap.servers from
> "localhost:9092" to NO_DEFAULT_VALUE
> >> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
> >
> >
> > I agree that the default value of “localhost:9092” is strange. However,
> it is still a breaking change, so please highlight this change in the KIP.
> >
> > Best,
> > Chia-Ping
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi chia,

I have updated the KIP with these changes.

Best Regards, 
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月12日 晚上9:54 寫道：
> 
> hi Jiunn
> 
>> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
>> 
>> 3. WorkerConfig – Change the default value of bootstrap.servers from "localhost:9092" to NO_DEFAULT_VALUE
>> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.
> 
> 
> I agree that the default value of “localhost:9092” is strange. However, it is still a breaking change, so please highlight this change in the KIP.
> 
> Best,
> Chia-Ping

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

hi Jiunn

> 黃竣陽 <s7...@gmail.com> 於 2025年8月12日 下午6:18 寫道：
> 
> 3. WorkerConfig – Change the default value of bootstrap.servers from "localhost:9092" to NO_DEFAULT_VALUE
> and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.


I agree that the default value of “localhost:9092” is strange. However, it is still a breaking change, so please highlight this change in the KIP.

Best,
Chia-Ping

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

2. My original conclusion was incorrect. bootstrap.servers can support NO_DEFAULT_VALUE with the validator ConfigDef.ValidList.anyNonDuplicateValues(false, false). I propose updating the KIP with the following changes:

1.ConsumerConfig – Change the default value of bootstrap.servers from List.of() to NO_DEFAULT_VALUE.
2. ProducerConfig – Change the default value of bootstrap.servers from List.of() to NO_DEFAULT_VALUE.
3. WorkerConfig – Change the default value of bootstrap.servers from "localhost:9092" to NO_DEFAULT_VALUE 
and deprecate the constant BOOTSTRAP_SERVERS_DEFAULT.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月12日 凌晨3:03 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply.
> 
> 1 and 3. Sounds good.
> 
> 2. Which bootstrap.servers are you referring to? Also, it's not clear to me
> why the validation fails since typically the validation happens when the
> config is initialized with the binding config file.
> 
> Jun
> 
> On Mon, Aug 11, 2025 at 6:38 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello all,
>> 
>> I’ve started implementing this KIP, and there are a few configurations
>> that need further discussion:
>> 
>> 1. HeaderFrom fields
>> This configuration should allow duplicate values. Therefore, its default
>> value
>> should be set to NO_DEFAULT_VALUE, and its validator should be
>> NonEmptyListValidator().
>> See the test case [1] for reference.
>> 
>> 2. bootstrap.servers
>> If we set its default value to NO_DEFAULT_VALUE and use the validator
>> ConfigDef.ValidList.anyNonDuplicateValues(false, false),
>> initialization will fail because the default value does not pass the
>> validator. I see two possible solutions:
>> 
>> Option 2.1: Set the default value to List.of("localhost:9092") (similar to
>> WorkerConfig) and use the validator
>> ConfigDef.ValidList.anyNonDuplicateValues(false, false). However, this
>> feels a bit odd, as the default value
>> should ideally be useful, and localhost:9092 is not meaningful for most
>> scenarios.
>> 
>> Option 2.2: Keep the default value as NO_DEFAULT_VALUE and use
>> NonEmptyListValidator(). In this case,
>> we would need to move NonEmptyListValidator from the connect.transforms
>> module to the client module,
>> and make it a public API within ConfigDef. This would also align with how
>> most bootstrap.servers configs are handled.
>> 
>> 3. Remove Cast fields config change
>> This configuration has its own validator, so we should leave it unchanged.
>> 
>> [1]
>> https://github.com/apache/kafka/blob/18045c6ac30921503deffbef1744bb365dc599fb/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/HeaderFromTest.java#L240
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月6日 晚上11:59 寫道：
>>> 
>>> Hi Jun,
>>> 
>>> Thanks for the reply,
>>> 
>>> I have updated the KIP
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月6日 晚上11:34 寫道：
>>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Regarding bootstrap.servers in MirrorClientConfig, it's weird that it
>>>> defaults to an empty list while empty list is not allowed. It seems that
>>>> this should have no default value.
>>>> 
>>>> Thanks,
>>>> 
>>>> Jun
>>>> 
>>>> On Wed, Aug 6, 2025 at 7:56 AM Chia-Ping Tsai <ch...@gmail.com>
>> wrote:
>>>> 
>>>>> Thanks to Jiunn-Yang for this hard KIP.
>>>>> 
>>>>> It might not be perfect, but we can refine it during the PR review
>> process
>>>>> :)
>>>>> 
>>>>> 
>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月6日 週三 下午7:32寫道：
>>>>> 
>>>>>> Hello chia,
>>>>>> 
>>>>>> Thanks for the reply.
>>>>>> 
>>>>>> chia05: I have updated the KIP to change the default values of
>>>>>> task.assigned.groups
>>>>>> and task.assigned.partitions from null to NO_DEFAULT_VALUE.
>>>>>> 
>>>>>> chia06: I have also updated the KIP based on your feedback.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月6日 凌晨12:54 寫道：
>>>>>>> 
>>>>>>> hi Jiunn-Yang
>>>>>>> 
>>>>>>> chia05:
>>>>>>> 
>>>>>>> `task.assigned.groups` and `task.assigned.partitions` should not have
>>>>>>> default value, right?
>>>>>>> 
>>>>>>> chia06:
>>>>>>> 
>>>>>>> `new HashSet<>(values).size()` -> `Set.copyOf(values).size()`
>>>>>>> 
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年8月5日 週二 下午11:39寫道：
>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the reply and your patience. The KIP looks good to me
>> now.
>>>>>>>> 
>>>>>>>> Hi, Luke, Mickael,
>>>>>>>> 
>>>>>>>> Could you take a look at the KIP again and see if it makes sense to
>>>>> you?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hello Jun,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply,
>>>>>>>>> 
>>>>>>>>> JR46: I have clarified the behavior for null and empty values, so
>>>>> users
>>>>>>>>> can
>>>>>>>>> understand the reasoning behind disallowing empty lists.
>>>>>>>>> 
>>>>>>>>> JR48: I've updated the KIP accordingly.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
>>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply.
>>>>>>>>>> 
>>>>>>>>>> JR46. "Currently, if plugin.path is set to an empty list, no
>>>>>>>>> user-specified
>>>>>>>>>> plugin locations are processed, but plugins from the parent
>>>>>> classloader
>>>>>>>>> are
>>>>>>>>>> still loaded." Could you add why the current behavior is bad for
>> the
>>>>>>>>> users?
>>>>>>>>>> 
>>>>>>>>>> JR48. If we want to preserve the current behavior, we should
>> remove
>>>>>> the
>>>>>>>>>> following validation, right?
>>>>>>>>>>    if (validStrings.length == 0) {
>>>>>>>>>>        throw new IllegalArgumentException("Valid strings list
>>>>>>>> cannot
>>>>>>>>>> be empty for in validator");
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello Jun,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>> 
>>>>>>>>>>> JR42, JR45, JR46, JR47: I have updated the KIP.
>>>>>>>>>>> 
>>>>>>>>>>> JR43: If bootstrap.servers is empty for these configurations,
>>>>>>>>>>> it will fail when creating the AdminClient. I have updated the
>> KIP
>>>>> to
>>>>>>>>>>> reflect this behavior.
>>>>>>>>>>> 
>>>>>>>>>>> JR44: Thanks for pointing this out. Kafka will indeed fail at the
>>>>>>>>> storage
>>>>>>>>>>> format phase
>>>>>>>>>>> if the configuration is invalid. I have updated the KIP
>>>>> accordingly.
>>>>>>>>>>> 
>>>>>>>>>>> JR48: The current in() method allows empty lists by default, so I
>>>>>>>>> believe
>>>>>>>>>>> setting isEmptyAllowed to true
>>>>>>>>>>> better preserves the original behavior. Therefore, I prefer not
>> to
>>>>>>>>> change
>>>>>>>>>>> this default.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the reply. A few more comments.
>>>>>>>>>>>> 
>>>>>>>>>>>> JR42. [6] [7] are not needed since currently an empty list
>> already
>>>>>>>>> causes
>>>>>>>>>>>> an exception.
>>>>>>>>>>>> 
>>>>>>>>>>>> JR43. Could you document the current behavior when
>>>>> bootstrap.servers
>>>>>>>> is
>>>>>>>>>>>> empty in MirrorClientConfig and WorkerConfig?
>>>>>>>>>>>> 
>>>>>>>>>>>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs
>> empty
>>>>>>>> and
>>>>>>>>>>> got
>>>>>>>>>>>> the following.
>>>>>>>>>>>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
>>>>>>>>>>>> config/server.properties
>>>>>>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>>>>>> requirement
>>>>>>>>>>>> failed: At least one log directory must be defined via log.dirs
>> or
>>>>>>>>>>> log.dir.
>>>>>>>>>>>> 
>>>>>>>>>>>> bin/kafka-server-start.sh config/server.properties
>>>>>>>>>>>> [2025-08-01 11:31:20,097] INFO Registered
>>>>>>>>>>>> `kafka:type=kafka.Log4jController` MBean
>>>>>>>>>>>> (kafka.utils.Log4jControllerRegistration$)
>>>>>>>>>>>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal
>>>>> exception
>>>>>>>>>>>> (kafka.Kafka$)
>>>>>>>>>>>> java.lang.IllegalArgumentException: requirement failed: At least
>>>>> one
>>>>>>>>> log
>>>>>>>>>>>> directory must be defined via log.dirs or log.dir.
>>>>>>>>>>>> 
>>>>>>>>>>>> JR45. Could you group all configs for WorkerConfig together?
>>>>>>>>>>>> 
>>>>>>>>>>>> JR46. plugin.path: We have the following code. So, currently, it
>>>>>>>> seems
>>>>>>>>>>> that
>>>>>>>>>>>> an empty list is allowed for plugin.path and we treat it by
>> adding
>>>>>>>> the
>>>>>>>>>>>> parent path of the classloader. It would be useful to justify
>> why
>>>>>>>>>>>> disallowing empty is better for the users.
>>>>>>>>>>>> 
>>>>>>>>>>>> public static Set<PluginSource> pluginSources(Set<Path>
>>>>>>>>>>>> pluginLocations, ClassLoader classLoader,
>> PluginClassLoaderFactory
>>>>>>>>>>> factory)
>>>>>>>>>>>> {
>>>>>>>>>>>>   Set<PluginSource> pluginSources = new LinkedHashSet<>();
>>>>>>>>>>>>   for (Path pluginLocation : pluginLocations) {
>>>>>>>>>>>>       try {
>>>>>>>>>>>> 
>>>>> pluginSources.add(isolatedPluginSource(pluginLocation,
>>>>>>>>>>>> classLoader, factory));
>>>>>>>>>>>>       } catch (InvalidPathException | MalformedURLException e)
>>>>> {
>>>>>>>>>>>>           log.error("Invalid path in plugin path: {}.
>>>>> Ignoring.",
>>>>>>>>>>>> pluginLocation, e);
>>>>>>>>>>>>       } catch (IOException e) {
>>>>>>>>>>>>           log.error("Could not get listing for plugin path: {}.
>>>>>>>>>>>> Ignoring.", pluginLocation, e);
>>>>>>>>>>>>       }
>>>>>>>>>>>>   }
>>>>>>>>>>>> 
>>>>>>>>> pluginSources.add(classpathPluginSource(classLoader.getParent()));
>>>>>>>>>>>>   return pluginSources;
>>>>>>>>>>>> }
>>>>>>>>>>>> 
>>>>>>>>>>>> JR47. typo in "Should not allow setting validStrings is empty"
>>>>>>>>>>>> 
>>>>>>>>>>>> JR48. We should pass in isEmptyAllowed as false in the code
>> below,
>>>>>>>>> right?
>>>>>>>>>>>> public static ValidList in(String... validStrings) {
>>>>>>>>>>>>   if (validStrings.length == 0) {
>>>>>>>>>>>>       throw new IllegalArgumentException("Valid strings list
>>>>>>>> cannot
>>>>>>>>>>>> be empty for in validator");
>>>>>>>>>>>>   }
>>>>>>>>>>>>   return new ValidList(List.of(validStrings), true, false);
>>>>>>>>>>>> }
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Sorry for the late reply,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR40, JR41: I have updated the KIP based on your comments.
>> Please
>>>>>>>>> take a
>>>>>>>>>>>>> look.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR40. In the table, there are a few configs for which we now
>>>>>>>> disallow
>>>>>>>>>>>>> empty
>>>>>>>>>>>>>> lists. It would be useful to document the current behavior for
>>>>>> each
>>>>>>>>> of
>>>>>>>>>>>>> them
>>>>>>>>>>>>>> more accurately. It seems that they should all lead to some
>> bad
>>>>>>>>>>> behavior.
>>>>>>>>>>>>>> For example, if log.dirs is empty, it seems that we can't
>> create
>>>>>>>>>>> topics?
>>>>>>>>>>>>>> If group.coordinator.rebalance.protocols is empty, the group
>>>>>>>>> rebalance
>>>>>>>>>>>>>> protocol will fail?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR41. It would be useful to clean up the "Compatibility,
>>>>>>>> Deprecation,
>>>>>>>>>>> and
>>>>>>>>>>>>>> Migration Plan" section a bit.
>>>>>>>>>>>>>> JR41.1 The migration plan has "Introduce a new properties
>>>>>>>>>>> isEmptyAllowed
>>>>>>>>>>>>> ,
>>>>>>>>>>>>>> isNullAllowed", which needs to be changed.
>>>>>>>>>>>>>> JR41.2 It started with "For users who configure an empty
>> list",
>>>>>> but
>>>>>>>>> not
>>>>>>>>>>>>> all
>>>>>>>>>>>>>> the bullet points are about empty lists.
>>>>>>>>>>>>>> JR41.3 To me, it would be useful to summarize why the behavior
>>>>>>>>> changes
>>>>>>>>>>>>> are
>>>>>>>>>>>>>> justified. We made 3 types of changes. (1) Disallow null. This
>>>>> is
>>>>>>>> ok
>>>>>>>>>>>>> since
>>>>>>>>>>>>>> one can't really set a null value through a config file. (2)
>>>>>>>> Disallow
>>>>>>>>>>>>>> duplicates. This is ok since it's rare and confusing. (3)
>>>>> Disallow
>>>>>>>>>>> empty.
>>>>>>>>>>>>>> Ideally, we want to claim that for those cases, they all lead
>> to
>>>>>>>> some
>>>>>>>>>>>>> other
>>>>>>>>>>>>>> errors or had behavior currently.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these
>>>>> changes.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR38: I clarified the following configurations:
>>>>>>>>>>>>>>> early.start.listeners: This config is optional and allows
>> both
>>>>>>>> null
>>>>>>>>>>> and
>>>>>>>>>>>>>>> empty lists.
>>>>>>>>>>>>>>> log.dirs and plugin.path: These are also optional, but only
>>>>> allow
>>>>>>>>>>>>>>> null—empty lists are not permitted.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR34. Sounds good.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR36. Could you document that the default value for
>>>>>>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites
>> have
>>>>>>>> been
>>>>>>>>>>>>>>> changed?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR38. My concern wasn't quite addressed. "If the
>> configuration
>>>>>> is
>>>>>>>>>>>>>>> optional,
>>>>>>>>>>>>>>>> we will reject any duplicate values in the list." This
>> implies
>>>>>>>> that
>>>>>>>>>>> if
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> config is optional, we allow empty lists. However, for
>>>>>>>> plugin.path,
>>>>>>>>>>> we
>>>>>>>>>>>>>>>> reject empty lists.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR39. We can remove the fields isEmptyAllowed and
>>>>> isNullAllowed
>>>>>>>>> since
>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>> are not public facing.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello Jun, Chia,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the reply,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR36: I’ve updated the following configurations:
>>>>>>>>>>>>>>>>> - sasl.oauthbearer.expected.audience
>>>>>>>>>>>>>>>>> - ssl.cipher.suites
>>>>>>>>>>>>>>>>> - ssl.enabled.protocols
>>>>>>>>>>>>>>>>> These now allow both empty and non-empty lists. Since the
>>>>>>>> default
>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites
>> were
>>>>>>>>>>>>> previously
>>>>>>>>>>>>>>>>> null,
>>>>>>>>>>>>>>>>> I’ve updated their defaults to List.of() for consistency.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR37: A ClassCastException does not occur in
>>>>> MirrorClientConfig
>>>>>>>>> and
>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> should be none.
>>>>>>>>>>>>>>>>> I’ve updated this in the KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate
>>>>>> which
>>>>>>>>>>>>>>>>> configurations require special attention.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR39: Yes, these properties are package-private.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04
>> 寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
>>>>>> return
>>>>>>>>> an
>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>>>>>>>>>>> maxTasks,
>>>>>>>>>>>>>>>>>> Chia-Ping?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an
>>>>> empty
>>>>>>>>>>> list,
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> means no `MirrorSourceTask` can be created.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the
>>>>> configs
>>>>>>>>> used
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions`
>>>>>>>> with a
>>>>>>>>>>>>>>>>> non-empty
>>>>>>>>>>>>>>>>>> value.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二
>>>>> 上午5:14寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
>>>>>>>> return
>>>>>>>>> an
>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less
>> than
>>>>>>>>>>> maxTasks,
>>>>>>>>>>>>>>>>>>> Chia-Ping?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should
>>>>> we
>>>>>>>> do
>>>>>>>>>>> the
>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>> for ssl.cipher.suites and
>>>>> sasl.oauthbearer.expected.audience?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR37. Is it true that
>> MirrorClientConfig.bootstrap.servers
>>>>>>>>> returns
>>>>>>>>>>>>>>>>>>> ClassCastException for null? It seems that null could be
>>>>>>>> casted
>>>>>>>>> to
>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR38. "If the configuration is optional, we will reject
>> any
>>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>>>>> values in the list." Could you clarify this? For example,
>>>>>>>>>>>>> plugin.path
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> optional, but with a default value. We reject duplicates
>> as
>>>>>>>> well
>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>> lists.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public
>>>>> fields,
>>>>>>>>>>> right?
>>>>>>>>>>>>>>>>> They
>>>>>>>>>>>>>>>>>>> are only public in the anyNonDuplicateValues() method.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR40. It would be useful to mention that if
>> cleanup.policy
>>>>> is
>>>>>>>>>>> empty
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> remote.storage.enable is true, the local log segments
>> will
>>>>> be
>>>>>>>>>>>>> cleaned
>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
>>>>>>>>>>> chia7712@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> JR34: For these two configurations, setting an empty
>> list
>>>>>>>>> feels
>>>>>>>>>>> a
>>>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>>>>>>>> unintuitive. If an empty list is
>>>>>>>>>>>>>>>>>>>>> provided, the consumer will call the unsubscribe
>> method,
>>>>>>>> which
>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>>>> seem appropriate given
>>>>>>>>>>>>>>>>>>>>> the documentation states: "Topic-partitions assigned to
>>>>>> this
>>>>>>>>>>> task
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> replicate."
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> agreed. Additionally, the source code [0] shows that it
>> is
>>>>>>>> not
>>>>>>>>>>>>>>> intended
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> generate empty tps for the source task
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply.

1 and 3. Sounds good.

2. Which bootstrap.servers are you referring to? Also, it's not clear to me
why the validation fails since typically the validation happens when the
config is initialized with the binding config file.

Jun

On Mon, Aug 11, 2025 at 6:38 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello all,
>
> I’ve started implementing this KIP, and there are a few configurations
> that need further discussion:
>
> 1. HeaderFrom fields
> This configuration should allow duplicate values. Therefore, its default
> value
> should be set to NO_DEFAULT_VALUE, and its validator should be
> NonEmptyListValidator().
> See the test case [1] for reference.
>
> 2. bootstrap.servers
> If we set its default value to NO_DEFAULT_VALUE and use the validator
> ConfigDef.ValidList.anyNonDuplicateValues(false, false),
> initialization will fail because the default value does not pass the
> validator. I see two possible solutions:
>
> Option 2.1: Set the default value to List.of("localhost:9092") (similar to
> WorkerConfig) and use the validator
> ConfigDef.ValidList.anyNonDuplicateValues(false, false). However, this
> feels a bit odd, as the default value
> should ideally be useful, and localhost:9092 is not meaningful for most
> scenarios.
>
> Option 2.2: Keep the default value as NO_DEFAULT_VALUE and use
> NonEmptyListValidator(). In this case,
> we would need to move NonEmptyListValidator from the connect.transforms
> module to the client module,
> and make it a public API within ConfigDef. This would also align with how
> most bootstrap.servers configs are handled.
>
> 3. Remove Cast fields config change
> This configuration has its own validator, so we should leave it unchanged.
>
> [1]
> https://github.com/apache/kafka/blob/18045c6ac30921503deffbef1744bb365dc599fb/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/HeaderFromTest.java#L240
>
> Best Regards,
> Jiunn-Yang
>
> > 黃竣陽 <s7...@gmail.com> 於 2025年8月6日 晚上11:59 寫道：
> >
> > Hi Jun,
> >
> > Thanks for the reply,
> >
> > I have updated the KIP
> >
> > Best Regards,
> > Jiunn-Yang
> >
> >> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月6日 晚上11:34 寫道：
> >>
> >> Hi, Jiunn-Yang,
> >>
> >> Regarding bootstrap.servers in MirrorClientConfig, it's weird that it
> >> defaults to an empty list while empty list is not allowed. It seems that
> >> this should have no default value.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Wed, Aug 6, 2025 at 7:56 AM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >>
> >>> Thanks to Jiunn-Yang for this hard KIP.
> >>>
> >>> It might not be perfect, but we can refine it during the PR review
> process
> >>> :)
> >>>
> >>>
> >>> 黃竣陽 <s7...@gmail.com> 於 2025年8月6日 週三 下午7:32寫道：
> >>>
> >>>> Hello chia,
> >>>>
> >>>> Thanks for the reply.
> >>>>
> >>>> chia05: I have updated the KIP to change the default values of
> >>>> task.assigned.groups
> >>>> and task.assigned.partitions from null to NO_DEFAULT_VALUE.
> >>>>
> >>>> chia06: I have also updated the KIP based on your feedback.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月6日 凌晨12:54 寫道：
> >>>>>
> >>>>> hi Jiunn-Yang
> >>>>>
> >>>>> chia05:
> >>>>>
> >>>>> `task.assigned.groups` and `task.assigned.partitions` should not have
> >>>>> default value, right?
> >>>>>
> >>>>> chia06:
> >>>>>
> >>>>> `new HashSet<>(values).size()` -> `Set.copyOf(values).size()`
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年8月5日 週二 下午11:39寫道：
> >>>>>
> >>>>>> Hi, Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the reply and your patience. The KIP looks good to me
> now.
> >>>>>>
> >>>>>> Hi, Luke, Mickael,
> >>>>>>
> >>>>>> Could you take a look at the KIP again and see if it makes sense to
> >>> you?
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hello Jun,
> >>>>>>>
> >>>>>>> Thanks for the reply,
> >>>>>>>
> >>>>>>> JR46: I have clarified the behavior for null and empty values, so
> >>> users
> >>>>>>> can
> >>>>>>> understand the reasoning behind disallowing empty lists.
> >>>>>>>
> >>>>>>> JR48: I've updated the KIP accordingly.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
> >>>>>>>>
> >>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> Thanks for the reply.
> >>>>>>>>
> >>>>>>>> JR46. "Currently, if plugin.path is set to an empty list, no
> >>>>>>> user-specified
> >>>>>>>> plugin locations are processed, but plugins from the parent
> >>>> classloader
> >>>>>>> are
> >>>>>>>> still loaded." Could you add why the current behavior is bad for
> the
> >>>>>>> users?
> >>>>>>>>
> >>>>>>>> JR48. If we want to preserve the current behavior, we should
> remove
> >>>> the
> >>>>>>>> following validation, right?
> >>>>>>>>     if (validStrings.length == 0) {
> >>>>>>>>         throw new IllegalArgumentException("Valid strings list
> >>>>>> cannot
> >>>>>>>> be empty for in validator");
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hello Jun,
> >>>>>>>>>
> >>>>>>>>> Thanks for the reply.
> >>>>>>>>>
> >>>>>>>>> JR42, JR45, JR46, JR47: I have updated the KIP.
> >>>>>>>>>
> >>>>>>>>> JR43: If bootstrap.servers is empty for these configurations,
> >>>>>>>>> it will fail when creating the AdminClient. I have updated the
> KIP
> >>> to
> >>>>>>>>> reflect this behavior.
> >>>>>>>>>
> >>>>>>>>> JR44: Thanks for pointing this out. Kafka will indeed fail at the
> >>>>>>> storage
> >>>>>>>>> format phase
> >>>>>>>>> if the configuration is invalid. I have updated the KIP
> >>> accordingly.
> >>>>>>>>>
> >>>>>>>>> JR48: The current in() method allows empty lists by default, so I
> >>>>>>> believe
> >>>>>>>>> setting isEmptyAllowed to true
> >>>>>>>>> better preserves the original behavior. Therefore, I prefer not
> to
> >>>>>>> change
> >>>>>>>>> this default.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
> >>>>>>>>>>
> >>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the reply. A few more comments.
> >>>>>>>>>>
> >>>>>>>>>> JR42. [6] [7] are not needed since currently an empty list
> already
> >>>>>>> causes
> >>>>>>>>>> an exception.
> >>>>>>>>>>
> >>>>>>>>>> JR43. Could you document the current behavior when
> >>> bootstrap.servers
> >>>>>> is
> >>>>>>>>>> empty in MirrorClientConfig and WorkerConfig?
> >>>>>>>>>>
> >>>>>>>>>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs
> empty
> >>>>>> and
> >>>>>>>>> got
> >>>>>>>>>> the following.
> >>>>>>>>>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
> >>>>>>>>>> config/server.properties
> >>>>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
> >>>>>>>>> requirement
> >>>>>>>>>> failed: At least one log directory must be defined via log.dirs
> or
> >>>>>>>>> log.dir.
> >>>>>>>>>>
> >>>>>>>>>> bin/kafka-server-start.sh config/server.properties
> >>>>>>>>>> [2025-08-01 11:31:20,097] INFO Registered
> >>>>>>>>>> `kafka:type=kafka.Log4jController` MBean
> >>>>>>>>>> (kafka.utils.Log4jControllerRegistration$)
> >>>>>>>>>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal
> >>> exception
> >>>>>>>>>> (kafka.Kafka$)
> >>>>>>>>>> java.lang.IllegalArgumentException: requirement failed: At least
> >>> one
> >>>>>>> log
> >>>>>>>>>> directory must be defined via log.dirs or log.dir.
> >>>>>>>>>>
> >>>>>>>>>> JR45. Could you group all configs for WorkerConfig together?
> >>>>>>>>>>
> >>>>>>>>>> JR46. plugin.path: We have the following code. So, currently, it
> >>>>>> seems
> >>>>>>>>> that
> >>>>>>>>>> an empty list is allowed for plugin.path and we treat it by
> adding
> >>>>>> the
> >>>>>>>>>> parent path of the classloader. It would be useful to justify
> why
> >>>>>>>>>> disallowing empty is better for the users.
> >>>>>>>>>>
> >>>>>>>>>> public static Set<PluginSource> pluginSources(Set<Path>
> >>>>>>>>>> pluginLocations, ClassLoader classLoader,
> PluginClassLoaderFactory
> >>>>>>>>> factory)
> >>>>>>>>>> {
> >>>>>>>>>>    Set<PluginSource> pluginSources = new LinkedHashSet<>();
> >>>>>>>>>>    for (Path pluginLocation : pluginLocations) {
> >>>>>>>>>>        try {
> >>>>>>>>>>
> >>> pluginSources.add(isolatedPluginSource(pluginLocation,
> >>>>>>>>>> classLoader, factory));
> >>>>>>>>>>        } catch (InvalidPathException | MalformedURLException e)
> >>> {
> >>>>>>>>>>            log.error("Invalid path in plugin path: {}.
> >>> Ignoring.",
> >>>>>>>>>> pluginLocation, e);
> >>>>>>>>>>        } catch (IOException e) {
> >>>>>>>>>>            log.error("Could not get listing for plugin path: {}.
> >>>>>>>>>> Ignoring.", pluginLocation, e);
> >>>>>>>>>>        }
> >>>>>>>>>>    }
> >>>>>>>>>>
> >>>>>>> pluginSources.add(classpathPluginSource(classLoader.getParent()));
> >>>>>>>>>>    return pluginSources;
> >>>>>>>>>> }
> >>>>>>>>>>
> >>>>>>>>>> JR47. typo in "Should not allow setting validStrings is empty"
> >>>>>>>>>>
> >>>>>>>>>> JR48. We should pass in isEmptyAllowed as false in the code
> below,
> >>>>>>> right?
> >>>>>>>>>> public static ValidList in(String... validStrings) {
> >>>>>>>>>>    if (validStrings.length == 0) {
> >>>>>>>>>>        throw new IllegalArgumentException("Valid strings list
> >>>>>> cannot
> >>>>>>>>>> be empty for in validator");
> >>>>>>>>>>    }
> >>>>>>>>>>    return new ValidList(List.of(validStrings), true, false);
> >>>>>>>>>> }
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>
> >>>>>>>>>>> Sorry for the late reply,
> >>>>>>>>>>>
> >>>>>>>>>>> JR40, JR41: I have updated the KIP based on your comments.
> Please
> >>>>>>> take a
> >>>>>>>>>>> look.
> >>>>>>>>>>>
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>
> >>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR40. In the table, there are a few configs for which we now
> >>>>>> disallow
> >>>>>>>>>>> empty
> >>>>>>>>>>>> lists. It would be useful to document the current behavior for
> >>>> each
> >>>>>>> of
> >>>>>>>>>>> them
> >>>>>>>>>>>> more accurately. It seems that they should all lead to some
> bad
> >>>>>>>>> behavior.
> >>>>>>>>>>>> For example, if log.dirs is empty, it seems that we can't
> create
> >>>>>>>>> topics?
> >>>>>>>>>>>> If group.coordinator.rebalance.protocols is empty, the group
> >>>>>>> rebalance
> >>>>>>>>>>>> protocol will fail?
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR41. It would be useful to clean up the "Compatibility,
> >>>>>> Deprecation,
> >>>>>>>>> and
> >>>>>>>>>>>> Migration Plan" section a bit.
> >>>>>>>>>>>> JR41.1 The migration plan has "Introduce a new properties
> >>>>>>>>> isEmptyAllowed
> >>>>>>>>>>> ,
> >>>>>>>>>>>> isNullAllowed", which needs to be changed.
> >>>>>>>>>>>> JR41.2 It started with "For users who configure an empty
> list",
> >>>> but
> >>>>>>> not
> >>>>>>>>>>> all
> >>>>>>>>>>>> the bullet points are about empty lists.
> >>>>>>>>>>>> JR41.3 To me, it would be useful to summarize why the behavior
> >>>>>>> changes
> >>>>>>>>>>> are
> >>>>>>>>>>>> justified. We made 3 types of changes. (1) Disallow null. This
> >>> is
> >>>>>> ok
> >>>>>>>>>>> since
> >>>>>>>>>>>> one can't really set a null value through a config file. (2)
> >>>>>> Disallow
> >>>>>>>>>>>> duplicates. This is ok since it's rare and confusing. (3)
> >>> Disallow
> >>>>>>>>> empty.
> >>>>>>>>>>>> Ideally, we want to claim that for those cases, they all lead
> to
> >>>>>> some
> >>>>>>>>>>> other
> >>>>>>>>>>>> errors or had behavior currently.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these
> >>> changes.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR38: I clarified the following configurations:
> >>>>>>>>>>>>> early.start.listeners: This config is optional and allows
> both
> >>>>>> null
> >>>>>>>>> and
> >>>>>>>>>>>>> empty lists.
> >>>>>>>>>>>>> log.dirs and plugin.path: These are also optional, but only
> >>> allow
> >>>>>>>>>>>>> null—empty lists are not permitted.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR34. Sounds good.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR36. Could you document that the default value for
> >>>>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites
> have
> >>>>>> been
> >>>>>>>>>>>>> changed?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR38. My concern wasn't quite addressed. "If the
> configuration
> >>>> is
> >>>>>>>>>>>>> optional,
> >>>>>>>>>>>>>> we will reject any duplicate values in the list." This
> implies
> >>>>>> that
> >>>>>>>>> if
> >>>>>>>>>>> a
> >>>>>>>>>>>>>> config is optional, we allow empty lists. However, for
> >>>>>> plugin.path,
> >>>>>>>>> we
> >>>>>>>>>>>>>> reject empty lists.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR39. We can remove the fields isEmptyAllowed and
> >>> isNullAllowed
> >>>>>>> since
> >>>>>>>>>>>>> they
> >>>>>>>>>>>>>> are not public facing.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com>
> >>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello Jun, Chia,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the reply,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR36: I’ve updated the following configurations:
> >>>>>>>>>>>>>>> - sasl.oauthbearer.expected.audience
> >>>>>>>>>>>>>>> - ssl.cipher.suites
> >>>>>>>>>>>>>>> - ssl.enabled.protocols
> >>>>>>>>>>>>>>> These now allow both empty and non-empty lists. Since the
> >>>>>> default
> >>>>>>>>>>> values
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites
> were
> >>>>>>>>>>> previously
> >>>>>>>>>>>>>>> null,
> >>>>>>>>>>>>>>> I’ve updated their defaults to List.of() for consistency.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR37: A ClassCastException does not occur in
> >>> MirrorClientConfig
> >>>>>>> and
> >>>>>>>>> it
> >>>>>>>>>>>>>>> should be none.
> >>>>>>>>>>>>>>> I’ve updated this in the KIP.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate
> >>>> which
> >>>>>>>>>>>>>>> configurations require special attention.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR39: Yes, these properties are package-private.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04
> 寫道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
> >>>> return
> >>>>>>> an
> >>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> >>>>>>>>> maxTasks,
> >>>>>>>>>>>>>>>> Chia-Ping?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an
> >>> empty
> >>>>>>>>> list,
> >>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>> means no `MirrorSourceTask` can be created.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the
> >>> configs
> >>>>>>> used
> >>>>>>>>>>> by
> >>>>>>>>>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions`
> >>>>>> with a
> >>>>>>>>>>>>>>> non-empty
> >>>>>>>>>>>>>>>> value.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二
> >>> 上午5:14寫道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
> >>>>>> return
> >>>>>>> an
> >>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less
> than
> >>>>>>>>> maxTasks,
> >>>>>>>>>>>>>>>>> Chia-Ping?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should
> >>> we
> >>>>>> do
> >>>>>>>>> the
> >>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>> for ssl.cipher.suites and
> >>> sasl.oauthbearer.expected.audience?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR37. Is it true that
> MirrorClientConfig.bootstrap.servers
> >>>>>>> returns
> >>>>>>>>>>>>>>>>> ClassCastException for null? It seems that null could be
> >>>>>> casted
> >>>>>>> to
> >>>>>>>>>>> any
> >>>>>>>>>>>>>>>>> class.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR38. "If the configuration is optional, we will reject
> any
> >>>>>>>>>>> duplicate
> >>>>>>>>>>>>>>>>> values in the list." Could you clarify this? For example,
> >>>>>>>>>>> plugin.path
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> optional, but with a default value. We reject duplicates
> as
> >>>>>> well
> >>>>>>>>> as
> >>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>> lists.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public
> >>> fields,
> >>>>>>>>> right?
> >>>>>>>>>>>>>>> They
> >>>>>>>>>>>>>>>>> are only public in the anyNonDuplicateValues() method.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR40. It would be useful to mention that if
> cleanup.policy
> >>> is
> >>>>>>>>> empty
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> remote.storage.enable is true, the local log segments
> will
> >>> be
> >>>>>>>>>>> cleaned
> >>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
> >>>>>>>>> chia7712@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> JR34: For these two configurations, setting an empty
> list
> >>>>>>> feels
> >>>>>>>>> a
> >>>>>>>>>>>>> bit
> >>>>>>>>>>>>>>>>>>> unintuitive. If an empty list is
> >>>>>>>>>>>>>>>>>>> provided, the consumer will call the unsubscribe
> method,
> >>>>>> which
> >>>>>>>>>>>>> doesn't
> >>>>>>>>>>>>>>>>>>> seem appropriate given
> >>>>>>>>>>>>>>>>>>> the documentation states: "Topic-partitions assigned to
> >>>> this
> >>>>>>>>> task
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> replicate."
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> agreed. Additionally, the source code [0] shows that it
> is
> >>>>>> not
> >>>>>>>>>>>>> intended
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> generate empty tps for the source task
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> [0]
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello all,

I’ve started implementing this KIP, and there are a few configurations that need further discussion:

1. HeaderFrom fields
This configuration should allow duplicate values. Therefore, its default value 
should be set to NO_DEFAULT_VALUE, and its validator should be NonEmptyListValidator(). 
See the test case [1] for reference.

2. bootstrap.servers
If we set its default value to NO_DEFAULT_VALUE and use the validator ConfigDef.ValidList.anyNonDuplicateValues(false, false), 
initialization will fail because the default value does not pass the validator. I see two possible solutions:

Option 2.1: Set the default value to List.of("localhost:9092") (similar to WorkerConfig) and use the validator ConfigDef.ValidList.anyNonDuplicateValues(false, false). However, this feels a bit odd, as the default value 
should ideally be useful, and localhost:9092 is not meaningful for most scenarios.

Option 2.2: Keep the default value as NO_DEFAULT_VALUE and use NonEmptyListValidator(). In this case, 
we would need to move NonEmptyListValidator from the connect.transforms module to the client module, 
and make it a public API within ConfigDef. This would also align with how most bootstrap.servers configs are handled.

3. Remove Cast fields config change
This configuration has its own validator, so we should leave it unchanged.

[1] https://github.com/apache/kafka/blob/18045c6ac30921503deffbef1744bb365dc599fb/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/HeaderFromTest.java#L240

Best Regards,
Jiunn-Yang

> 黃竣陽 <s7...@gmail.com> 於 2025年8月6日 晚上11:59 寫道：
> 
> Hi Jun,
> 
> Thanks for the reply,
> 
> I have updated the KIP
> 
> Best Regards,
> Jiunn-Yang
> 
>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月6日 晚上11:34 寫道：
>> 
>> Hi, Jiunn-Yang,
>> 
>> Regarding bootstrap.servers in MirrorClientConfig, it's weird that it
>> defaults to an empty list while empty list is not allowed. It seems that
>> this should have no default value.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Wed, Aug 6, 2025 at 7:56 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>> 
>>> Thanks to Jiunn-Yang for this hard KIP.
>>> 
>>> It might not be perfect, but we can refine it during the PR review process
>>> :)
>>> 
>>> 
>>> 黃竣陽 <s7...@gmail.com> 於 2025年8月6日 週三 下午7:32寫道：
>>> 
>>>> Hello chia,
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> chia05: I have updated the KIP to change the default values of
>>>> task.assigned.groups
>>>> and task.assigned.partitions from null to NO_DEFAULT_VALUE.
>>>> 
>>>> chia06: I have also updated the KIP based on your feedback.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月6日 凌晨12:54 寫道：
>>>>> 
>>>>> hi Jiunn-Yang
>>>>> 
>>>>> chia05:
>>>>> 
>>>>> `task.assigned.groups` and `task.assigned.partitions` should not have
>>>>> default value, right?
>>>>> 
>>>>> chia06:
>>>>> 
>>>>> `new HashSet<>(values).size()` -> `Set.copyOf(values).size()`
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年8月5日 週二 下午11:39寫道：
>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply and your patience. The KIP looks good to me now.
>>>>>> 
>>>>>> Hi, Luke, Mickael,
>>>>>> 
>>>>>> Could you take a look at the KIP again and see if it makes sense to
>>> you?
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hello Jun,
>>>>>>> 
>>>>>>> Thanks for the reply,
>>>>>>> 
>>>>>>> JR46: I have clarified the behavior for null and empty values, so
>>> users
>>>>>>> can
>>>>>>> understand the reasoning behind disallowing empty lists.
>>>>>>> 
>>>>>>> JR48: I've updated the KIP accordingly.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
>>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> JR46. "Currently, if plugin.path is set to an empty list, no
>>>>>>> user-specified
>>>>>>>> plugin locations are processed, but plugins from the parent
>>>> classloader
>>>>>>> are
>>>>>>>> still loaded." Could you add why the current behavior is bad for the
>>>>>>> users?
>>>>>>>> 
>>>>>>>> JR48. If we want to preserve the current behavior, we should remove
>>>> the
>>>>>>>> following validation, right?
>>>>>>>>     if (validStrings.length == 0) {
>>>>>>>>         throw new IllegalArgumentException("Valid strings list
>>>>>> cannot
>>>>>>>> be empty for in validator");
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hello Jun,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply.
>>>>>>>>> 
>>>>>>>>> JR42, JR45, JR46, JR47: I have updated the KIP.
>>>>>>>>> 
>>>>>>>>> JR43: If bootstrap.servers is empty for these configurations,
>>>>>>>>> it will fail when creating the AdminClient. I have updated the KIP
>>> to
>>>>>>>>> reflect this behavior.
>>>>>>>>> 
>>>>>>>>> JR44: Thanks for pointing this out. Kafka will indeed fail at the
>>>>>>> storage
>>>>>>>>> format phase
>>>>>>>>> if the configuration is invalid. I have updated the KIP
>>> accordingly.
>>>>>>>>> 
>>>>>>>>> JR48: The current in() method allows empty lists by default, so I
>>>>>>> believe
>>>>>>>>> setting isEmptyAllowed to true
>>>>>>>>> better preserves the original behavior. Therefore, I prefer not to
>>>>>>> change
>>>>>>>>> this default.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
>>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply. A few more comments.
>>>>>>>>>> 
>>>>>>>>>> JR42. [6] [7] are not needed since currently an empty list already
>>>>>>> causes
>>>>>>>>>> an exception.
>>>>>>>>>> 
>>>>>>>>>> JR43. Could you document the current behavior when
>>> bootstrap.servers
>>>>>> is
>>>>>>>>>> empty in MirrorClientConfig and WorkerConfig?
>>>>>>>>>> 
>>>>>>>>>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty
>>>>>> and
>>>>>>>>> got
>>>>>>>>>> the following.
>>>>>>>>>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
>>>>>>>>>> config/server.properties
>>>>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>>>> requirement
>>>>>>>>>> failed: At least one log directory must be defined via log.dirs or
>>>>>>>>> log.dir.
>>>>>>>>>> 
>>>>>>>>>> bin/kafka-server-start.sh config/server.properties
>>>>>>>>>> [2025-08-01 11:31:20,097] INFO Registered
>>>>>>>>>> `kafka:type=kafka.Log4jController` MBean
>>>>>>>>>> (kafka.utils.Log4jControllerRegistration$)
>>>>>>>>>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal
>>> exception
>>>>>>>>>> (kafka.Kafka$)
>>>>>>>>>> java.lang.IllegalArgumentException: requirement failed: At least
>>> one
>>>>>>> log
>>>>>>>>>> directory must be defined via log.dirs or log.dir.
>>>>>>>>>> 
>>>>>>>>>> JR45. Could you group all configs for WorkerConfig together?
>>>>>>>>>> 
>>>>>>>>>> JR46. plugin.path: We have the following code. So, currently, it
>>>>>> seems
>>>>>>>>> that
>>>>>>>>>> an empty list is allowed for plugin.path and we treat it by adding
>>>>>> the
>>>>>>>>>> parent path of the classloader. It would be useful to justify why
>>>>>>>>>> disallowing empty is better for the users.
>>>>>>>>>> 
>>>>>>>>>> public static Set<PluginSource> pluginSources(Set<Path>
>>>>>>>>>> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
>>>>>>>>> factory)
>>>>>>>>>> {
>>>>>>>>>>    Set<PluginSource> pluginSources = new LinkedHashSet<>();
>>>>>>>>>>    for (Path pluginLocation : pluginLocations) {
>>>>>>>>>>        try {
>>>>>>>>>> 
>>> pluginSources.add(isolatedPluginSource(pluginLocation,
>>>>>>>>>> classLoader, factory));
>>>>>>>>>>        } catch (InvalidPathException | MalformedURLException e)
>>> {
>>>>>>>>>>            log.error("Invalid path in plugin path: {}.
>>> Ignoring.",
>>>>>>>>>> pluginLocation, e);
>>>>>>>>>>        } catch (IOException e) {
>>>>>>>>>>            log.error("Could not get listing for plugin path: {}.
>>>>>>>>>> Ignoring.", pluginLocation, e);
>>>>>>>>>>        }
>>>>>>>>>>    }
>>>>>>>>>> 
>>>>>>> pluginSources.add(classpathPluginSource(classLoader.getParent()));
>>>>>>>>>>    return pluginSources;
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> JR47. typo in "Should not allow setting validStrings is empty"
>>>>>>>>>> 
>>>>>>>>>> JR48. We should pass in isEmptyAllowed as false in the code below,
>>>>>>> right?
>>>>>>>>>> public static ValidList in(String... validStrings) {
>>>>>>>>>>    if (validStrings.length == 0) {
>>>>>>>>>>        throw new IllegalArgumentException("Valid strings list
>>>>>> cannot
>>>>>>>>>> be empty for in validator");
>>>>>>>>>>    }
>>>>>>>>>>    return new ValidList(List.of(validStrings), true, false);
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello Jun,
>>>>>>>>>>> 
>>>>>>>>>>> Sorry for the late reply,
>>>>>>>>>>> 
>>>>>>>>>>> JR40, JR41: I have updated the KIP based on your comments. Please
>>>>>>> take a
>>>>>>>>>>> look.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>> 
>>>>>>>>>>>> JR40. In the table, there are a few configs for which we now
>>>>>> disallow
>>>>>>>>>>> empty
>>>>>>>>>>>> lists. It would be useful to document the current behavior for
>>>> each
>>>>>>> of
>>>>>>>>>>> them
>>>>>>>>>>>> more accurately. It seems that they should all lead to some bad
>>>>>>>>> behavior.
>>>>>>>>>>>> For example, if log.dirs is empty, it seems that we can't create
>>>>>>>>> topics?
>>>>>>>>>>>> If group.coordinator.rebalance.protocols is empty, the group
>>>>>>> rebalance
>>>>>>>>>>>> protocol will fail?
>>>>>>>>>>>> 
>>>>>>>>>>>> JR41. It would be useful to clean up the "Compatibility,
>>>>>> Deprecation,
>>>>>>>>> and
>>>>>>>>>>>> Migration Plan" section a bit.
>>>>>>>>>>>> JR41.1 The migration plan has "Introduce a new properties
>>>>>>>>> isEmptyAllowed
>>>>>>>>>>> ,
>>>>>>>>>>>> isNullAllowed", which needs to be changed.
>>>>>>>>>>>> JR41.2 It started with "For users who configure an empty list",
>>>> but
>>>>>>> not
>>>>>>>>>>> all
>>>>>>>>>>>> the bullet points are about empty lists.
>>>>>>>>>>>> JR41.3 To me, it would be useful to summarize why the behavior
>>>>>>> changes
>>>>>>>>>>> are
>>>>>>>>>>>> justified. We made 3 types of changes. (1) Disallow null. This
>>> is
>>>>>> ok
>>>>>>>>>>> since
>>>>>>>>>>>> one can't really set a null value through a config file. (2)
>>>>>> Disallow
>>>>>>>>>>>> duplicates. This is ok since it's rare and confusing. (3)
>>> Disallow
>>>>>>>>> empty.
>>>>>>>>>>>> Ideally, we want to claim that for those cases, they all lead to
>>>>>> some
>>>>>>>>>>> other
>>>>>>>>>>>> errors or had behavior currently.
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these
>>> changes.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR38: I clarified the following configurations:
>>>>>>>>>>>>> early.start.listeners: This config is optional and allows both
>>>>>> null
>>>>>>>>> and
>>>>>>>>>>>>> empty lists.
>>>>>>>>>>>>> log.dirs and plugin.path: These are also optional, but only
>>> allow
>>>>>>>>>>>>> null—empty lists are not permitted.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR34. Sounds good.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR36. Could you document that the default value for
>>>>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have
>>>>>> been
>>>>>>>>>>>>> changed?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR38. My concern wasn't quite addressed. "If the configuration
>>>> is
>>>>>>>>>>>>> optional,
>>>>>>>>>>>>>> we will reject any duplicate values in the list." This implies
>>>>>> that
>>>>>>>>> if
>>>>>>>>>>> a
>>>>>>>>>>>>>> config is optional, we allow empty lists. However, for
>>>>>> plugin.path,
>>>>>>>>> we
>>>>>>>>>>>>>> reject empty lists.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR39. We can remove the fields isEmptyAllowed and
>>> isNullAllowed
>>>>>>> since
>>>>>>>>>>>>> they
>>>>>>>>>>>>>> are not public facing.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Jun, Chia,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the reply,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR36: I’ve updated the following configurations:
>>>>>>>>>>>>>>> - sasl.oauthbearer.expected.audience
>>>>>>>>>>>>>>> - ssl.cipher.suites
>>>>>>>>>>>>>>> - ssl.enabled.protocols
>>>>>>>>>>>>>>> These now allow both empty and non-empty lists. Since the
>>>>>> default
>>>>>>>>>>> values
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
>>>>>>>>>>> previously
>>>>>>>>>>>>>>> null,
>>>>>>>>>>>>>>> I’ve updated their defaults to List.of() for consistency.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR37: A ClassCastException does not occur in
>>> MirrorClientConfig
>>>>>>> and
>>>>>>>>> it
>>>>>>>>>>>>>>> should be none.
>>>>>>>>>>>>>>> I’ve updated this in the KIP.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate
>>>> which
>>>>>>>>>>>>>>> configurations require special attention.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR39: Yes, these properties are package-private.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
>>>> return
>>>>>>> an
>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>>>>>>>>> maxTasks,
>>>>>>>>>>>>>>>> Chia-Ping?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an
>>> empty
>>>>>>>>> list,
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>> means no `MirrorSourceTask` can be created.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the
>>> configs
>>>>>>> used
>>>>>>>>>>> by
>>>>>>>>>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions`
>>>>>> with a
>>>>>>>>>>>>>>> non-empty
>>>>>>>>>>>>>>>> value.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二
>>> 上午5:14寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
>>>>>> return
>>>>>>> an
>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>>>>>>>>> maxTasks,
>>>>>>>>>>>>>>>>> Chia-Ping?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should
>>> we
>>>>>> do
>>>>>>>>> the
>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>> for ssl.cipher.suites and
>>> sasl.oauthbearer.expected.audience?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers
>>>>>>> returns
>>>>>>>>>>>>>>>>> ClassCastException for null? It seems that null could be
>>>>>> casted
>>>>>>> to
>>>>>>>>>>> any
>>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR38. "If the configuration is optional, we will reject any
>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>>> values in the list." Could you clarify this? For example,
>>>>>>>>>>> plugin.path
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> optional, but with a default value. We reject duplicates as
>>>>>> well
>>>>>>>>> as
>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>> lists.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public
>>> fields,
>>>>>>>>> right?
>>>>>>>>>>>>>>> They
>>>>>>>>>>>>>>>>> are only public in the anyNonDuplicateValues() method.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR40. It would be useful to mention that if cleanup.policy
>>> is
>>>>>>>>> empty
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> remote.storage.enable is true, the local log segments will
>>> be
>>>>>>>>>>> cleaned
>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
>>>>>>>>> chia7712@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR34: For these two configurations, setting an empty list
>>>>>>> feels
>>>>>>>>> a
>>>>>>>>>>>>> bit
>>>>>>>>>>>>>>>>>>> unintuitive. If an empty list is
>>>>>>>>>>>>>>>>>>> provided, the consumer will call the unsubscribe method,
>>>>>> which
>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>>> seem appropriate given
>>>>>>>>>>>>>>>>>>> the documentation states: "Topic-partitions assigned to
>>>> this
>>>>>>>>> task
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> replicate."
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> agreed. Additionally, the source code [0] shows that it is
>>>>>> not
>>>>>>>>>>>>> intended
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> generate empty tps for the source task
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi Jun,

Thanks for the reply,

I have updated the KIP

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月6日 晚上11:34 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Regarding bootstrap.servers in MirrorClientConfig, it's weird that it
> defaults to an empty list while empty list is not allowed. It seems that
> this should have no default value.
> 
> Thanks,
> 
> Jun
> 
> On Wed, Aug 6, 2025 at 7:56 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
> 
>> Thanks to Jiunn-Yang for this hard KIP.
>> 
>> It might not be perfect, but we can refine it during the PR review process
>> :)
>> 
>> 
>> 黃竣陽 <s7...@gmail.com> 於 2025年8月6日 週三 下午7:32寫道：
>> 
>>> Hello chia,
>>> 
>>> Thanks for the reply.
>>> 
>>> chia05: I have updated the KIP to change the default values of
>>> task.assigned.groups
>>> and task.assigned.partitions from null to NO_DEFAULT_VALUE.
>>> 
>>> chia06: I have also updated the KIP based on your feedback.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月6日 凌晨12:54 寫道：
>>>> 
>>>> hi Jiunn-Yang
>>>> 
>>>> chia05:
>>>> 
>>>> `task.assigned.groups` and `task.assigned.partitions` should not have
>>>> default value, right?
>>>> 
>>>> chia06:
>>>> 
>>>> `new HashSet<>(values).size()` -> `Set.copyOf(values).size()`
>>>> 
>>>> Best,
>>>> Chia-Ping
>>>> 
>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年8月5日 週二 下午11:39寫道：
>>>> 
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> Thanks for the reply and your patience. The KIP looks good to me now.
>>>>> 
>>>>> Hi, Luke, Mickael,
>>>>> 
>>>>> Could you take a look at the KIP again and see if it makes sense to
>> you?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hello Jun,
>>>>>> 
>>>>>> Thanks for the reply,
>>>>>> 
>>>>>> JR46: I have clarified the behavior for null and empty values, so
>> users
>>>>>> can
>>>>>> understand the reasoning behind disallowing empty lists.
>>>>>> 
>>>>>> JR48: I've updated the KIP accordingly.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
>>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the reply.
>>>>>>> 
>>>>>>> JR46. "Currently, if plugin.path is set to an empty list, no
>>>>>> user-specified
>>>>>>> plugin locations are processed, but plugins from the parent
>>> classloader
>>>>>> are
>>>>>>> still loaded." Could you add why the current behavior is bad for the
>>>>>> users?
>>>>>>> 
>>>>>>> JR48. If we want to preserve the current behavior, we should remove
>>> the
>>>>>>> following validation, right?
>>>>>>>      if (validStrings.length == 0) {
>>>>>>>          throw new IllegalArgumentException("Valid strings list
>>>>> cannot
>>>>>>> be empty for in validator");
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hello Jun,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> JR42, JR45, JR46, JR47: I have updated the KIP.
>>>>>>>> 
>>>>>>>> JR43: If bootstrap.servers is empty for these configurations,
>>>>>>>> it will fail when creating the AdminClient. I have updated the KIP
>> to
>>>>>>>> reflect this behavior.
>>>>>>>> 
>>>>>>>> JR44: Thanks for pointing this out. Kafka will indeed fail at the
>>>>>> storage
>>>>>>>> format phase
>>>>>>>> if the configuration is invalid. I have updated the KIP
>> accordingly.
>>>>>>>> 
>>>>>>>> JR48: The current in() method allows empty lists by default, so I
>>>>>> believe
>>>>>>>> setting isEmptyAllowed to true
>>>>>>>> better preserves the original behavior. Therefore, I prefer not to
>>>>>> change
>>>>>>>> this default.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
>>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply. A few more comments.
>>>>>>>>> 
>>>>>>>>> JR42. [6] [7] are not needed since currently an empty list already
>>>>>> causes
>>>>>>>>> an exception.
>>>>>>>>> 
>>>>>>>>> JR43. Could you document the current behavior when
>> bootstrap.servers
>>>>> is
>>>>>>>>> empty in MirrorClientConfig and WorkerConfig?
>>>>>>>>> 
>>>>>>>>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty
>>>>> and
>>>>>>>> got
>>>>>>>>> the following.
>>>>>>>>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
>>>>>>>>> config/server.properties
>>>>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>>>>> requirement
>>>>>>>>> failed: At least one log directory must be defined via log.dirs or
>>>>>>>> log.dir.
>>>>>>>>> 
>>>>>>>>> bin/kafka-server-start.sh config/server.properties
>>>>>>>>> [2025-08-01 11:31:20,097] INFO Registered
>>>>>>>>> `kafka:type=kafka.Log4jController` MBean
>>>>>>>>> (kafka.utils.Log4jControllerRegistration$)
>>>>>>>>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal
>> exception
>>>>>>>>> (kafka.Kafka$)
>>>>>>>>> java.lang.IllegalArgumentException: requirement failed: At least
>> one
>>>>>> log
>>>>>>>>> directory must be defined via log.dirs or log.dir.
>>>>>>>>> 
>>>>>>>>> JR45. Could you group all configs for WorkerConfig together?
>>>>>>>>> 
>>>>>>>>> JR46. plugin.path: We have the following code. So, currently, it
>>>>> seems
>>>>>>>> that
>>>>>>>>> an empty list is allowed for plugin.path and we treat it by adding
>>>>> the
>>>>>>>>> parent path of the classloader. It would be useful to justify why
>>>>>>>>> disallowing empty is better for the users.
>>>>>>>>> 
>>>>>>>>> public static Set<PluginSource> pluginSources(Set<Path>
>>>>>>>>> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
>>>>>>>> factory)
>>>>>>>>> {
>>>>>>>>>     Set<PluginSource> pluginSources = new LinkedHashSet<>();
>>>>>>>>>     for (Path pluginLocation : pluginLocations) {
>>>>>>>>>         try {
>>>>>>>>> 
>> pluginSources.add(isolatedPluginSource(pluginLocation,
>>>>>>>>> classLoader, factory));
>>>>>>>>>         } catch (InvalidPathException | MalformedURLException e)
>> {
>>>>>>>>>             log.error("Invalid path in plugin path: {}.
>> Ignoring.",
>>>>>>>>> pluginLocation, e);
>>>>>>>>>         } catch (IOException e) {
>>>>>>>>>             log.error("Could not get listing for plugin path: {}.
>>>>>>>>> Ignoring.", pluginLocation, e);
>>>>>>>>>         }
>>>>>>>>>     }
>>>>>>>>> 
>>>>>> pluginSources.add(classpathPluginSource(classLoader.getParent()));
>>>>>>>>>     return pluginSources;
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> JR47. typo in "Should not allow setting validStrings is empty"
>>>>>>>>> 
>>>>>>>>> JR48. We should pass in isEmptyAllowed as false in the code below,
>>>>>> right?
>>>>>>>>> public static ValidList in(String... validStrings) {
>>>>>>>>>     if (validStrings.length == 0) {
>>>>>>>>>         throw new IllegalArgumentException("Valid strings list
>>>>> cannot
>>>>>>>>> be empty for in validator");
>>>>>>>>>     }
>>>>>>>>>     return new ValidList(List.of(validStrings), true, false);
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hello Jun,
>>>>>>>>>> 
>>>>>>>>>> Sorry for the late reply,
>>>>>>>>>> 
>>>>>>>>>> JR40, JR41: I have updated the KIP based on your comments. Please
>>>>>> take a
>>>>>>>>>> look.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>> 
>>>>>>>>>>> JR40. In the table, there are a few configs for which we now
>>>>> disallow
>>>>>>>>>> empty
>>>>>>>>>>> lists. It would be useful to document the current behavior for
>>> each
>>>>>> of
>>>>>>>>>> them
>>>>>>>>>>> more accurately. It seems that they should all lead to some bad
>>>>>>>> behavior.
>>>>>>>>>>> For example, if log.dirs is empty, it seems that we can't create
>>>>>>>> topics?
>>>>>>>>>>> If group.coordinator.rebalance.protocols is empty, the group
>>>>>> rebalance
>>>>>>>>>>> protocol will fail?
>>>>>>>>>>> 
>>>>>>>>>>> JR41. It would be useful to clean up the "Compatibility,
>>>>> Deprecation,
>>>>>>>> and
>>>>>>>>>>> Migration Plan" section a bit.
>>>>>>>>>>> JR41.1 The migration plan has "Introduce a new properties
>>>>>>>> isEmptyAllowed
>>>>>>>>>> ,
>>>>>>>>>>> isNullAllowed", which needs to be changed.
>>>>>>>>>>> JR41.2 It started with "For users who configure an empty list",
>>> but
>>>>>> not
>>>>>>>>>> all
>>>>>>>>>>> the bullet points are about empty lists.
>>>>>>>>>>> JR41.3 To me, it would be useful to summarize why the behavior
>>>>>> changes
>>>>>>>>>> are
>>>>>>>>>>> justified. We made 3 types of changes. (1) Disallow null. This
>> is
>>>>> ok
>>>>>>>>>> since
>>>>>>>>>>> one can't really set a null value through a config file. (2)
>>>>> Disallow
>>>>>>>>>>> duplicates. This is ok since it's rare and confusing. (3)
>> Disallow
>>>>>>>> empty.
>>>>>>>>>>> Ideally, we want to claim that for those cases, they all lead to
>>>>> some
>>>>>>>>>> other
>>>>>>>>>>> errors or had behavior currently.
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>> 
>>>>>>>>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these
>> changes.
>>>>>>>>>>>> 
>>>>>>>>>>>> JR38: I clarified the following configurations:
>>>>>>>>>>>> early.start.listeners: This config is optional and allows both
>>>>> null
>>>>>>>> and
>>>>>>>>>>>> empty lists.
>>>>>>>>>>>> log.dirs and plugin.path: These are also optional, but only
>> allow
>>>>>>>>>>>> null—empty lists are not permitted.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR34. Sounds good.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR36. Could you document that the default value for
>>>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have
>>>>> been
>>>>>>>>>>>> changed?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR38. My concern wasn't quite addressed. "If the configuration
>>> is
>>>>>>>>>>>> optional,
>>>>>>>>>>>>> we will reject any duplicate values in the list." This implies
>>>>> that
>>>>>>>> if
>>>>>>>>>> a
>>>>>>>>>>>>> config is optional, we allow empty lists. However, for
>>>>> plugin.path,
>>>>>>>> we
>>>>>>>>>>>>> reject empty lists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR39. We can remove the fields isEmptyAllowed and
>> isNullAllowed
>>>>>> since
>>>>>>>>>>>> they
>>>>>>>>>>>>> are not public facing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello Jun, Chia,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the reply,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR36: I’ve updated the following configurations:
>>>>>>>>>>>>>> - sasl.oauthbearer.expected.audience
>>>>>>>>>>>>>> - ssl.cipher.suites
>>>>>>>>>>>>>> - ssl.enabled.protocols
>>>>>>>>>>>>>> These now allow both empty and non-empty lists. Since the
>>>>> default
>>>>>>>>>> values
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
>>>>>>>>>> previously
>>>>>>>>>>>>>> null,
>>>>>>>>>>>>>> I’ve updated their defaults to List.of() for consistency.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR37: A ClassCastException does not occur in
>> MirrorClientConfig
>>>>>> and
>>>>>>>> it
>>>>>>>>>>>>>> should be none.
>>>>>>>>>>>>>> I’ve updated this in the KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate
>>> which
>>>>>>>>>>>>>> configurations require special attention.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR39: Yes, these properties are package-private.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
>>> return
>>>>>> an
>>>>>>>>>>>> empty
>>>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>>>>>>>> maxTasks,
>>>>>>>>>>>>>>> Chia-Ping?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an
>> empty
>>>>>>>> list,
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>> means no `MirrorSourceTask` can be created.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the
>> configs
>>>>>> used
>>>>>>>>>> by
>>>>>>>>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions`
>>>>> with a
>>>>>>>>>>>>>> non-empty
>>>>>>>>>>>>>>> value.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二
>> 上午5:14寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
>>>>> return
>>>>>> an
>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>>>>>>>> maxTasks,
>>>>>>>>>>>>>>>> Chia-Ping?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should
>> we
>>>>> do
>>>>>>>> the
>>>>>>>>>>>> same
>>>>>>>>>>>>>>>> for ssl.cipher.suites and
>> sasl.oauthbearer.expected.audience?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers
>>>>>> returns
>>>>>>>>>>>>>>>> ClassCastException for null? It seems that null could be
>>>>> casted
>>>>>> to
>>>>>>>>>> any
>>>>>>>>>>>>>>>> class.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR38. "If the configuration is optional, we will reject any
>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>> values in the list." Could you clarify this? For example,
>>>>>>>>>> plugin.path
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> optional, but with a default value. We reject duplicates as
>>>>> well
>>>>>>>> as
>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>> lists.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public
>> fields,
>>>>>>>> right?
>>>>>>>>>>>>>> They
>>>>>>>>>>>>>>>> are only public in the anyNonDuplicateValues() method.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR40. It would be useful to mention that if cleanup.policy
>> is
>>>>>>>> empty
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> remote.storage.enable is true, the local log segments will
>> be
>>>>>>>>>> cleaned
>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
>>>>>>>> chia7712@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> JR34: For these two configurations, setting an empty list
>>>>>> feels
>>>>>>>> a
>>>>>>>>>>>> bit
>>>>>>>>>>>>>>>>>> unintuitive. If an empty list is
>>>>>>>>>>>>>>>>>> provided, the consumer will call the unsubscribe method,
>>>>> which
>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>> seem appropriate given
>>>>>>>>>>>>>>>>>> the documentation states: "Topic-partitions assigned to
>>> this
>>>>>>>> task
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> replicate."
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> agreed. Additionally, the source code [0] shows that it is
>>>>> not
>>>>>>>>>>>> intended
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> generate empty tps for the source task
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Regarding bootstrap.servers in MirrorClientConfig, it's weird that it
defaults to an empty list while empty list is not allowed. It seems that
this should have no default value.

Thanks,

Jun

On Wed, Aug 6, 2025 at 7:56 AM Chia-Ping Tsai <ch...@gmail.com> wrote:

> Thanks to Jiunn-Yang for this hard KIP.
>
> It might not be perfect, but we can refine it during the PR review process
> :)
>
>
> 黃竣陽 <s7...@gmail.com> 於 2025年8月6日 週三 下午7:32寫道：
>
> > Hello chia,
> >
> > Thanks for the reply.
> >
> > chia05: I have updated the KIP to change the default values of
> > task.assigned.groups
> > and task.assigned.partitions from null to NO_DEFAULT_VALUE.
> >
> > chia06: I have also updated the KIP based on your feedback.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月6日 凌晨12:54 寫道：
> > >
> > > hi Jiunn-Yang
> > >
> > > chia05:
> > >
> > > `task.assigned.groups` and `task.assigned.partitions` should not have
> > > default value, right?
> > >
> > > chia06:
> > >
> > > `new HashSet<>(values).size()` -> `Set.copyOf(values).size()`
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > Jun Rao <ju...@confluent.io.invalid> 於 2025年8月5日 週二 下午11:39寫道：
> > >
> > >> Hi, Jiunn-Yang,
> > >>
> > >> Thanks for the reply and your patience. The KIP looks good to me now.
> > >>
> > >> Hi, Luke, Mickael,
> > >>
> > >> Could you take a look at the KIP again and see if it makes sense to
> you?
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>
> > >>> Hello Jun,
> > >>>
> > >>> Thanks for the reply,
> > >>>
> > >>> JR46: I have clarified the behavior for null and empty values, so
> users
> > >>> can
> > >>> understand the reasoning behind disallowing empty lists.
> > >>>
> > >>> JR48: I've updated the KIP accordingly.
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
> > >>>>
> > >>>> Hi, Jiunn-Yang,
> > >>>>
> > >>>> Thanks for the reply.
> > >>>>
> > >>>> JR46. "Currently, if plugin.path is set to an empty list, no
> > >>> user-specified
> > >>>> plugin locations are processed, but plugins from the parent
> > classloader
> > >>> are
> > >>>> still loaded." Could you add why the current behavior is bad for the
> > >>> users?
> > >>>>
> > >>>> JR48. If we want to preserve the current behavior, we should remove
> > the
> > >>>> following validation, right?
> > >>>>       if (validStrings.length == 0) {
> > >>>>           throw new IllegalArgumentException("Valid strings list
> > >> cannot
> > >>>> be empty for in validator");
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>> On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>
> > >>>>> Hello Jun,
> > >>>>>
> > >>>>> Thanks for the reply.
> > >>>>>
> > >>>>> JR42, JR45, JR46, JR47: I have updated the KIP.
> > >>>>>
> > >>>>> JR43: If bootstrap.servers is empty for these configurations,
> > >>>>> it will fail when creating the AdminClient. I have updated the KIP
> to
> > >>>>> reflect this behavior.
> > >>>>>
> > >>>>> JR44: Thanks for pointing this out. Kafka will indeed fail at the
> > >>> storage
> > >>>>> format phase
> > >>>>> if the configuration is invalid. I have updated the KIP
> accordingly.
> > >>>>>
> > >>>>> JR48: The current in() method allows empty lists by default, so I
> > >>> believe
> > >>>>> setting isEmptyAllowed to true
> > >>>>> better preserves the original behavior. Therefore, I prefer not to
> > >>> change
> > >>>>> this default.
> > >>>>>
> > >>>>> Best Regards,
> > >>>>> Jiunn-Yang
> > >>>>>
> > >>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
> > >>>>>>
> > >>>>>> Hi, Jiunn-Yang,
> > >>>>>>
> > >>>>>> Thanks for the reply. A few more comments.
> > >>>>>>
> > >>>>>> JR42. [6] [7] are not needed since currently an empty list already
> > >>> causes
> > >>>>>> an exception.
> > >>>>>>
> > >>>>>> JR43. Could you document the current behavior when
> bootstrap.servers
> > >> is
> > >>>>>> empty in MirrorClientConfig and WorkerConfig?
> > >>>>>>
> > >>>>>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty
> > >> and
> > >>>>> got
> > >>>>>> the following.
> > >>>>>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
> > >>>>>> config/server.properties
> > >>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
> > >>>>> requirement
> > >>>>>> failed: At least one log directory must be defined via log.dirs or
> > >>>>> log.dir.
> > >>>>>>
> > >>>>>> bin/kafka-server-start.sh config/server.properties
> > >>>>>> [2025-08-01 11:31:20,097] INFO Registered
> > >>>>>> `kafka:type=kafka.Log4jController` MBean
> > >>>>>> (kafka.utils.Log4jControllerRegistration$)
> > >>>>>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal
> exception
> > >>>>>> (kafka.Kafka$)
> > >>>>>> java.lang.IllegalArgumentException: requirement failed: At least
> one
> > >>> log
> > >>>>>> directory must be defined via log.dirs or log.dir.
> > >>>>>>
> > >>>>>> JR45. Could you group all configs for WorkerConfig together?
> > >>>>>>
> > >>>>>> JR46. plugin.path: We have the following code. So, currently, it
> > >> seems
> > >>>>> that
> > >>>>>> an empty list is allowed for plugin.path and we treat it by adding
> > >> the
> > >>>>>> parent path of the classloader. It would be useful to justify why
> > >>>>>> disallowing empty is better for the users.
> > >>>>>>
> > >>>>>>  public static Set<PluginSource> pluginSources(Set<Path>
> > >>>>>> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
> > >>>>> factory)
> > >>>>>> {
> > >>>>>>      Set<PluginSource> pluginSources = new LinkedHashSet<>();
> > >>>>>>      for (Path pluginLocation : pluginLocations) {
> > >>>>>>          try {
> > >>>>>>
> pluginSources.add(isolatedPluginSource(pluginLocation,
> > >>>>>> classLoader, factory));
> > >>>>>>          } catch (InvalidPathException | MalformedURLException e)
> {
> > >>>>>>              log.error("Invalid path in plugin path: {}.
> Ignoring.",
> > >>>>>> pluginLocation, e);
> > >>>>>>          } catch (IOException e) {
> > >>>>>>              log.error("Could not get listing for plugin path: {}.
> > >>>>>> Ignoring.", pluginLocation, e);
> > >>>>>>          }
> > >>>>>>      }
> > >>>>>>
> > >>> pluginSources.add(classpathPluginSource(classLoader.getParent()));
> > >>>>>>      return pluginSources;
> > >>>>>>  }
> > >>>>>>
> > >>>>>> JR47. typo in "Should not allow setting validStrings is empty"
> > >>>>>>
> > >>>>>> JR48. We should pass in isEmptyAllowed as false in the code below,
> > >>> right?
> > >>>>>>  public static ValidList in(String... validStrings) {
> > >>>>>>      if (validStrings.length == 0) {
> > >>>>>>          throw new IllegalArgumentException("Valid strings list
> > >> cannot
> > >>>>>> be empty for in validator");
> > >>>>>>      }
> > >>>>>>      return new ValidList(List.of(validStrings), true, false);
> > >>>>>>  }
> > >>>>>>
> > >>>>>> Jun
> > >>>>>>
> > >>>>>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> Hello Jun,
> > >>>>>>>
> > >>>>>>> Sorry for the late reply,
> > >>>>>>>
> > >>>>>>> JR40, JR41: I have updated the KIP based on your comments. Please
> > >>> take a
> > >>>>>>> look.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>>>
> > >>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
> > >>>>>>>>
> > >>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>
> > >>>>>>>> Thanks for the reply.
> > >>>>>>>>
> > >>>>>>>> JR40. In the table, there are a few configs for which we now
> > >> disallow
> > >>>>>>> empty
> > >>>>>>>> lists. It would be useful to document the current behavior for
> > each
> > >>> of
> > >>>>>>> them
> > >>>>>>>> more accurately. It seems that they should all lead to some bad
> > >>>>> behavior.
> > >>>>>>>> For example, if log.dirs is empty, it seems that we can't create
> > >>>>> topics?
> > >>>>>>>> If group.coordinator.rebalance.protocols is empty, the group
> > >>> rebalance
> > >>>>>>>> protocol will fail?
> > >>>>>>>>
> > >>>>>>>> JR41. It would be useful to clean up the "Compatibility,
> > >> Deprecation,
> > >>>>> and
> > >>>>>>>> Migration Plan" section a bit.
> > >>>>>>>> JR41.1 The migration plan has "Introduce a new properties
> > >>>>> isEmptyAllowed
> > >>>>>>> ,
> > >>>>>>>> isNullAllowed", which needs to be changed.
> > >>>>>>>> JR41.2 It started with "For users who configure an empty list",
> > but
> > >>> not
> > >>>>>>> all
> > >>>>>>>> the bullet points are about empty lists.
> > >>>>>>>> JR41.3 To me, it would be useful to summarize why the behavior
> > >>> changes
> > >>>>>>> are
> > >>>>>>>> justified. We made 3 types of changes. (1) Disallow null. This
> is
> > >> ok
> > >>>>>>> since
> > >>>>>>>> one can't really set a null value through a config file. (2)
> > >> Disallow
> > >>>>>>>> duplicates. This is ok since it's rare and confusing. (3)
> Disallow
> > >>>>> empty.
> > >>>>>>>> Ideally, we want to claim that for those cases, they all lead to
> > >> some
> > >>>>>>> other
> > >>>>>>>> errors or had behavior currently.
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hello Jun,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for the reply.
> > >>>>>>>>>
> > >>>>>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these
> changes.
> > >>>>>>>>>
> > >>>>>>>>> JR38: I clarified the following configurations:
> > >>>>>>>>> early.start.listeners: This config is optional and allows both
> > >> null
> > >>>>> and
> > >>>>>>>>> empty lists.
> > >>>>>>>>> log.dirs and plugin.path: These are also optional, but only
> allow
> > >>>>>>>>> null—empty lists are not permitted.
> > >>>>>>>>>
> > >>>>>>>>> Best Regards,
> > >>>>>>>>> Jiunn-Yang
> > >>>>>>>>>
> > >>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> > >>>>>>>>>>
> > >>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>
> > >>>>>>>>>> JR34. Sounds good.
> > >>>>>>>>>>
> > >>>>>>>>>> JR36. Could you document that the default value for
> > >>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have
> > >> been
> > >>>>>>>>> changed?
> > >>>>>>>>>>
> > >>>>>>>>>> JR38. My concern wasn't quite addressed. "If the configuration
> > is
> > >>>>>>>>> optional,
> > >>>>>>>>>> we will reject any duplicate values in the list." This implies
> > >> that
> > >>>>> if
> > >>>>>>> a
> > >>>>>>>>>> config is optional, we allow empty lists. However, for
> > >> plugin.path,
> > >>>>> we
> > >>>>>>>>>> reject empty lists.
> > >>>>>>>>>>
> > >>>>>>>>>> JR39. We can remove the fields isEmptyAllowed and
> isNullAllowed
> > >>> since
> > >>>>>>>>> they
> > >>>>>>>>>> are not public facing.
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hello Jun, Chia,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the reply,
> > >>>>>>>>>>>
> > >>>>>>>>>>> JR36: I’ve updated the following configurations:
> > >>>>>>>>>>> - sasl.oauthbearer.expected.audience
> > >>>>>>>>>>> - ssl.cipher.suites
> > >>>>>>>>>>> - ssl.enabled.protocols
> > >>>>>>>>>>> These now allow both empty and non-empty lists. Since the
> > >> default
> > >>>>>>> values
> > >>>>>>>>>>> of
> > >>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
> > >>>>>>> previously
> > >>>>>>>>>>> null,
> > >>>>>>>>>>> I’ve updated their defaults to List.of() for consistency.
> > >>>>>>>>>>>
> > >>>>>>>>>>> JR37: A ClassCastException does not occur in
> MirrorClientConfig
> > >>> and
> > >>>>> it
> > >>>>>>>>>>> should be none.
> > >>>>>>>>>>> I’ve updated this in the KIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate
> > which
> > >>>>>>>>>>> configurations require special attention.
> > >>>>>>>>>>>
> > >>>>>>>>>>> JR39: Yes, these properties are package-private.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best Regards,
> > >>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
> > return
> > >>> an
> > >>>>>>>>> empty
> > >>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> > >>>>> maxTasks,
> > >>>>>>>>>>>> Chia-Ping?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an
> empty
> > >>>>> list,
> > >>>>>>>>>>> which
> > >>>>>>>>>>>> means no `MirrorSourceTask` can be created.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the
> configs
> > >>> used
> > >>>>>>> by
> > >>>>>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions`
> > >> with a
> > >>>>>>>>>>> non-empty
> > >>>>>>>>>>>> value.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二
> 上午5:14寫道：
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
> > >> return
> > >>> an
> > >>>>>>>>> empty
> > >>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> > >>>>> maxTasks,
> > >>>>>>>>>>>>> Chia-Ping?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should
> we
> > >> do
> > >>>>> the
> > >>>>>>>>> same
> > >>>>>>>>>>>>> for ssl.cipher.suites and
> sasl.oauthbearer.expected.audience?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers
> > >>> returns
> > >>>>>>>>>>>>> ClassCastException for null? It seems that null could be
> > >> casted
> > >>> to
> > >>>>>>> any
> > >>>>>>>>>>>>> class.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR38. "If the configuration is optional, we will reject any
> > >>>>>>> duplicate
> > >>>>>>>>>>>>> values in the list." Could you clarify this? For example,
> > >>>>>>> plugin.path
> > >>>>>>>>> is
> > >>>>>>>>>>>>> optional, but with a default value. We reject duplicates as
> > >> well
> > >>>>> as
> > >>>>>>>>>>> empty
> > >>>>>>>>>>>>> lists.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public
> fields,
> > >>>>> right?
> > >>>>>>>>>>> They
> > >>>>>>>>>>>>> are only public in the anyNonDuplicateValues() method.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR40. It would be useful to mention that if cleanup.policy
> is
> > >>>>> empty
> > >>>>>>>>> and
> > >>>>>>>>>>>>> remote.storage.enable is true, the local log segments will
> be
> > >>>>>>> cleaned
> > >>>>>>>>>>> based
> > >>>>>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
> > >>>>> chia7712@gmail.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> JR34: For these two configurations, setting an empty list
> > >>> feels
> > >>>>> a
> > >>>>>>>>> bit
> > >>>>>>>>>>>>>>> unintuitive. If an empty list is
> > >>>>>>>>>>>>>>> provided, the consumer will call the unsubscribe method,
> > >> which
> > >>>>>>>>> doesn't
> > >>>>>>>>>>>>>>> seem appropriate given
> > >>>>>>>>>>>>>>> the documentation states: "Topic-partitions assigned to
> > this
> > >>>>> task
> > >>>>>>> to
> > >>>>>>>>>>>>>>> replicate."
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> agreed. Additionally, the source code [0] shows that it is
> > >> not
> > >>>>>>>>> intended
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>> generate empty tps for the source task
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> [0]
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

Thanks to Jiunn-Yang for this hard KIP.

It might not be perfect, but we can refine it during the PR review process
:)


黃竣陽 <s7...@gmail.com> 於 2025年8月6日 週三 下午7:32寫道：

> Hello chia,
>
> Thanks for the reply.
>
> chia05: I have updated the KIP to change the default values of
> task.assigned.groups
> and task.assigned.partitions from null to NO_DEFAULT_VALUE.
>
> chia06: I have also updated the KIP based on your feedback.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月6日 凌晨12:54 寫道：
> >
> > hi Jiunn-Yang
> >
> > chia05:
> >
> > `task.assigned.groups` and `task.assigned.partitions` should not have
> > default value, right?
> >
> > chia06:
> >
> > `new HashSet<>(values).size()` -> `Set.copyOf(values).size()`
> >
> > Best,
> > Chia-Ping
> >
> > Jun Rao <ju...@confluent.io.invalid> 於 2025年8月5日 週二 下午11:39寫道：
> >
> >> Hi, Jiunn-Yang,
> >>
> >> Thanks for the reply and your patience. The KIP looks good to me now.
> >>
> >> Hi, Luke, Mickael,
> >>
> >> Could you take a look at the KIP again and see if it makes sense to you?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hello Jun,
> >>>
> >>> Thanks for the reply,
> >>>
> >>> JR46: I have clarified the behavior for null and empty values, so users
> >>> can
> >>> understand the reasoning behind disallowing empty lists.
> >>>
> >>> JR48: I've updated the KIP accordingly.
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
> >>>>
> >>>> Hi, Jiunn-Yang,
> >>>>
> >>>> Thanks for the reply.
> >>>>
> >>>> JR46. "Currently, if plugin.path is set to an empty list, no
> >>> user-specified
> >>>> plugin locations are processed, but plugins from the parent
> classloader
> >>> are
> >>>> still loaded." Could you add why the current behavior is bad for the
> >>> users?
> >>>>
> >>>> JR48. If we want to preserve the current behavior, we should remove
> the
> >>>> following validation, right?
> >>>>       if (validStrings.length == 0) {
> >>>>           throw new IllegalArgumentException("Valid strings list
> >> cannot
> >>>> be empty for in validator");
> >>>>
> >>>> Jun
> >>>>
> >>>> On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>
> >>>>> Hello Jun,
> >>>>>
> >>>>> Thanks for the reply.
> >>>>>
> >>>>> JR42, JR45, JR46, JR47: I have updated the KIP.
> >>>>>
> >>>>> JR43: If bootstrap.servers is empty for these configurations,
> >>>>> it will fail when creating the AdminClient. I have updated the KIP to
> >>>>> reflect this behavior.
> >>>>>
> >>>>> JR44: Thanks for pointing this out. Kafka will indeed fail at the
> >>> storage
> >>>>> format phase
> >>>>> if the configuration is invalid. I have updated the KIP accordingly.
> >>>>>
> >>>>> JR48: The current in() method allows empty lists by default, so I
> >>> believe
> >>>>> setting isEmptyAllowed to true
> >>>>> better preserves the original behavior. Therefore, I prefer not to
> >>> change
> >>>>> this default.
> >>>>>
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
> >>>>>>
> >>>>>> Hi, Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the reply. A few more comments.
> >>>>>>
> >>>>>> JR42. [6] [7] are not needed since currently an empty list already
> >>> causes
> >>>>>> an exception.
> >>>>>>
> >>>>>> JR43. Could you document the current behavior when bootstrap.servers
> >> is
> >>>>>> empty in MirrorClientConfig and WorkerConfig?
> >>>>>>
> >>>>>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty
> >> and
> >>>>> got
> >>>>>> the following.
> >>>>>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
> >>>>>> config/server.properties
> >>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
> >>>>> requirement
> >>>>>> failed: At least one log directory must be defined via log.dirs or
> >>>>> log.dir.
> >>>>>>
> >>>>>> bin/kafka-server-start.sh config/server.properties
> >>>>>> [2025-08-01 11:31:20,097] INFO Registered
> >>>>>> `kafka:type=kafka.Log4jController` MBean
> >>>>>> (kafka.utils.Log4jControllerRegistration$)
> >>>>>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal exception
> >>>>>> (kafka.Kafka$)
> >>>>>> java.lang.IllegalArgumentException: requirement failed: At least one
> >>> log
> >>>>>> directory must be defined via log.dirs or log.dir.
> >>>>>>
> >>>>>> JR45. Could you group all configs for WorkerConfig together?
> >>>>>>
> >>>>>> JR46. plugin.path: We have the following code. So, currently, it
> >> seems
> >>>>> that
> >>>>>> an empty list is allowed for plugin.path and we treat it by adding
> >> the
> >>>>>> parent path of the classloader. It would be useful to justify why
> >>>>>> disallowing empty is better for the users.
> >>>>>>
> >>>>>>  public static Set<PluginSource> pluginSources(Set<Path>
> >>>>>> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
> >>>>> factory)
> >>>>>> {
> >>>>>>      Set<PluginSource> pluginSources = new LinkedHashSet<>();
> >>>>>>      for (Path pluginLocation : pluginLocations) {
> >>>>>>          try {
> >>>>>>              pluginSources.add(isolatedPluginSource(pluginLocation,
> >>>>>> classLoader, factory));
> >>>>>>          } catch (InvalidPathException | MalformedURLException e) {
> >>>>>>              log.error("Invalid path in plugin path: {}. Ignoring.",
> >>>>>> pluginLocation, e);
> >>>>>>          } catch (IOException e) {
> >>>>>>              log.error("Could not get listing for plugin path: {}.
> >>>>>> Ignoring.", pluginLocation, e);
> >>>>>>          }
> >>>>>>      }
> >>>>>>
> >>> pluginSources.add(classpathPluginSource(classLoader.getParent()));
> >>>>>>      return pluginSources;
> >>>>>>  }
> >>>>>>
> >>>>>> JR47. typo in "Should not allow setting validStrings is empty"
> >>>>>>
> >>>>>> JR48. We should pass in isEmptyAllowed as false in the code below,
> >>> right?
> >>>>>>  public static ValidList in(String... validStrings) {
> >>>>>>      if (validStrings.length == 0) {
> >>>>>>          throw new IllegalArgumentException("Valid strings list
> >> cannot
> >>>>>> be empty for in validator");
> >>>>>>      }
> >>>>>>      return new ValidList(List.of(validStrings), true, false);
> >>>>>>  }
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hello Jun,
> >>>>>>>
> >>>>>>> Sorry for the late reply,
> >>>>>>>
> >>>>>>> JR40, JR41: I have updated the KIP based on your comments. Please
> >>> take a
> >>>>>>> look.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
> >>>>>>>>
> >>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> Thanks for the reply.
> >>>>>>>>
> >>>>>>>> JR40. In the table, there are a few configs for which we now
> >> disallow
> >>>>>>> empty
> >>>>>>>> lists. It would be useful to document the current behavior for
> each
> >>> of
> >>>>>>> them
> >>>>>>>> more accurately. It seems that they should all lead to some bad
> >>>>> behavior.
> >>>>>>>> For example, if log.dirs is empty, it seems that we can't create
> >>>>> topics?
> >>>>>>>> If group.coordinator.rebalance.protocols is empty, the group
> >>> rebalance
> >>>>>>>> protocol will fail?
> >>>>>>>>
> >>>>>>>> JR41. It would be useful to clean up the "Compatibility,
> >> Deprecation,
> >>>>> and
> >>>>>>>> Migration Plan" section a bit.
> >>>>>>>> JR41.1 The migration plan has "Introduce a new properties
> >>>>> isEmptyAllowed
> >>>>>>> ,
> >>>>>>>> isNullAllowed", which needs to be changed.
> >>>>>>>> JR41.2 It started with "For users who configure an empty list",
> but
> >>> not
> >>>>>>> all
> >>>>>>>> the bullet points are about empty lists.
> >>>>>>>> JR41.3 To me, it would be useful to summarize why the behavior
> >>> changes
> >>>>>>> are
> >>>>>>>> justified. We made 3 types of changes. (1) Disallow null. This is
> >> ok
> >>>>>>> since
> >>>>>>>> one can't really set a null value through a config file. (2)
> >> Disallow
> >>>>>>>> duplicates. This is ok since it's rare and confusing. (3) Disallow
> >>>>> empty.
> >>>>>>>> Ideally, we want to claim that for those cases, they all lead to
> >> some
> >>>>>>> other
> >>>>>>>> errors or had behavior currently.
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hello Jun,
> >>>>>>>>>
> >>>>>>>>> Thanks for the reply.
> >>>>>>>>>
> >>>>>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
> >>>>>>>>>
> >>>>>>>>> JR38: I clarified the following configurations:
> >>>>>>>>> early.start.listeners: This config is optional and allows both
> >> null
> >>>>> and
> >>>>>>>>> empty lists.
> >>>>>>>>> log.dirs and plugin.path: These are also optional, but only allow
> >>>>>>>>> null—empty lists are not permitted.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> >>>>>>>>>>
> >>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>
> >>>>>>>>>> JR34. Sounds good.
> >>>>>>>>>>
> >>>>>>>>>> JR36. Could you document that the default value for
> >>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have
> >> been
> >>>>>>>>> changed?
> >>>>>>>>>>
> >>>>>>>>>> JR38. My concern wasn't quite addressed. "If the configuration
> is
> >>>>>>>>> optional,
> >>>>>>>>>> we will reject any duplicate values in the list." This implies
> >> that
> >>>>> if
> >>>>>>> a
> >>>>>>>>>> config is optional, we allow empty lists. However, for
> >> plugin.path,
> >>>>> we
> >>>>>>>>>> reject empty lists.
> >>>>>>>>>>
> >>>>>>>>>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed
> >>> since
> >>>>>>>>> they
> >>>>>>>>>> are not public facing.
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello Jun, Chia,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the reply,
> >>>>>>>>>>>
> >>>>>>>>>>> JR36: I’ve updated the following configurations:
> >>>>>>>>>>> - sasl.oauthbearer.expected.audience
> >>>>>>>>>>> - ssl.cipher.suites
> >>>>>>>>>>> - ssl.enabled.protocols
> >>>>>>>>>>> These now allow both empty and non-empty lists. Since the
> >> default
> >>>>>>> values
> >>>>>>>>>>> of
> >>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
> >>>>>>> previously
> >>>>>>>>>>> null,
> >>>>>>>>>>> I’ve updated their defaults to List.of() for consistency.
> >>>>>>>>>>>
> >>>>>>>>>>> JR37: A ClassCastException does not occur in MirrorClientConfig
> >>> and
> >>>>> it
> >>>>>>>>>>> should be none.
> >>>>>>>>>>> I’ve updated this in the KIP.
> >>>>>>>>>>>
> >>>>>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate
> which
> >>>>>>>>>>> configurations require special attention.
> >>>>>>>>>>>
> >>>>>>>>>>> JR39: Yes, these properties are package-private.
> >>>>>>>>>>>
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>
> >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
> return
> >>> an
> >>>>>>>>> empty
> >>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> >>>>> maxTasks,
> >>>>>>>>>>>> Chia-Ping?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty
> >>>>> list,
> >>>>>>>>>>> which
> >>>>>>>>>>>> means no `MirrorSourceTask` can be created.
> >>>>>>>>>>>>
> >>>>>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the configs
> >>> used
> >>>>>>> by
> >>>>>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions`
> >> with a
> >>>>>>>>>>> non-empty
> >>>>>>>>>>>> value.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
> >> return
> >>> an
> >>>>>>>>> empty
> >>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> >>>>> maxTasks,
> >>>>>>>>>>>>> Chia-Ping?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should we
> >> do
> >>>>> the
> >>>>>>>>> same
> >>>>>>>>>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers
> >>> returns
> >>>>>>>>>>>>> ClassCastException for null? It seems that null could be
> >> casted
> >>> to
> >>>>>>> any
> >>>>>>>>>>>>> class.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR38. "If the configuration is optional, we will reject any
> >>>>>>> duplicate
> >>>>>>>>>>>>> values in the list." Could you clarify this? For example,
> >>>>>>> plugin.path
> >>>>>>>>> is
> >>>>>>>>>>>>> optional, but with a default value. We reject duplicates as
> >> well
> >>>>> as
> >>>>>>>>>>> empty
> >>>>>>>>>>>>> lists.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields,
> >>>>> right?
> >>>>>>>>>>> They
> >>>>>>>>>>>>> are only public in the anyNonDuplicateValues() method.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR40. It would be useful to mention that if cleanup.policy is
> >>>>> empty
> >>>>>>>>> and
> >>>>>>>>>>>>> remote.storage.enable is true, the local log segments will be
> >>>>>>> cleaned
> >>>>>>>>>>> based
> >>>>>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
> >>>>> chia7712@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR34: For these two configurations, setting an empty list
> >>> feels
> >>>>> a
> >>>>>>>>> bit
> >>>>>>>>>>>>>>> unintuitive. If an empty list is
> >>>>>>>>>>>>>>> provided, the consumer will call the unsubscribe method,
> >> which
> >>>>>>>>> doesn't
> >>>>>>>>>>>>>>> seem appropriate given
> >>>>>>>>>>>>>>> the documentation states: "Topic-partitions assigned to
> this
> >>>>> task
> >>>>>>> to
> >>>>>>>>>>>>>>> replicate."
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> agreed. Additionally, the source code [0] shows that it is
> >> not
> >>>>>>>>> intended
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>> generate empty tps for the source task
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [0]
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello chia,

Thanks for the reply. 

chia05: I have updated the KIP to change the default values of task.assigned.groups 
and task.assigned.partitions from null to NO_DEFAULT_VALUE. 

chia06: I have also updated the KIP based on your feedback.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年8月6日 凌晨12:54 寫道：
> 
> hi Jiunn-Yang
> 
> chia05:
> 
> `task.assigned.groups` and `task.assigned.partitions` should not have
> default value, right?
> 
> chia06:
> 
> `new HashSet<>(values).size()` -> `Set.copyOf(values).size()`
> 
> Best,
> Chia-Ping
> 
> Jun Rao <ju...@confluent.io.invalid> 於 2025年8月5日 週二 下午11:39寫道：
> 
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the reply and your patience. The KIP looks good to me now.
>> 
>> Hi, Luke, Mickael,
>> 
>> Could you take a look at the KIP again and see if it makes sense to you?
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hello Jun,
>>> 
>>> Thanks for the reply,
>>> 
>>> JR46: I have clarified the behavior for null and empty values, so users
>>> can
>>> understand the reasoning behind disallowing empty lists.
>>> 
>>> JR48: I've updated the KIP accordingly.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
>>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> JR46. "Currently, if plugin.path is set to an empty list, no
>>> user-specified
>>>> plugin locations are processed, but plugins from the parent classloader
>>> are
>>>> still loaded." Could you add why the current behavior is bad for the
>>> users?
>>>> 
>>>> JR48. If we want to preserve the current behavior, we should remove the
>>>> following validation, right?
>>>>       if (validStrings.length == 0) {
>>>>           throw new IllegalArgumentException("Valid strings list
>> cannot
>>>> be empty for in validator");
>>>> 
>>>> Jun
>>>> 
>>>> On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hello Jun,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> JR42, JR45, JR46, JR47: I have updated the KIP.
>>>>> 
>>>>> JR43: If bootstrap.servers is empty for these configurations,
>>>>> it will fail when creating the AdminClient. I have updated the KIP to
>>>>> reflect this behavior.
>>>>> 
>>>>> JR44: Thanks for pointing this out. Kafka will indeed fail at the
>>> storage
>>>>> format phase
>>>>> if the configuration is invalid. I have updated the KIP accordingly.
>>>>> 
>>>>> JR48: The current in() method allows empty lists by default, so I
>>> believe
>>>>> setting isEmptyAllowed to true
>>>>> better preserves the original behavior. Therefore, I prefer not to
>>> change
>>>>> this default.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
>>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply. A few more comments.
>>>>>> 
>>>>>> JR42. [6] [7] are not needed since currently an empty list already
>>> causes
>>>>>> an exception.
>>>>>> 
>>>>>> JR43. Could you document the current behavior when bootstrap.servers
>> is
>>>>>> empty in MirrorClientConfig and WorkerConfig?
>>>>>> 
>>>>>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty
>> and
>>>>> got
>>>>>> the following.
>>>>>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
>>>>>> config/server.properties
>>>>>> Exception in thread "main" java.lang.IllegalArgumentException:
>>>>> requirement
>>>>>> failed: At least one log directory must be defined via log.dirs or
>>>>> log.dir.
>>>>>> 
>>>>>> bin/kafka-server-start.sh config/server.properties
>>>>>> [2025-08-01 11:31:20,097] INFO Registered
>>>>>> `kafka:type=kafka.Log4jController` MBean
>>>>>> (kafka.utils.Log4jControllerRegistration$)
>>>>>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal exception
>>>>>> (kafka.Kafka$)
>>>>>> java.lang.IllegalArgumentException: requirement failed: At least one
>>> log
>>>>>> directory must be defined via log.dirs or log.dir.
>>>>>> 
>>>>>> JR45. Could you group all configs for WorkerConfig together?
>>>>>> 
>>>>>> JR46. plugin.path: We have the following code. So, currently, it
>> seems
>>>>> that
>>>>>> an empty list is allowed for plugin.path and we treat it by adding
>> the
>>>>>> parent path of the classloader. It would be useful to justify why
>>>>>> disallowing empty is better for the users.
>>>>>> 
>>>>>>  public static Set<PluginSource> pluginSources(Set<Path>
>>>>>> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
>>>>> factory)
>>>>>> {
>>>>>>      Set<PluginSource> pluginSources = new LinkedHashSet<>();
>>>>>>      for (Path pluginLocation : pluginLocations) {
>>>>>>          try {
>>>>>>              pluginSources.add(isolatedPluginSource(pluginLocation,
>>>>>> classLoader, factory));
>>>>>>          } catch (InvalidPathException | MalformedURLException e) {
>>>>>>              log.error("Invalid path in plugin path: {}. Ignoring.",
>>>>>> pluginLocation, e);
>>>>>>          } catch (IOException e) {
>>>>>>              log.error("Could not get listing for plugin path: {}.
>>>>>> Ignoring.", pluginLocation, e);
>>>>>>          }
>>>>>>      }
>>>>>> 
>>> pluginSources.add(classpathPluginSource(classLoader.getParent()));
>>>>>>      return pluginSources;
>>>>>>  }
>>>>>> 
>>>>>> JR47. typo in "Should not allow setting validStrings is empty"
>>>>>> 
>>>>>> JR48. We should pass in isEmptyAllowed as false in the code below,
>>> right?
>>>>>>  public static ValidList in(String... validStrings) {
>>>>>>      if (validStrings.length == 0) {
>>>>>>          throw new IllegalArgumentException("Valid strings list
>> cannot
>>>>>> be empty for in validator");
>>>>>>      }
>>>>>>      return new ValidList(List.of(validStrings), true, false);
>>>>>>  }
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hello Jun,
>>>>>>> 
>>>>>>> Sorry for the late reply,
>>>>>>> 
>>>>>>> JR40, JR41: I have updated the KIP based on your comments. Please
>>> take a
>>>>>>> look.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
>>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> JR40. In the table, there are a few configs for which we now
>> disallow
>>>>>>> empty
>>>>>>>> lists. It would be useful to document the current behavior for each
>>> of
>>>>>>> them
>>>>>>>> more accurately. It seems that they should all lead to some bad
>>>>> behavior.
>>>>>>>> For example, if log.dirs is empty, it seems that we can't create
>>>>> topics?
>>>>>>>> If group.coordinator.rebalance.protocols is empty, the group
>>> rebalance
>>>>>>>> protocol will fail?
>>>>>>>> 
>>>>>>>> JR41. It would be useful to clean up the "Compatibility,
>> Deprecation,
>>>>> and
>>>>>>>> Migration Plan" section a bit.
>>>>>>>> JR41.1 The migration plan has "Introduce a new properties
>>>>> isEmptyAllowed
>>>>>>> ,
>>>>>>>> isNullAllowed", which needs to be changed.
>>>>>>>> JR41.2 It started with "For users who configure an empty list", but
>>> not
>>>>>>> all
>>>>>>>> the bullet points are about empty lists.
>>>>>>>> JR41.3 To me, it would be useful to summarize why the behavior
>>> changes
>>>>>>> are
>>>>>>>> justified. We made 3 types of changes. (1) Disallow null. This is
>> ok
>>>>>>> since
>>>>>>>> one can't really set a null value through a config file. (2)
>> Disallow
>>>>>>>> duplicates. This is ok since it's rare and confusing. (3) Disallow
>>>>> empty.
>>>>>>>> Ideally, we want to claim that for those cases, they all lead to
>> some
>>>>>>> other
>>>>>>>> errors or had behavior currently.
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hello Jun,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply.
>>>>>>>>> 
>>>>>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
>>>>>>>>> 
>>>>>>>>> JR38: I clarified the following configurations:
>>>>>>>>> early.start.listeners: This config is optional and allows both
>> null
>>>>> and
>>>>>>>>> empty lists.
>>>>>>>>> log.dirs and plugin.path: These are also optional, but only allow
>>>>>>>>> null—empty lists are not permitted.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
>>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply.
>>>>>>>>>> 
>>>>>>>>>> JR34. Sounds good.
>>>>>>>>>> 
>>>>>>>>>> JR36. Could you document that the default value for
>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have
>> been
>>>>>>>>> changed?
>>>>>>>>>> 
>>>>>>>>>> JR38. My concern wasn't quite addressed. "If the configuration is
>>>>>>>>> optional,
>>>>>>>>>> we will reject any duplicate values in the list." This implies
>> that
>>>>> if
>>>>>>> a
>>>>>>>>>> config is optional, we allow empty lists. However, for
>> plugin.path,
>>>>> we
>>>>>>>>>> reject empty lists.
>>>>>>>>>> 
>>>>>>>>>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed
>>> since
>>>>>>>>> they
>>>>>>>>>> are not public facing.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello Jun, Chia,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the reply,
>>>>>>>>>>> 
>>>>>>>>>>> JR36: I’ve updated the following configurations:
>>>>>>>>>>> - sasl.oauthbearer.expected.audience
>>>>>>>>>>> - ssl.cipher.suites
>>>>>>>>>>> - ssl.enabled.protocols
>>>>>>>>>>> These now allow both empty and non-empty lists. Since the
>> default
>>>>>>> values
>>>>>>>>>>> of
>>>>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
>>>>>>> previously
>>>>>>>>>>> null,
>>>>>>>>>>> I’ve updated their defaults to List.of() for consistency.
>>>>>>>>>>> 
>>>>>>>>>>> JR37: A ClassCastException does not occur in MirrorClientConfig
>>> and
>>>>> it
>>>>>>>>>>> should be none.
>>>>>>>>>>> I’ve updated this in the KIP.
>>>>>>>>>>> 
>>>>>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate which
>>>>>>>>>>> configurations require special attention.
>>>>>>>>>>> 
>>>>>>>>>>> JR39: Yes, these properties are package-private.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return
>>> an
>>>>>>>>> empty
>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>>>>> maxTasks,
>>>>>>>>>>>> Chia-Ping?
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty
>>>>> list,
>>>>>>>>>>> which
>>>>>>>>>>>> means no `MirrorSourceTask` can be created.
>>>>>>>>>>>> 
>>>>>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the configs
>>> used
>>>>>>> by
>>>>>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions`
>> with a
>>>>>>>>>>> non-empty
>>>>>>>>>>>> value.
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
>> return
>>> an
>>>>>>>>> empty
>>>>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>>>>> maxTasks,
>>>>>>>>>>>>> Chia-Ping?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should we
>> do
>>>>> the
>>>>>>>>> same
>>>>>>>>>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers
>>> returns
>>>>>>>>>>>>> ClassCastException for null? It seems that null could be
>> casted
>>> to
>>>>>>> any
>>>>>>>>>>>>> class.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR38. "If the configuration is optional, we will reject any
>>>>>>> duplicate
>>>>>>>>>>>>> values in the list." Could you clarify this? For example,
>>>>>>> plugin.path
>>>>>>>>> is
>>>>>>>>>>>>> optional, but with a default value. We reject duplicates as
>> well
>>>>> as
>>>>>>>>>>> empty
>>>>>>>>>>>>> lists.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields,
>>>>> right?
>>>>>>>>>>> They
>>>>>>>>>>>>> are only public in the anyNonDuplicateValues() method.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR40. It would be useful to mention that if cleanup.policy is
>>>>> empty
>>>>>>>>> and
>>>>>>>>>>>>> remote.storage.enable is true, the local log segments will be
>>>>>>> cleaned
>>>>>>>>>>> based
>>>>>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
>>>>> chia7712@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR34: For these two configurations, setting an empty list
>>> feels
>>>>> a
>>>>>>>>> bit
>>>>>>>>>>>>>>> unintuitive. If an empty list is
>>>>>>>>>>>>>>> provided, the consumer will call the unsubscribe method,
>> which
>>>>>>>>> doesn't
>>>>>>>>>>>>>>> seem appropriate given
>>>>>>>>>>>>>>> the documentation states: "Topic-partitions assigned to this
>>>>> task
>>>>>>> to
>>>>>>>>>>>>>>> replicate."
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> agreed. Additionally, the source code [0] shows that it is
>> not
>>>>>>>>> intended
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> generate empty tps for the source task
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

hi Jiunn-Yang

chia05:

`task.assigned.groups` and `task.assigned.partitions` should not have
default value, right?

chia06:

`new HashSet<>(values).size()` -> `Set.copyOf(values).size()`

Best,
Chia-Ping

Jun Rao <ju...@confluent.io.invalid> 於 2025年8月5日 週二 下午11:39寫道：

> Hi, Jiunn-Yang,
>
> Thanks for the reply and your patience. The KIP looks good to me now.
>
> Hi, Luke, Mickael,
>
> Could you take a look at the KIP again and see if it makes sense to you?
>
> Thanks,
>
> Jun
>
> On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hello Jun,
> >
> > Thanks for the reply,
> >
> > JR46: I have clarified the behavior for null and empty values, so users
> > can
> > understand the reasoning behind disallowing empty lists.
> >
> > JR48: I've updated the KIP accordingly.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
> > >
> > > Hi, Jiunn-Yang,
> > >
> > > Thanks for the reply.
> > >
> > > JR46. "Currently, if plugin.path is set to an empty list, no
> > user-specified
> > > plugin locations are processed, but plugins from the parent classloader
> > are
> > > still loaded." Could you add why the current behavior is bad for the
> > users?
> > >
> > > JR48. If we want to preserve the current behavior, we should remove the
> > > following validation, right?
> > >        if (validStrings.length == 0) {
> > >            throw new IllegalArgumentException("Valid strings list
> cannot
> > > be empty for in validator");
> > >
> > > Jun
> > >
> > > On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
> > >
> > >> Hello Jun,
> > >>
> > >> Thanks for the reply.
> > >>
> > >> JR42, JR45, JR46, JR47: I have updated the KIP.
> > >>
> > >> JR43: If bootstrap.servers is empty for these configurations,
> > >> it will fail when creating the AdminClient. I have updated the KIP to
> > >> reflect this behavior.
> > >>
> > >> JR44: Thanks for pointing this out. Kafka will indeed fail at the
> > storage
> > >> format phase
> > >> if the configuration is invalid. I have updated the KIP accordingly.
> > >>
> > >> JR48: The current in() method allows empty lists by default, so I
> > believe
> > >> setting isEmptyAllowed to true
> > >> better preserves the original behavior. Therefore, I prefer not to
> > change
> > >> this default.
> > >>
> > >> Best Regards,
> > >> Jiunn-Yang
> > >>
> > >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
> > >>>
> > >>> Hi, Jiunn-Yang,
> > >>>
> > >>> Thanks for the reply. A few more comments.
> > >>>
> > >>> JR42. [6] [7] are not needed since currently an empty list already
> > causes
> > >>> an exception.
> > >>>
> > >>> JR43. Could you document the current behavior when bootstrap.servers
> is
> > >>> empty in MirrorClientConfig and WorkerConfig?
> > >>>
> > >>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty
> and
> > >> got
> > >>> the following.
> > >>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
> > >>> config/server.properties
> > >>> Exception in thread "main" java.lang.IllegalArgumentException:
> > >> requirement
> > >>> failed: At least one log directory must be defined via log.dirs or
> > >> log.dir.
> > >>>
> > >>> bin/kafka-server-start.sh config/server.properties
> > >>> [2025-08-01 11:31:20,097] INFO Registered
> > >>> `kafka:type=kafka.Log4jController` MBean
> > >>> (kafka.utils.Log4jControllerRegistration$)
> > >>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal exception
> > >>> (kafka.Kafka$)
> > >>> java.lang.IllegalArgumentException: requirement failed: At least one
> > log
> > >>> directory must be defined via log.dirs or log.dir.
> > >>>
> > >>> JR45. Could you group all configs for WorkerConfig together?
> > >>>
> > >>> JR46. plugin.path: We have the following code. So, currently, it
> seems
> > >> that
> > >>> an empty list is allowed for plugin.path and we treat it by adding
> the
> > >>> parent path of the classloader. It would be useful to justify why
> > >>> disallowing empty is better for the users.
> > >>>
> > >>>   public static Set<PluginSource> pluginSources(Set<Path>
> > >>> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
> > >> factory)
> > >>> {
> > >>>       Set<PluginSource> pluginSources = new LinkedHashSet<>();
> > >>>       for (Path pluginLocation : pluginLocations) {
> > >>>           try {
> > >>>               pluginSources.add(isolatedPluginSource(pluginLocation,
> > >>> classLoader, factory));
> > >>>           } catch (InvalidPathException | MalformedURLException e) {
> > >>>               log.error("Invalid path in plugin path: {}. Ignoring.",
> > >>> pluginLocation, e);
> > >>>           } catch (IOException e) {
> > >>>               log.error("Could not get listing for plugin path: {}.
> > >>> Ignoring.", pluginLocation, e);
> > >>>           }
> > >>>       }
> > >>>
> >  pluginSources.add(classpathPluginSource(classLoader.getParent()));
> > >>>       return pluginSources;
> > >>>   }
> > >>>
> > >>> JR47. typo in "Should not allow setting validStrings is empty"
> > >>>
> > >>> JR48. We should pass in isEmptyAllowed as false in the code below,
> > right?
> > >>>   public static ValidList in(String... validStrings) {
> > >>>       if (validStrings.length == 0) {
> > >>>           throw new IllegalArgumentException("Valid strings list
> cannot
> > >>> be empty for in validator");
> > >>>       }
> > >>>       return new ValidList(List.of(validStrings), true, false);
> > >>>   }
> > >>>
> > >>> Jun
> > >>>
> > >>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>
> > >>>> Hello Jun,
> > >>>>
> > >>>> Sorry for the late reply,
> > >>>>
> > >>>> JR40, JR41: I have updated the KIP based on your comments. Please
> > take a
> > >>>> look.
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
> > >>>>>
> > >>>>> Hi, Jiunn-Yang,
> > >>>>>
> > >>>>> Thanks for the reply.
> > >>>>>
> > >>>>> JR40. In the table, there are a few configs for which we now
> disallow
> > >>>> empty
> > >>>>> lists. It would be useful to document the current behavior for each
> > of
> > >>>> them
> > >>>>> more accurately. It seems that they should all lead to some bad
> > >> behavior.
> > >>>>> For example, if log.dirs is empty, it seems that we can't create
> > >> topics?
> > >>>>> If group.coordinator.rebalance.protocols is empty, the group
> > rebalance
> > >>>>> protocol will fail?
> > >>>>>
> > >>>>> JR41. It would be useful to clean up the "Compatibility,
> Deprecation,
> > >> and
> > >>>>> Migration Plan" section a bit.
> > >>>>> JR41.1 The migration plan has "Introduce a new properties
> > >> isEmptyAllowed
> > >>>> ,
> > >>>>> isNullAllowed", which needs to be changed.
> > >>>>> JR41.2 It started with "For users who configure an empty list", but
> > not
> > >>>> all
> > >>>>> the bullet points are about empty lists.
> > >>>>> JR41.3 To me, it would be useful to summarize why the behavior
> > changes
> > >>>> are
> > >>>>> justified. We made 3 types of changes. (1) Disallow null. This is
> ok
> > >>>> since
> > >>>>> one can't really set a null value through a config file. (2)
> Disallow
> > >>>>> duplicates. This is ok since it's rare and confusing. (3) Disallow
> > >> empty.
> > >>>>> Ideally, we want to claim that for those cases, they all lead to
> some
> > >>>> other
> > >>>>> errors or had behavior currently.
> > >>>>>
> > >>>>> Jun
> > >>>>>
> > >>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>
> > >>>>>> Hello Jun,
> > >>>>>>
> > >>>>>> Thanks for the reply.
> > >>>>>>
> > >>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
> > >>>>>>
> > >>>>>> JR38: I clarified the following configurations:
> > >>>>>> early.start.listeners: This config is optional and allows both
> null
> > >> and
> > >>>>>> empty lists.
> > >>>>>> log.dirs and plugin.path: These are also optional, but only allow
> > >>>>>> null—empty lists are not permitted.
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>> Jiunn-Yang
> > >>>>>>
> > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> > >>>>>>>
> > >>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> > >>>>>>>
> > >>>>>>> Thanks for the reply.
> > >>>>>>>
> > >>>>>>> JR34. Sounds good.
> > >>>>>>>
> > >>>>>>> JR36. Could you document that the default value for
> > >>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have
> been
> > >>>>>> changed?
> > >>>>>>>
> > >>>>>>> JR38. My concern wasn't quite addressed. "If the configuration is
> > >>>>>> optional,
> > >>>>>>> we will reject any duplicate values in the list." This implies
> that
> > >> if
> > >>>> a
> > >>>>>>> config is optional, we allow empty lists. However, for
> plugin.path,
> > >> we
> > >>>>>>> reject empty lists.
> > >>>>>>>
> > >>>>>>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed
> > since
> > >>>>>> they
> > >>>>>>> are not public facing.
> > >>>>>>>
> > >>>>>>> Jun
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> Hello Jun, Chia,
> > >>>>>>>>
> > >>>>>>>> Thanks for the reply,
> > >>>>>>>>
> > >>>>>>>> JR36: I’ve updated the following configurations:
> > >>>>>>>> - sasl.oauthbearer.expected.audience
> > >>>>>>>> - ssl.cipher.suites
> > >>>>>>>> - ssl.enabled.protocols
> > >>>>>>>> These now allow both empty and non-empty lists. Since the
> default
> > >>>> values
> > >>>>>>>> of
> > >>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
> > >>>> previously
> > >>>>>>>> null,
> > >>>>>>>> I’ve updated their defaults to List.of() for consistency.
> > >>>>>>>>
> > >>>>>>>> JR37: A ClassCastException does not occur in MirrorClientConfig
> > and
> > >> it
> > >>>>>>>> should be none.
> > >>>>>>>> I’ve updated this in the KIP.
> > >>>>>>>>
> > >>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate which
> > >>>>>>>> configurations require special attention.
> > >>>>>>>>
> > >>>>>>>> JR39: Yes, these properties are package-private.
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Jiunn-Yang
> > >>>>>>>>
> > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> > >>>>>>>>>
> > >>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return
> > an
> > >>>>>> empty
> > >>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> > >> maxTasks,
> > >>>>>>>>> Chia-Ping?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty
> > >> list,
> > >>>>>>>> which
> > >>>>>>>>> means no `MirrorSourceTask` can be created.
> > >>>>>>>>>
> > >>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the configs
> > used
> > >>>> by
> > >>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions`
> with a
> > >>>>>>>> non-empty
> > >>>>>>>>> value.
> > >>>>>>>>>
> > >>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
> > >>>>>>>>>
> > >>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>
> > >>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could
> return
> > an
> > >>>>>> empty
> > >>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> > >> maxTasks,
> > >>>>>>>>>> Chia-Ping?
> > >>>>>>>>>>
> > >>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should we
> do
> > >> the
> > >>>>>> same
> > >>>>>>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
> > >>>>>>>>>>
> > >>>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers
> > returns
> > >>>>>>>>>> ClassCastException for null? It seems that null could be
> casted
> > to
> > >>>> any
> > >>>>>>>>>> class.
> > >>>>>>>>>>
> > >>>>>>>>>> JR38. "If the configuration is optional, we will reject any
> > >>>> duplicate
> > >>>>>>>>>> values in the list." Could you clarify this? For example,
> > >>>> plugin.path
> > >>>>>> is
> > >>>>>>>>>> optional, but with a default value. We reject duplicates as
> well
> > >> as
> > >>>>>>>> empty
> > >>>>>>>>>> lists.
> > >>>>>>>>>>
> > >>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields,
> > >> right?
> > >>>>>>>> They
> > >>>>>>>>>> are only public in the anyNonDuplicateValues() method.
> > >>>>>>>>>>
> > >>>>>>>>>> JR40. It would be useful to mention that if cleanup.policy is
> > >> empty
> > >>>>>> and
> > >>>>>>>>>> remote.storage.enable is true, the local log segments will be
> > >>>> cleaned
> > >>>>>>>> based
> > >>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
> > >> chia7712@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> JR34: For these two configurations, setting an empty list
> > feels
> > >> a
> > >>>>>> bit
> > >>>>>>>>>>>> unintuitive. If an empty list is
> > >>>>>>>>>>>> provided, the consumer will call the unsubscribe method,
> which
> > >>>>>> doesn't
> > >>>>>>>>>>>> seem appropriate given
> > >>>>>>>>>>>> the documentation states: "Topic-partitions assigned to this
> > >> task
> > >>>> to
> > >>>>>>>>>>>> replicate."
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> agreed. Additionally, the source code [0] shows that it is
> not
> > >>>>>> intended
> > >>>>>>>>>> to
> > >>>>>>>>>>> generate empty tps for the source task
> > >>>>>>>>>>>
> > >>>>>>>>>>> [0]
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply and your patience. The KIP looks good to me now.

Hi, Luke, Mickael,

Could you take a look at the KIP again and see if it makes sense to you?

Thanks,

Jun

On Tue, Aug 5, 2025 at 3:04 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> Thanks for the reply,
>
> JR46: I have clarified the behavior for null and empty values, so users
> can
> understand the reasoning behind disallowing empty lists.
>
> JR48: I've updated the KIP accordingly.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the reply.
> >
> > JR46. "Currently, if plugin.path is set to an empty list, no
> user-specified
> > plugin locations are processed, but plugins from the parent classloader
> are
> > still loaded." Could you add why the current behavior is bad for the
> users?
> >
> > JR48. If we want to preserve the current behavior, we should remove the
> > following validation, right?
> >        if (validStrings.length == 0) {
> >            throw new IllegalArgumentException("Valid strings list cannot
> > be empty for in validator");
> >
> > Jun
> >
> > On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello Jun,
> >>
> >> Thanks for the reply.
> >>
> >> JR42, JR45, JR46, JR47: I have updated the KIP.
> >>
> >> JR43: If bootstrap.servers is empty for these configurations,
> >> it will fail when creating the AdminClient. I have updated the KIP to
> >> reflect this behavior.
> >>
> >> JR44: Thanks for pointing this out. Kafka will indeed fail at the
> storage
> >> format phase
> >> if the configuration is invalid. I have updated the KIP accordingly.
> >>
> >> JR48: The current in() method allows empty lists by default, so I
> believe
> >> setting isEmptyAllowed to true
> >> better preserves the original behavior. Therefore, I prefer not to
> change
> >> this default.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
> >>>
> >>> Hi, Jiunn-Yang,
> >>>
> >>> Thanks for the reply. A few more comments.
> >>>
> >>> JR42. [6] [7] are not needed since currently an empty list already
> causes
> >>> an exception.
> >>>
> >>> JR43. Could you document the current behavior when bootstrap.servers is
> >>> empty in MirrorClientConfig and WorkerConfig?
> >>>
> >>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty and
> >> got
> >>> the following.
> >>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
> >>> config/server.properties
> >>> Exception in thread "main" java.lang.IllegalArgumentException:
> >> requirement
> >>> failed: At least one log directory must be defined via log.dirs or
> >> log.dir.
> >>>
> >>> bin/kafka-server-start.sh config/server.properties
> >>> [2025-08-01 11:31:20,097] INFO Registered
> >>> `kafka:type=kafka.Log4jController` MBean
> >>> (kafka.utils.Log4jControllerRegistration$)
> >>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal exception
> >>> (kafka.Kafka$)
> >>> java.lang.IllegalArgumentException: requirement failed: At least one
> log
> >>> directory must be defined via log.dirs or log.dir.
> >>>
> >>> JR45. Could you group all configs for WorkerConfig together?
> >>>
> >>> JR46. plugin.path: We have the following code. So, currently, it seems
> >> that
> >>> an empty list is allowed for plugin.path and we treat it by adding the
> >>> parent path of the classloader. It would be useful to justify why
> >>> disallowing empty is better for the users.
> >>>
> >>>   public static Set<PluginSource> pluginSources(Set<Path>
> >>> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
> >> factory)
> >>> {
> >>>       Set<PluginSource> pluginSources = new LinkedHashSet<>();
> >>>       for (Path pluginLocation : pluginLocations) {
> >>>           try {
> >>>               pluginSources.add(isolatedPluginSource(pluginLocation,
> >>> classLoader, factory));
> >>>           } catch (InvalidPathException | MalformedURLException e) {
> >>>               log.error("Invalid path in plugin path: {}. Ignoring.",
> >>> pluginLocation, e);
> >>>           } catch (IOException e) {
> >>>               log.error("Could not get listing for plugin path: {}.
> >>> Ignoring.", pluginLocation, e);
> >>>           }
> >>>       }
> >>>
>  pluginSources.add(classpathPluginSource(classLoader.getParent()));
> >>>       return pluginSources;
> >>>   }
> >>>
> >>> JR47. typo in "Should not allow setting validStrings is empty"
> >>>
> >>> JR48. We should pass in isEmptyAllowed as false in the code below,
> right?
> >>>   public static ValidList in(String... validStrings) {
> >>>       if (validStrings.length == 0) {
> >>>           throw new IllegalArgumentException("Valid strings list cannot
> >>> be empty for in validator");
> >>>       }
> >>>       return new ValidList(List.of(validStrings), true, false);
> >>>   }
> >>>
> >>> Jun
> >>>
> >>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>
> >>>> Hello Jun,
> >>>>
> >>>> Sorry for the late reply,
> >>>>
> >>>> JR40, JR41: I have updated the KIP based on your comments. Please
> take a
> >>>> look.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
> >>>>>
> >>>>> Hi, Jiunn-Yang,
> >>>>>
> >>>>> Thanks for the reply.
> >>>>>
> >>>>> JR40. In the table, there are a few configs for which we now disallow
> >>>> empty
> >>>>> lists. It would be useful to document the current behavior for each
> of
> >>>> them
> >>>>> more accurately. It seems that they should all lead to some bad
> >> behavior.
> >>>>> For example, if log.dirs is empty, it seems that we can't create
> >> topics?
> >>>>> If group.coordinator.rebalance.protocols is empty, the group
> rebalance
> >>>>> protocol will fail?
> >>>>>
> >>>>> JR41. It would be useful to clean up the "Compatibility, Deprecation,
> >> and
> >>>>> Migration Plan" section a bit.
> >>>>> JR41.1 The migration plan has "Introduce a new properties
> >> isEmptyAllowed
> >>>> ,
> >>>>> isNullAllowed", which needs to be changed.
> >>>>> JR41.2 It started with "For users who configure an empty list", but
> not
> >>>> all
> >>>>> the bullet points are about empty lists.
> >>>>> JR41.3 To me, it would be useful to summarize why the behavior
> changes
> >>>> are
> >>>>> justified. We made 3 types of changes. (1) Disallow null. This is ok
> >>>> since
> >>>>> one can't really set a null value through a config file. (2) Disallow
> >>>>> duplicates. This is ok since it's rare and confusing. (3) Disallow
> >> empty.
> >>>>> Ideally, we want to claim that for those cases, they all lead to some
> >>>> other
> >>>>> errors or had behavior currently.
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>
> >>>>>> Hello Jun,
> >>>>>>
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
> >>>>>>
> >>>>>> JR38: I clarified the following configurations:
> >>>>>> early.start.listeners: This config is optional and allows both null
> >> and
> >>>>>> empty lists.
> >>>>>> log.dirs and plugin.path: These are also optional, but only allow
> >>>>>> null—empty lists are not permitted.
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> >>>>>>>
> >>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>>>
> >>>>>>> Thanks for the reply.
> >>>>>>>
> >>>>>>> JR34. Sounds good.
> >>>>>>>
> >>>>>>> JR36. Could you document that the default value for
> >>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have been
> >>>>>> changed?
> >>>>>>>
> >>>>>>> JR38. My concern wasn't quite addressed. "If the configuration is
> >>>>>> optional,
> >>>>>>> we will reject any duplicate values in the list." This implies that
> >> if
> >>>> a
> >>>>>>> config is optional, we allow empty lists. However, for plugin.path,
> >> we
> >>>>>>> reject empty lists.
> >>>>>>>
> >>>>>>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed
> since
> >>>>>> they
> >>>>>>> are not public facing.
> >>>>>>>
> >>>>>>> Jun
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hello Jun, Chia,
> >>>>>>>>
> >>>>>>>> Thanks for the reply,
> >>>>>>>>
> >>>>>>>> JR36: I’ve updated the following configurations:
> >>>>>>>> - sasl.oauthbearer.expected.audience
> >>>>>>>> - ssl.cipher.suites
> >>>>>>>> - ssl.enabled.protocols
> >>>>>>>> These now allow both empty and non-empty lists. Since the default
> >>>> values
> >>>>>>>> of
> >>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
> >>>> previously
> >>>>>>>> null,
> >>>>>>>> I’ve updated their defaults to List.of() for consistency.
> >>>>>>>>
> >>>>>>>> JR37: A ClassCastException does not occur in MirrorClientConfig
> and
> >> it
> >>>>>>>> should be none.
> >>>>>>>> I’ve updated this in the KIP.
> >>>>>>>>
> >>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate which
> >>>>>>>> configurations require special attention.
> >>>>>>>>
> >>>>>>>> JR39: Yes, these properties are package-private.
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>>
> >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> >>>>>>>>>
> >>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return
> an
> >>>>>> empty
> >>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> >> maxTasks,
> >>>>>>>>> Chia-Ping?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty
> >> list,
> >>>>>>>> which
> >>>>>>>>> means no `MirrorSourceTask` can be created.
> >>>>>>>>>
> >>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the configs
> used
> >>>> by
> >>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions` with a
> >>>>>>>> non-empty
> >>>>>>>>> value.
> >>>>>>>>>
> >>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
> >>>>>>>>>
> >>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>
> >>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return
> an
> >>>>>> empty
> >>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
> >> maxTasks,
> >>>>>>>>>> Chia-Ping?
> >>>>>>>>>>
> >>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should we do
> >> the
> >>>>>> same
> >>>>>>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
> >>>>>>>>>>
> >>>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers
> returns
> >>>>>>>>>> ClassCastException for null? It seems that null could be casted
> to
> >>>> any
> >>>>>>>>>> class.
> >>>>>>>>>>
> >>>>>>>>>> JR38. "If the configuration is optional, we will reject any
> >>>> duplicate
> >>>>>>>>>> values in the list." Could you clarify this? For example,
> >>>> plugin.path
> >>>>>> is
> >>>>>>>>>> optional, but with a default value. We reject duplicates as well
> >> as
> >>>>>>>> empty
> >>>>>>>>>> lists.
> >>>>>>>>>>
> >>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields,
> >> right?
> >>>>>>>> They
> >>>>>>>>>> are only public in the anyNonDuplicateValues() method.
> >>>>>>>>>>
> >>>>>>>>>> JR40. It would be useful to mention that if cleanup.policy is
> >> empty
> >>>>>> and
> >>>>>>>>>> remote.storage.enable is true, the local log segments will be
> >>>> cleaned
> >>>>>>>> based
> >>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
> >> chia7712@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR34: For these two configurations, setting an empty list
> feels
> >> a
> >>>>>> bit
> >>>>>>>>>>>> unintuitive. If an empty list is
> >>>>>>>>>>>> provided, the consumer will call the unsubscribe method, which
> >>>>>> doesn't
> >>>>>>>>>>>> seem appropriate given
> >>>>>>>>>>>> the documentation states: "Topic-partitions assigned to this
> >> task
> >>>> to
> >>>>>>>>>>>> replicate."
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> agreed. Additionally, the source code [0] shows that it is not
> >>>>>> intended
> >>>>>>>>>> to
> >>>>>>>>>>> generate empty tps for the source task
> >>>>>>>>>>>
> >>>>>>>>>>> [0]
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun, 

Thanks for the reply,

JR46: I have clarified the behavior for null and empty values, so users can 
understand the reasoning behind disallowing empty lists. 

JR48: I've updated the KIP accordingly.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月5日 凌晨12:45 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply.
> 
> JR46. "Currently, if plugin.path is set to an empty list, no user-specified
> plugin locations are processed, but plugins from the parent classloader are
> still loaded." Could you add why the current behavior is bad for the users?
> 
> JR48. If we want to preserve the current behavior, we should remove the
> following validation, right?
>        if (validStrings.length == 0) {
>            throw new IllegalArgumentException("Valid strings list cannot
> be empty for in validator");
> 
> Jun
> 
> On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun,
>> 
>> Thanks for the reply.
>> 
>> JR42, JR45, JR46, JR47: I have updated the KIP.
>> 
>> JR43: If bootstrap.servers is empty for these configurations,
>> it will fail when creating the AdminClient. I have updated the KIP to
>> reflect this behavior.
>> 
>> JR44: Thanks for pointing this out. Kafka will indeed fail at the storage
>> format phase
>> if the configuration is invalid. I have updated the KIP accordingly.
>> 
>> JR48: The current in() method allows empty lists by default, so I believe
>> setting isEmptyAllowed to true
>> better preserves the original behavior. Therefore, I prefer not to change
>> this default.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
>>> 
>>> Hi, Jiunn-Yang,
>>> 
>>> Thanks for the reply. A few more comments.
>>> 
>>> JR42. [6] [7] are not needed since currently an empty list already causes
>>> an exception.
>>> 
>>> JR43. Could you document the current behavior when bootstrap.servers is
>>> empty in MirrorClientConfig and WorkerConfig?
>>> 
>>> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty and
>> got
>>> the following.
>>> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
>>> config/server.properties
>>> Exception in thread "main" java.lang.IllegalArgumentException:
>> requirement
>>> failed: At least one log directory must be defined via log.dirs or
>> log.dir.
>>> 
>>> bin/kafka-server-start.sh config/server.properties
>>> [2025-08-01 11:31:20,097] INFO Registered
>>> `kafka:type=kafka.Log4jController` MBean
>>> (kafka.utils.Log4jControllerRegistration$)
>>> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal exception
>>> (kafka.Kafka$)
>>> java.lang.IllegalArgumentException: requirement failed: At least one log
>>> directory must be defined via log.dirs or log.dir.
>>> 
>>> JR45. Could you group all configs for WorkerConfig together?
>>> 
>>> JR46. plugin.path: We have the following code. So, currently, it seems
>> that
>>> an empty list is allowed for plugin.path and we treat it by adding the
>>> parent path of the classloader. It would be useful to justify why
>>> disallowing empty is better for the users.
>>> 
>>>   public static Set<PluginSource> pluginSources(Set<Path>
>>> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
>> factory)
>>> {
>>>       Set<PluginSource> pluginSources = new LinkedHashSet<>();
>>>       for (Path pluginLocation : pluginLocations) {
>>>           try {
>>>               pluginSources.add(isolatedPluginSource(pluginLocation,
>>> classLoader, factory));
>>>           } catch (InvalidPathException | MalformedURLException e) {
>>>               log.error("Invalid path in plugin path: {}. Ignoring.",
>>> pluginLocation, e);
>>>           } catch (IOException e) {
>>>               log.error("Could not get listing for plugin path: {}.
>>> Ignoring.", pluginLocation, e);
>>>           }
>>>       }
>>>       pluginSources.add(classpathPluginSource(classLoader.getParent()));
>>>       return pluginSources;
>>>   }
>>> 
>>> JR47. typo in "Should not allow setting validStrings is empty"
>>> 
>>> JR48. We should pass in isEmptyAllowed as false in the code below, right?
>>>   public static ValidList in(String... validStrings) {
>>>       if (validStrings.length == 0) {
>>>           throw new IllegalArgumentException("Valid strings list cannot
>>> be empty for in validator");
>>>       }
>>>       return new ValidList(List.of(validStrings), true, false);
>>>   }
>>> 
>>> Jun
>>> 
>>> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hello Jun,
>>>> 
>>>> Sorry for the late reply,
>>>> 
>>>> JR40, JR41: I have updated the KIP based on your comments. Please take a
>>>> look.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
>>>>> 
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> JR40. In the table, there are a few configs for which we now disallow
>>>> empty
>>>>> lists. It would be useful to document the current behavior for each of
>>>> them
>>>>> more accurately. It seems that they should all lead to some bad
>> behavior.
>>>>> For example, if log.dirs is empty, it seems that we can't create
>> topics?
>>>>> If group.coordinator.rebalance.protocols is empty, the group rebalance
>>>>> protocol will fail?
>>>>> 
>>>>> JR41. It would be useful to clean up the "Compatibility, Deprecation,
>> and
>>>>> Migration Plan" section a bit.
>>>>> JR41.1 The migration plan has "Introduce a new properties
>> isEmptyAllowed
>>>> ,
>>>>> isNullAllowed", which needs to be changed.
>>>>> JR41.2 It started with "For users who configure an empty list", but not
>>>> all
>>>>> the bullet points are about empty lists.
>>>>> JR41.3 To me, it would be useful to summarize why the behavior changes
>>>> are
>>>>> justified. We made 3 types of changes. (1) Disallow null. This is ok
>>>> since
>>>>> one can't really set a null value through a config file. (2) Disallow
>>>>> duplicates. This is ok since it's rare and confusing. (3) Disallow
>> empty.
>>>>> Ideally, we want to claim that for those cases, they all lead to some
>>>> other
>>>>> errors or had behavior currently.
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hello Jun,
>>>>>> 
>>>>>> Thanks for the reply.
>>>>>> 
>>>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
>>>>>> 
>>>>>> JR38: I clarified the following configurations:
>>>>>> early.start.listeners: This config is optional and allows both null
>> and
>>>>>> empty lists.
>>>>>> log.dirs and plugin.path: These are also optional, but only allow
>>>>>> null—empty lists are not permitted.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
>>>>>>> 
>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>> 
>>>>>>> Thanks for the reply.
>>>>>>> 
>>>>>>> JR34. Sounds good.
>>>>>>> 
>>>>>>> JR36. Could you document that the default value for
>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have been
>>>>>> changed?
>>>>>>> 
>>>>>>> JR38. My concern wasn't quite addressed. "If the configuration is
>>>>>> optional,
>>>>>>> we will reject any duplicate values in the list." This implies that
>> if
>>>> a
>>>>>>> config is optional, we allow empty lists. However, for plugin.path,
>> we
>>>>>>> reject empty lists.
>>>>>>> 
>>>>>>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed since
>>>>>> they
>>>>>>> are not public facing.
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hello Jun, Chia,
>>>>>>>> 
>>>>>>>> Thanks for the reply,
>>>>>>>> 
>>>>>>>> JR36: I’ve updated the following configurations:
>>>>>>>> - sasl.oauthbearer.expected.audience
>>>>>>>> - ssl.cipher.suites
>>>>>>>> - ssl.enabled.protocols
>>>>>>>> These now allow both empty and non-empty lists. Since the default
>>>> values
>>>>>>>> of
>>>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
>>>> previously
>>>>>>>> null,
>>>>>>>> I’ve updated their defaults to List.of() for consistency.
>>>>>>>> 
>>>>>>>> JR37: A ClassCastException does not occur in MirrorClientConfig and
>> it
>>>>>>>> should be none.
>>>>>>>> I’ve updated this in the KIP.
>>>>>>>> 
>>>>>>>> JR38 & JR40: I’ve added a note below the table to indicate which
>>>>>>>> configurations require special attention.
>>>>>>>> 
>>>>>>>> JR39: Yes, these properties are package-private.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
>>>>>>>>> 
>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
>>>>>> empty
>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>> maxTasks,
>>>>>>>>> Chia-Ping?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty
>> list,
>>>>>>>> which
>>>>>>>>> means no `MirrorSourceTask` can be created.
>>>>>>>>> 
>>>>>>>>> By contrast, if any `MirrorSourceTask` is created, the configs used
>>>> by
>>>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions` with a
>>>>>>>> non-empty
>>>>>>>>> value.
>>>>>>>>> 
>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply.
>>>>>>>>>> 
>>>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
>>>>>> empty
>>>>>>>>>> list for a task if knownSourceTopicPartitions is less than
>> maxTasks,
>>>>>>>>>> Chia-Ping?
>>>>>>>>>> 
>>>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should we do
>> the
>>>>>> same
>>>>>>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
>>>>>>>>>> 
>>>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
>>>>>>>>>> ClassCastException for null? It seems that null could be casted to
>>>> any
>>>>>>>>>> class.
>>>>>>>>>> 
>>>>>>>>>> JR38. "If the configuration is optional, we will reject any
>>>> duplicate
>>>>>>>>>> values in the list." Could you clarify this? For example,
>>>> plugin.path
>>>>>> is
>>>>>>>>>> optional, but with a default value. We reject duplicates as well
>> as
>>>>>>>> empty
>>>>>>>>>> lists.
>>>>>>>>>> 
>>>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields,
>> right?
>>>>>>>> They
>>>>>>>>>> are only public in the anyNonDuplicateValues() method.
>>>>>>>>>> 
>>>>>>>>>> JR40. It would be useful to mention that if cleanup.policy is
>> empty
>>>>>> and
>>>>>>>>>> remote.storage.enable is true, the local log segments will be
>>>> cleaned
>>>>>>>> based
>>>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
>> chia7712@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> JR34: For these two configurations, setting an empty list feels
>> a
>>>>>> bit
>>>>>>>>>>>> unintuitive. If an empty list is
>>>>>>>>>>>> provided, the consumer will call the unsubscribe method, which
>>>>>> doesn't
>>>>>>>>>>>> seem appropriate given
>>>>>>>>>>>> the documentation states: "Topic-partitions assigned to this
>> task
>>>> to
>>>>>>>>>>>> replicate."
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> agreed. Additionally, the source code [0] shows that it is not
>>>>>> intended
>>>>>>>>>> to
>>>>>>>>>>> generate empty tps for the source task
>>>>>>>>>>> 
>>>>>>>>>>> [0]
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply.

JR46. "Currently, if plugin.path is set to an empty list, no user-specified
plugin locations are processed, but plugins from the parent classloader are
still loaded." Could you add why the current behavior is bad for the users?

JR48. If we want to preserve the current behavior, we should remove the
following validation, right?
        if (validStrings.length == 0) {
            throw new IllegalArgumentException("Valid strings list cannot
be empty for in validator");

Jun

On Fri, Aug 1, 2025 at 6:47 PM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> Thanks for the reply.
>
> JR42, JR45, JR46, JR47: I have updated the KIP.
>
> JR43: If bootstrap.servers is empty for these configurations,
> it will fail when creating the AdminClient. I have updated the KIP to
> reflect this behavior.
>
> JR44: Thanks for pointing this out. Kafka will indeed fail at the storage
> format phase
> if the configuration is invalid. I have updated the KIP accordingly.
>
> JR48: The current in() method allows empty lists by default, so I believe
> setting isEmptyAllowed to true
> better preserves the original behavior. Therefore, I prefer not to change
> this default.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the reply. A few more comments.
> >
> > JR42. [6] [7] are not needed since currently an empty list already causes
> > an exception.
> >
> > JR43. Could you document the current behavior when bootstrap.servers is
> > empty in MirrorClientConfig and WorkerConfig?
> >
> > JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty and
> got
> > the following.
> > bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
> > config/server.properties
> > Exception in thread "main" java.lang.IllegalArgumentException:
> requirement
> > failed: At least one log directory must be defined via log.dirs or
> log.dir.
> >
> > bin/kafka-server-start.sh config/server.properties
> > [2025-08-01 11:31:20,097] INFO Registered
> > `kafka:type=kafka.Log4jController` MBean
> > (kafka.utils.Log4jControllerRegistration$)
> > [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal exception
> > (kafka.Kafka$)
> > java.lang.IllegalArgumentException: requirement failed: At least one log
> > directory must be defined via log.dirs or log.dir.
> >
> > JR45. Could you group all configs for WorkerConfig together?
> >
> > JR46. plugin.path: We have the following code. So, currently, it seems
> that
> > an empty list is allowed for plugin.path and we treat it by adding the
> > parent path of the classloader. It would be useful to justify why
> > disallowing empty is better for the users.
> >
> >    public static Set<PluginSource> pluginSources(Set<Path>
> > pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory
> factory)
> > {
> >        Set<PluginSource> pluginSources = new LinkedHashSet<>();
> >        for (Path pluginLocation : pluginLocations) {
> >            try {
> >                pluginSources.add(isolatedPluginSource(pluginLocation,
> > classLoader, factory));
> >            } catch (InvalidPathException | MalformedURLException e) {
> >                log.error("Invalid path in plugin path: {}. Ignoring.",
> > pluginLocation, e);
> >            } catch (IOException e) {
> >                log.error("Could not get listing for plugin path: {}.
> > Ignoring.", pluginLocation, e);
> >            }
> >        }
> >        pluginSources.add(classpathPluginSource(classLoader.getParent()));
> >        return pluginSources;
> >    }
> >
> > JR47. typo in "Should not allow setting validStrings is empty"
> >
> > JR48. We should pass in isEmptyAllowed as false in the code below, right?
> >    public static ValidList in(String... validStrings) {
> >        if (validStrings.length == 0) {
> >            throw new IllegalArgumentException("Valid strings list cannot
> > be empty for in validator");
> >        }
> >        return new ValidList(List.of(validStrings), true, false);
> >    }
> >
> > Jun
> >
> > On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello Jun,
> >>
> >> Sorry for the late reply,
> >>
> >> JR40, JR41: I have updated the KIP based on your comments. Please take a
> >> look.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
> >>>
> >>> Hi, Jiunn-Yang,
> >>>
> >>> Thanks for the reply.
> >>>
> >>> JR40. In the table, there are a few configs for which we now disallow
> >> empty
> >>> lists. It would be useful to document the current behavior for each of
> >> them
> >>> more accurately. It seems that they should all lead to some bad
> behavior.
> >>> For example, if log.dirs is empty, it seems that we can't create
> topics?
> >>> If group.coordinator.rebalance.protocols is empty, the group rebalance
> >>> protocol will fail?
> >>>
> >>> JR41. It would be useful to clean up the "Compatibility, Deprecation,
> and
> >>> Migration Plan" section a bit.
> >>> JR41.1 The migration plan has "Introduce a new properties
> isEmptyAllowed
> >> ,
> >>> isNullAllowed", which needs to be changed.
> >>> JR41.2 It started with "For users who configure an empty list", but not
> >> all
> >>> the bullet points are about empty lists.
> >>> JR41.3 To me, it would be useful to summarize why the behavior changes
> >> are
> >>> justified. We made 3 types of changes. (1) Disallow null. This is ok
> >> since
> >>> one can't really set a null value through a config file. (2) Disallow
> >>> duplicates. This is ok since it's rare and confusing. (3) Disallow
> empty.
> >>> Ideally, we want to claim that for those cases, they all lead to some
> >> other
> >>> errors or had behavior currently.
> >>>
> >>> Jun
> >>>
> >>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>
> >>>> Hello Jun,
> >>>>
> >>>> Thanks for the reply.
> >>>>
> >>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
> >>>>
> >>>> JR38: I clarified the following configurations:
> >>>> early.start.listeners: This config is optional and allows both null
> and
> >>>> empty lists.
> >>>> log.dirs and plugin.path: These are also optional, but only allow
> >>>> null—empty lists are not permitted.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> >>>>>
> >>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>
> >>>>> Thanks for the reply.
> >>>>>
> >>>>> JR34. Sounds good.
> >>>>>
> >>>>> JR36. Could you document that the default value for
> >>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have been
> >>>> changed?
> >>>>>
> >>>>> JR38. My concern wasn't quite addressed. "If the configuration is
> >>>> optional,
> >>>>> we will reject any duplicate values in the list." This implies that
> if
> >> a
> >>>>> config is optional, we allow empty lists. However, for plugin.path,
> we
> >>>>> reject empty lists.
> >>>>>
> >>>>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed since
> >>>> they
> >>>>> are not public facing.
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>>
> >>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>
> >>>>>> Hello Jun, Chia,
> >>>>>>
> >>>>>> Thanks for the reply,
> >>>>>>
> >>>>>> JR36: I’ve updated the following configurations:
> >>>>>> - sasl.oauthbearer.expected.audience
> >>>>>> - ssl.cipher.suites
> >>>>>> - ssl.enabled.protocols
> >>>>>> These now allow both empty and non-empty lists. Since the default
> >> values
> >>>>>> of
> >>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
> >> previously
> >>>>>> null,
> >>>>>> I’ve updated their defaults to List.of() for consistency.
> >>>>>>
> >>>>>> JR37: A ClassCastException does not occur in MirrorClientConfig and
> it
> >>>>>> should be none.
> >>>>>> I’ve updated this in the KIP.
> >>>>>>
> >>>>>> JR38 & JR40: I’ve added a note below the table to indicate which
> >>>>>> configurations require special attention.
> >>>>>>
> >>>>>> JR39: Yes, these properties are package-private.
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> >>>>>>>
> >>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
> >>>> empty
> >>>>>>> list for a task if knownSourceTopicPartitions is less than
> maxTasks,
> >>>>>>> Chia-Ping?
> >>>>>>>
> >>>>>>>
> >>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty
> list,
> >>>>>> which
> >>>>>>> means no `MirrorSourceTask` can be created.
> >>>>>>>
> >>>>>>> By contrast, if any `MirrorSourceTask` is created, the configs used
> >> by
> >>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions` with a
> >>>>>> non-empty
> >>>>>>> value.
> >>>>>>>
> >>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
> >>>>>>>
> >>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>>>>
> >>>>>>>> Thanks for the reply.
> >>>>>>>>
> >>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
> >>>> empty
> >>>>>>>> list for a task if knownSourceTopicPartitions is less than
> maxTasks,
> >>>>>>>> Chia-Ping?
> >>>>>>>>
> >>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should we do
> the
> >>>> same
> >>>>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
> >>>>>>>>
> >>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
> >>>>>>>> ClassCastException for null? It seems that null could be casted to
> >> any
> >>>>>>>> class.
> >>>>>>>>
> >>>>>>>> JR38. "If the configuration is optional, we will reject any
> >> duplicate
> >>>>>>>> values in the list." Could you clarify this? For example,
> >> plugin.path
> >>>> is
> >>>>>>>> optional, but with a default value. We reject duplicates as well
> as
> >>>>>> empty
> >>>>>>>> lists.
> >>>>>>>>
> >>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields,
> right?
> >>>>>> They
> >>>>>>>> are only public in the anyNonDuplicateValues() method.
> >>>>>>>>
> >>>>>>>> JR40. It would be useful to mention that if cleanup.policy is
> empty
> >>>> and
> >>>>>>>> remote.storage.enable is true, the local log segments will be
> >> cleaned
> >>>>>> based
> >>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <
> chia7712@gmail.com>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> JR34: For these two configurations, setting an empty list feels
> a
> >>>> bit
> >>>>>>>>>> unintuitive. If an empty list is
> >>>>>>>>>> provided, the consumer will call the unsubscribe method, which
> >>>> doesn't
> >>>>>>>>>> seem appropriate given
> >>>>>>>>>> the documentation states: "Topic-partitions assigned to this
> task
> >> to
> >>>>>>>>>> replicate."
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> agreed. Additionally, the source code [0] shows that it is not
> >>>> intended
> >>>>>>>> to
> >>>>>>>>> generate empty tps for the source task
> >>>>>>>>>
> >>>>>>>>> [0]
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

Thanks for the reply.

JR42, JR45, JR46, JR47: I have updated the KIP.

JR43: If bootstrap.servers is empty for these configurations, 
it will fail when creating the AdminClient. I have updated the KIP to reflect this behavior. 

JR44: Thanks for pointing this out. Kafka will indeed fail at the storage format phase 
if the configuration is invalid. I have updated the KIP accordingly. 

JR48: The current in() method allows empty lists by default, so I believe setting isEmptyAllowed to true 
better preserves the original behavior. Therefore, I prefer not to change this default.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年8月2日 凌晨2:38 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply. A few more comments.
> 
> JR42. [6] [7] are not needed since currently an empty list already causes
> an exception.
> 
> JR43. Could you document the current behavior when bootstrap.servers is
> empty in MirrorClientConfig and WorkerConfig?
> 
> JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty and got
> the following.
> bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
> config/server.properties
> Exception in thread "main" java.lang.IllegalArgumentException: requirement
> failed: At least one log directory must be defined via log.dirs or log.dir.
> 
> bin/kafka-server-start.sh config/server.properties
> [2025-08-01 11:31:20,097] INFO Registered
> `kafka:type=kafka.Log4jController` MBean
> (kafka.utils.Log4jControllerRegistration$)
> [2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal exception
> (kafka.Kafka$)
> java.lang.IllegalArgumentException: requirement failed: At least one log
> directory must be defined via log.dirs or log.dir.
> 
> JR45. Could you group all configs for WorkerConfig together?
> 
> JR46. plugin.path: We have the following code. So, currently, it seems that
> an empty list is allowed for plugin.path and we treat it by adding the
> parent path of the classloader. It would be useful to justify why
> disallowing empty is better for the users.
> 
>    public static Set<PluginSource> pluginSources(Set<Path>
> pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory factory)
> {
>        Set<PluginSource> pluginSources = new LinkedHashSet<>();
>        for (Path pluginLocation : pluginLocations) {
>            try {
>                pluginSources.add(isolatedPluginSource(pluginLocation,
> classLoader, factory));
>            } catch (InvalidPathException | MalformedURLException e) {
>                log.error("Invalid path in plugin path: {}. Ignoring.",
> pluginLocation, e);
>            } catch (IOException e) {
>                log.error("Could not get listing for plugin path: {}.
> Ignoring.", pluginLocation, e);
>            }
>        }
>        pluginSources.add(classpathPluginSource(classLoader.getParent()));
>        return pluginSources;
>    }
> 
> JR47. typo in "Should not allow setting validStrings is empty"
> 
> JR48. We should pass in isEmptyAllowed as false in the code below, right?
>    public static ValidList in(String... validStrings) {
>        if (validStrings.length == 0) {
>            throw new IllegalArgumentException("Valid strings list cannot
> be empty for in validator");
>        }
>        return new ValidList(List.of(validStrings), true, false);
>    }
> 
> Jun
> 
> On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun,
>> 
>> Sorry for the late reply,
>> 
>> JR40, JR41: I have updated the KIP based on your comments. Please take a
>> look.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
>>> 
>>> Hi, Jiunn-Yang,
>>> 
>>> Thanks for the reply.
>>> 
>>> JR40. In the table, there are a few configs for which we now disallow
>> empty
>>> lists. It would be useful to document the current behavior for each of
>> them
>>> more accurately. It seems that they should all lead to some bad behavior.
>>> For example, if log.dirs is empty, it seems that we can't create topics?
>>> If group.coordinator.rebalance.protocols is empty, the group rebalance
>>> protocol will fail?
>>> 
>>> JR41. It would be useful to clean up the "Compatibility, Deprecation, and
>>> Migration Plan" section a bit.
>>> JR41.1 The migration plan has "Introduce a new properties isEmptyAllowed
>> ,
>>> isNullAllowed", which needs to be changed.
>>> JR41.2 It started with "For users who configure an empty list", but not
>> all
>>> the bullet points are about empty lists.
>>> JR41.3 To me, it would be useful to summarize why the behavior changes
>> are
>>> justified. We made 3 types of changes. (1) Disallow null. This is ok
>> since
>>> one can't really set a null value through a config file. (2) Disallow
>>> duplicates. This is ok since it's rare and confusing. (3) Disallow empty.
>>> Ideally, we want to claim that for those cases, they all lead to some
>> other
>>> errors or had behavior currently.
>>> 
>>> Jun
>>> 
>>> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hello Jun,
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
>>>> 
>>>> JR38: I clarified the following configurations:
>>>> early.start.listeners: This config is optional and allows both null and
>>>> empty lists.
>>>> log.dirs and plugin.path: These are also optional, but only allow
>>>> null—empty lists are not permitted.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
>>>>> 
>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> JR34. Sounds good.
>>>>> 
>>>>> JR36. Could you document that the default value for
>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have been
>>>> changed?
>>>>> 
>>>>> JR38. My concern wasn't quite addressed. "If the configuration is
>>>> optional,
>>>>> we will reject any duplicate values in the list." This implies that if
>> a
>>>>> config is optional, we allow empty lists. However, for plugin.path, we
>>>>> reject empty lists.
>>>>> 
>>>>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed since
>>>> they
>>>>> are not public facing.
>>>>> 
>>>>> Jun
>>>>> 
>>>>> 
>>>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hello Jun, Chia,
>>>>>> 
>>>>>> Thanks for the reply,
>>>>>> 
>>>>>> JR36: I’ve updated the following configurations:
>>>>>> - sasl.oauthbearer.expected.audience
>>>>>> - ssl.cipher.suites
>>>>>> - ssl.enabled.protocols
>>>>>> These now allow both empty and non-empty lists. Since the default
>> values
>>>>>> of
>>>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
>> previously
>>>>>> null,
>>>>>> I’ve updated their defaults to List.of() for consistency.
>>>>>> 
>>>>>> JR37: A ClassCastException does not occur in MirrorClientConfig and it
>>>>>> should be none.
>>>>>> I’ve updated this in the KIP.
>>>>>> 
>>>>>> JR38 & JR40: I’ve added a note below the table to indicate which
>>>>>> configurations require special attention.
>>>>>> 
>>>>>> JR39: Yes, these properties are package-private.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
>>>>>>> 
>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
>>>> empty
>>>>>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
>>>>>>> Chia-Ping?
>>>>>>> 
>>>>>>> 
>>>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty list,
>>>>>> which
>>>>>>> means no `MirrorSourceTask` can be created.
>>>>>>> 
>>>>>>> By contrast, if any `MirrorSourceTask` is created, the configs used
>> by
>>>>>>> `MirrorSourceTask` must contain `task.assigned.partitions` with a
>>>>>> non-empty
>>>>>>> value.
>>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
>>>> empty
>>>>>>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
>>>>>>>> Chia-Ping?
>>>>>>>> 
>>>>>>>> JR36. Sounds good. Could you update the KIP? Also, should we do the
>>>> same
>>>>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
>>>>>>>> 
>>>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
>>>>>>>> ClassCastException for null? It seems that null could be casted to
>> any
>>>>>>>> class.
>>>>>>>> 
>>>>>>>> JR38. "If the configuration is optional, we will reject any
>> duplicate
>>>>>>>> values in the list." Could you clarify this? For example,
>> plugin.path
>>>> is
>>>>>>>> optional, but with a default value. We reject duplicates as well as
>>>>>> empty
>>>>>>>> lists.
>>>>>>>> 
>>>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields, right?
>>>>>> They
>>>>>>>> are only public in the anyNonDuplicateValues() method.
>>>>>>>> 
>>>>>>>> JR40. It would be useful to mention that if cleanup.policy is empty
>>>> and
>>>>>>>> remote.storage.enable is true, the local log segments will be
>> cleaned
>>>>>> based
>>>>>>>> on log.local.retention.bytes and log.local.retention.ms.
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> JR34: For these two configurations, setting an empty list feels a
>>>> bit
>>>>>>>>>> unintuitive. If an empty list is
>>>>>>>>>> provided, the consumer will call the unsubscribe method, which
>>>> doesn't
>>>>>>>>>> seem appropriate given
>>>>>>>>>> the documentation states: "Topic-partitions assigned to this task
>> to
>>>>>>>>>> replicate."
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> agreed. Additionally, the source code [0] shows that it is not
>>>> intended
>>>>>>>> to
>>>>>>>>> generate empty tps for the source task
>>>>>>>>> 
>>>>>>>>> [0]
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply. A few more comments.

JR42. [6] [7] are not needed since currently an empty list already causes
an exception.

JR43. Could you document the current behavior when bootstrap.servers is
empty in MirrorClientConfig and WorkerConfig?

JR44. [12] seems incorrect. I set log.dir, but left log.dirs empty and got
the following.
 bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c
config/server.properties
Exception in thread "main" java.lang.IllegalArgumentException: requirement
failed: At least one log directory must be defined via log.dirs or log.dir.

bin/kafka-server-start.sh config/server.properties
[2025-08-01 11:31:20,097] INFO Registered
`kafka:type=kafka.Log4jController` MBean
(kafka.utils.Log4jControllerRegistration$)
[2025-08-01 11:31:20,238] ERROR Exiting Kafka due to fatal exception
(kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: At least one log
directory must be defined via log.dirs or log.dir.

JR45. Could you group all configs for WorkerConfig together?

JR46. plugin.path: We have the following code. So, currently, it seems that
an empty list is allowed for plugin.path and we treat it by adding the
parent path of the classloader. It would be useful to justify why
disallowing empty is better for the users.

    public static Set<PluginSource> pluginSources(Set<Path>
pluginLocations, ClassLoader classLoader, PluginClassLoaderFactory factory)
{
        Set<PluginSource> pluginSources = new LinkedHashSet<>();
        for (Path pluginLocation : pluginLocations) {
            try {
                pluginSources.add(isolatedPluginSource(pluginLocation,
classLoader, factory));
            } catch (InvalidPathException | MalformedURLException e) {
                log.error("Invalid path in plugin path: {}. Ignoring.",
pluginLocation, e);
            } catch (IOException e) {
                log.error("Could not get listing for plugin path: {}.
Ignoring.", pluginLocation, e);
            }
        }
        pluginSources.add(classpathPluginSource(classLoader.getParent()));
        return pluginSources;
    }

JR47. typo in "Should not allow setting validStrings is empty"

JR48. We should pass in isEmptyAllowed as false in the code below, right?
    public static ValidList in(String... validStrings) {
        if (validStrings.length == 0) {
            throw new IllegalArgumentException("Valid strings list cannot
be empty for in validator");
        }
        return new ValidList(List.of(validStrings), true, false);
    }

Jun

On Tue, Jul 29, 2025 at 8:53 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> Sorry for the late reply,
>
> JR40, JR41: I have updated the KIP based on your comments. Please take a
> look.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the reply.
> >
> > JR40. In the table, there are a few configs for which we now disallow
> empty
> > lists. It would be useful to document the current behavior for each of
> them
> > more accurately. It seems that they should all lead to some bad behavior.
> > For example, if log.dirs is empty, it seems that we can't create topics?
> > If group.coordinator.rebalance.protocols is empty, the group rebalance
> > protocol will fail?
> >
> > JR41. It would be useful to clean up the "Compatibility, Deprecation, and
> > Migration Plan" section a bit.
> > JR41.1 The migration plan has "Introduce a new properties isEmptyAllowed
> ,
> > isNullAllowed", which needs to be changed.
> > JR41.2 It started with "For users who configure an empty list", but not
> all
> > the bullet points are about empty lists.
> > JR41.3 To me, it would be useful to summarize why the behavior changes
> are
> > justified. We made 3 types of changes. (1) Disallow null. This is ok
> since
> > one can't really set a null value through a config file. (2) Disallow
> > duplicates. This is ok since it's rare and confusing. (3) Disallow empty.
> > Ideally, we want to claim that for those cases, they all lead to some
> other
> > errors or had behavior currently.
> >
> > Jun
> >
> > On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello Jun,
> >>
> >> Thanks for the reply.
> >>
> >> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
> >>
> >> JR38: I clarified the following configurations:
> >> early.start.listeners: This config is optional and allows both null and
> >> empty lists.
> >> log.dirs and plugin.path: These are also optional, but only allow
> >> null—empty lists are not permitted.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> >>>
> >>> Hi, Jiunn-Yang and Chia-Ping,
> >>>
> >>> Thanks for the reply.
> >>>
> >>> JR34. Sounds good.
> >>>
> >>> JR36. Could you document that the default value for
> >>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have been
> >> changed?
> >>>
> >>> JR38. My concern wasn't quite addressed. "If the configuration is
> >> optional,
> >>> we will reject any duplicate values in the list." This implies that if
> a
> >>> config is optional, we allow empty lists. However, for plugin.path, we
> >>> reject empty lists.
> >>>
> >>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed since
> >> they
> >>> are not public facing.
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>
> >>>> Hello Jun, Chia,
> >>>>
> >>>> Thanks for the reply,
> >>>>
> >>>> JR36: I’ve updated the following configurations:
> >>>> - sasl.oauthbearer.expected.audience
> >>>> - ssl.cipher.suites
> >>>> - ssl.enabled.protocols
> >>>> These now allow both empty and non-empty lists. Since the default
> values
> >>>> of
> >>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were
> previously
> >>>> null,
> >>>> I’ve updated their defaults to List.of() for consistency.
> >>>>
> >>>> JR37: A ClassCastException does not occur in MirrorClientConfig and it
> >>>> should be none.
> >>>> I’ve updated this in the KIP.
> >>>>
> >>>> JR38 & JR40: I’ve added a note below the table to indicate which
> >>>> configurations require special attention.
> >>>>
> >>>> JR39: Yes, these properties are package-private.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> >>>>>
> >>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
> >> empty
> >>>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
> >>>>> Chia-Ping?
> >>>>>
> >>>>>
> >>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty list,
> >>>> which
> >>>>> means no `MirrorSourceTask` can be created.
> >>>>>
> >>>>> By contrast, if any `MirrorSourceTask` is created, the configs used
> by
> >>>>> `MirrorSourceTask` must contain `task.assigned.partitions` with a
> >>>> non-empty
> >>>>> value.
> >>>>>
> >>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
> >>>>>
> >>>>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>>>
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
> >> empty
> >>>>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
> >>>>>> Chia-Ping?
> >>>>>>
> >>>>>> JR36. Sounds good. Could you update the KIP? Also, should we do the
> >> same
> >>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
> >>>>>>
> >>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
> >>>>>> ClassCastException for null? It seems that null could be casted to
> any
> >>>>>> class.
> >>>>>>
> >>>>>> JR38. "If the configuration is optional, we will reject any
> duplicate
> >>>>>> values in the list." Could you clarify this? For example,
> plugin.path
> >> is
> >>>>>> optional, but with a default value. We reject duplicates as well as
> >>>> empty
> >>>>>> lists.
> >>>>>>
> >>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields, right?
> >>>> They
> >>>>>> are only public in the anyNonDuplicateValues() method.
> >>>>>>
> >>>>>> JR40. It would be useful to mention that if cleanup.policy is empty
> >> and
> >>>>>> remote.storage.enable is true, the local log segments will be
> cleaned
> >>>> based
> >>>>>> on log.local.retention.bytes and log.local.retention.ms.
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>>>
> >>>>>>>> JR34: For these two configurations, setting an empty list feels a
> >> bit
> >>>>>>>> unintuitive. If an empty list is
> >>>>>>>> provided, the consumer will call the unsubscribe method, which
> >> doesn't
> >>>>>>>> seem appropriate given
> >>>>>>>> the documentation states: "Topic-partitions assigned to this task
> to
> >>>>>>>> replicate."
> >>>>>>>>
> >>>>>>>
> >>>>>>> agreed. Additionally, the source code [0] shows that it is not
> >> intended
> >>>>>> to
> >>>>>>> generate empty tps for the source task
> >>>>>>>
> >>>>>>> [0]
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

Sorry for the late reply,

JR40, JR41: I have updated the KIP based on your comments. Please take a look.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月24日 凌晨2:56 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply.
> 
> JR40. In the table, there are a few configs for which we now disallow empty
> lists. It would be useful to document the current behavior for each of them
> more accurately. It seems that they should all lead to some bad behavior.
> For example, if log.dirs is empty, it seems that we can't create topics?
> If group.coordinator.rebalance.protocols is empty, the group rebalance
> protocol will fail?
> 
> JR41. It would be useful to clean up the "Compatibility, Deprecation, and
> Migration Plan" section a bit.
> JR41.1 The migration plan has "Introduce a new properties isEmptyAllowed ,
> isNullAllowed", which needs to be changed.
> JR41.2 It started with "For users who configure an empty list", but not all
> the bullet points are about empty lists.
> JR41.3 To me, it would be useful to summarize why the behavior changes are
> justified. We made 3 types of changes. (1) Disallow null. This is ok since
> one can't really set a null value through a config file. (2) Disallow
> duplicates. This is ok since it's rare and confusing. (3) Disallow empty.
> Ideally, we want to claim that for those cases, they all lead to some other
> errors or had behavior currently.
> 
> Jun
> 
> On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun,
>> 
>> Thanks for the reply.
>> 
>> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
>> 
>> JR38: I clarified the following configurations:
>> early.start.listeners: This config is optional and allows both null and
>> empty lists.
>> log.dirs and plugin.path: These are also optional, but only allow
>> null—empty lists are not permitted.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
>>> 
>>> Hi, Jiunn-Yang and Chia-Ping,
>>> 
>>> Thanks for the reply.
>>> 
>>> JR34. Sounds good.
>>> 
>>> JR36. Could you document that the default value for
>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites have been
>> changed?
>>> 
>>> JR38. My concern wasn't quite addressed. "If the configuration is
>> optional,
>>> we will reject any duplicate values in the list." This implies that if a
>>> config is optional, we allow empty lists. However, for plugin.path, we
>>> reject empty lists.
>>> 
>>> JR39. We can remove the fields isEmptyAllowed and isNullAllowed since
>> they
>>> are not public facing.
>>> 
>>> Jun
>>> 
>>> 
>>> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hello Jun, Chia,
>>>> 
>>>> Thanks for the reply,
>>>> 
>>>> JR36: I’ve updated the following configurations:
>>>> - sasl.oauthbearer.expected.audience
>>>> - ssl.cipher.suites
>>>> - ssl.enabled.protocols
>>>> These now allow both empty and non-empty lists. Since the default values
>>>> of
>>>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were previously
>>>> null,
>>>> I’ve updated their defaults to List.of() for consistency.
>>>> 
>>>> JR37: A ClassCastException does not occur in MirrorClientConfig and it
>>>> should be none.
>>>> I’ve updated this in the KIP.
>>>> 
>>>> JR38 & JR40: I’ve added a note below the table to indicate which
>>>> configurations require special attention.
>>>> 
>>>> JR39: Yes, these properties are package-private.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
>>>>> 
>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
>> empty
>>>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
>>>>> Chia-Ping?
>>>>> 
>>>>> 
>>>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty list,
>>>> which
>>>>> means no `MirrorSourceTask` can be created.
>>>>> 
>>>>> By contrast, if any `MirrorSourceTask` is created, the configs used by
>>>>> `MirrorSourceTask` must contain `task.assigned.partitions` with a
>>>> non-empty
>>>>> value.
>>>>> 
>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
>>>>> 
>>>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>>>> 
>>>>>> Thanks for the reply.
>>>>>> 
>>>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
>> empty
>>>>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
>>>>>> Chia-Ping?
>>>>>> 
>>>>>> JR36. Sounds good. Could you update the KIP? Also, should we do the
>> same
>>>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
>>>>>> 
>>>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
>>>>>> ClassCastException for null? It seems that null could be casted to any
>>>>>> class.
>>>>>> 
>>>>>> JR38. "If the configuration is optional, we will reject any duplicate
>>>>>> values in the list." Could you clarify this? For example, plugin.path
>> is
>>>>>> optional, but with a default value. We reject duplicates as well as
>>>> empty
>>>>>> lists.
>>>>>> 
>>>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields, right?
>>>> They
>>>>>> are only public in the anyNonDuplicateValues() method.
>>>>>> 
>>>>>> JR40. It would be useful to mention that if cleanup.policy is empty
>> and
>>>>>> remote.storage.enable is true, the local log segments will be cleaned
>>>> based
>>>>>> on log.local.retention.bytes and log.local.retention.ms.
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com>
>>>> wrote:
>>>>>> 
>>>>>>>> 
>>>>>>>> JR34: For these two configurations, setting an empty list feels a
>> bit
>>>>>>>> unintuitive. If an empty list is
>>>>>>>> provided, the consumer will call the unsubscribe method, which
>> doesn't
>>>>>>>> seem appropriate given
>>>>>>>> the documentation states: "Topic-partitions assigned to this task to
>>>>>>>> replicate."
>>>>>>>> 
>>>>>>> 
>>>>>>> agreed. Additionally, the source code [0] shows that it is not
>> intended
>>>>>> to
>>>>>>> generate empty tps for the source task
>>>>>>> 
>>>>>>> [0]
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply.

JR40. In the table, there are a few configs for which we now disallow empty
lists. It would be useful to document the current behavior for each of them
more accurately. It seems that they should all lead to some bad behavior.
For example, if log.dirs is empty, it seems that we can't create topics?
If group.coordinator.rebalance.protocols is empty, the group rebalance
protocol will fail?

JR41. It would be useful to clean up the "Compatibility, Deprecation, and
Migration Plan" section a bit.
JR41.1 The migration plan has "Introduce a new properties isEmptyAllowed ,
isNullAllowed", which needs to be changed.
JR41.2 It started with "For users who configure an empty list", but not all
the bullet points are about empty lists.
JR41.3 To me, it would be useful to summarize why the behavior changes are
justified. We made 3 types of changes. (1) Disallow null. This is ok since
one can't really set a null value through a config file. (2) Disallow
duplicates. This is ok since it's rare and confusing. (3) Disallow empty.
Ideally, we want to claim that for those cases, they all lead to some other
errors or had behavior currently.

Jun

On Wed, Jul 23, 2025 at 4:42 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> Thanks for the reply.
>
> JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes.
>
> JR38: I clarified the following configurations:
> early.start.listeners: This config is optional and allows both null and
> empty lists.
> log.dirs and plugin.path: These are also optional, but only allow
> null—empty lists are not permitted.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> >
> > Hi, Jiunn-Yang and Chia-Ping,
> >
> > Thanks for the reply.
> >
> > JR34. Sounds good.
> >
> > JR36. Could you document that the default value for
> > sasl.oauthbearer.expected.audience and ssl.cipher.suites have been
> changed?
> >
> > JR38. My concern wasn't quite addressed. "If the configuration is
> optional,
> > we will reject any duplicate values in the list." This implies that if a
> > config is optional, we allow empty lists. However, for plugin.path, we
> > reject empty lists.
> >
> > JR39. We can remove the fields isEmptyAllowed and isNullAllowed since
> they
> > are not public facing.
> >
> > Jun
> >
> >
> > On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello Jun, Chia,
> >>
> >> Thanks for the reply,
> >>
> >> JR36: I’ve updated the following configurations:
> >> - sasl.oauthbearer.expected.audience
> >> - ssl.cipher.suites
> >> - ssl.enabled.protocols
> >> These now allow both empty and non-empty lists. Since the default values
> >> of
> >> sasl.oauthbearer.expected.audience and ssl.cipher.suites were previously
> >> null,
> >> I’ve updated their defaults to List.of() for consistency.
> >>
> >> JR37: A ClassCastException does not occur in MirrorClientConfig and it
> >> should be none.
> >> I’ve updated this in the KIP.
> >>
> >> JR38 & JR40: I’ve added a note below the table to indicate which
> >> configurations require special attention.
> >>
> >> JR39: Yes, these properties are package-private.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> >>>
> >>> JR34. Hmm, it seems that the code in taskConfigs() could return an
> empty
> >>> list for a task if knownSourceTopicPartitions is less than maxTasks,
> >>> Chia-Ping?
> >>>
> >>>
> >>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty list,
> >> which
> >>> means no `MirrorSourceTask` can be created.
> >>>
> >>> By contrast, if any `MirrorSourceTask` is created, the configs used by
> >>> `MirrorSourceTask` must contain `task.assigned.partitions` with a
> >> non-empty
> >>> value.
> >>>
> >>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
> >>>
> >>>> Hi, Jiunn-Yang and Chia-Ping,
> >>>>
> >>>> Thanks for the reply.
> >>>>
> >>>> JR34. Hmm, it seems that the code in taskConfigs() could return an
> empty
> >>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
> >>>> Chia-Ping?
> >>>>
> >>>> JR36. Sounds good. Could you update the KIP? Also, should we do the
> same
> >>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
> >>>>
> >>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
> >>>> ClassCastException for null? It seems that null could be casted to any
> >>>> class.
> >>>>
> >>>> JR38. "If the configuration is optional, we will reject any duplicate
> >>>> values in the list." Could you clarify this? For example, plugin.path
> is
> >>>> optional, but with a default value. We reject duplicates as well as
> >> empty
> >>>> lists.
> >>>>
> >>>> JR39. isEmptyAllowed and isNullAllowed are not public fields, right?
> >> They
> >>>> are only public in the anyNonDuplicateValues() method.
> >>>>
> >>>> JR40. It would be useful to mention that if cleanup.policy is empty
> and
> >>>> remote.storage.enable is true, the local log segments will be cleaned
> >> based
> >>>> on log.local.retention.bytes and log.local.retention.ms.
> >>>>
> >>>> Jun
> >>>>
> >>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com>
> >> wrote:
> >>>>
> >>>>>>
> >>>>>> JR34: For these two configurations, setting an empty list feels a
> bit
> >>>>>> unintuitive. If an empty list is
> >>>>>> provided, the consumer will call the unsubscribe method, which
> doesn't
> >>>>>> seem appropriate given
> >>>>>> the documentation states: "Topic-partitions assigned to this task to
> >>>>>> replicate."
> >>>>>>
> >>>>>
> >>>>> agreed. Additionally, the source code [0] shows that it is not
> intended
> >>>> to
> >>>>> generate empty tps for the source task
> >>>>>
> >>>>> [0]
> >>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> >>>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

Thanks for the reply.

JR34 ,JR36, JR39: I’ve updated the KIP to reflect these changes. 

JR38: I clarified the following configurations: 
early.start.listeners: This config is optional and allows both null and empty lists. 
log.dirs and plugin.path: These are also optional, but only allow null—empty lists are not permitted.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月23日 凌晨2:28 寫道：
> 
> Hi, Jiunn-Yang and Chia-Ping,
> 
> Thanks for the reply.
> 
> JR34. Sounds good.
> 
> JR36. Could you document that the default value for
> sasl.oauthbearer.expected.audience and ssl.cipher.suites have been changed?
> 
> JR38. My concern wasn't quite addressed. "If the configuration is optional,
> we will reject any duplicate values in the list." This implies that if a
> config is optional, we allow empty lists. However, for plugin.path, we
> reject empty lists.
> 
> JR39. We can remove the fields isEmptyAllowed and isNullAllowed since they
> are not public facing.
> 
> Jun
> 
> 
> On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun, Chia,
>> 
>> Thanks for the reply,
>> 
>> JR36: I’ve updated the following configurations:
>> - sasl.oauthbearer.expected.audience
>> - ssl.cipher.suites
>> - ssl.enabled.protocols
>> These now allow both empty and non-empty lists. Since the default values
>> of
>> sasl.oauthbearer.expected.audience and ssl.cipher.suites were previously
>> null,
>> I’ve updated their defaults to List.of() for consistency.
>> 
>> JR37: A ClassCastException does not occur in MirrorClientConfig and it
>> should be none.
>> I’ve updated this in the KIP.
>> 
>> JR38 & JR40: I’ve added a note below the table to indicate which
>> configurations require special attention.
>> 
>> JR39: Yes, these properties are package-private.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
>>> 
>>> JR34. Hmm, it seems that the code in taskConfigs() could return an empty
>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
>>> Chia-Ping?
>>> 
>>> 
>>> Yes, `MirrorSourceConnector#taskConfigs` could return an empty list,
>> which
>>> means no `MirrorSourceTask` can be created.
>>> 
>>> By contrast, if any `MirrorSourceTask` is created, the configs used by
>>> `MirrorSourceTask` must contain `task.assigned.partitions` with a
>> non-empty
>>> value.
>>> 
>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
>>> 
>>>> Hi, Jiunn-Yang and Chia-Ping,
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> JR34. Hmm, it seems that the code in taskConfigs() could return an empty
>>>> list for a task if knownSourceTopicPartitions is less than maxTasks,
>>>> Chia-Ping?
>>>> 
>>>> JR36. Sounds good. Could you update the KIP? Also, should we do the same
>>>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
>>>> 
>>>> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
>>>> ClassCastException for null? It seems that null could be casted to any
>>>> class.
>>>> 
>>>> JR38. "If the configuration is optional, we will reject any duplicate
>>>> values in the list." Could you clarify this? For example, plugin.path is
>>>> optional, but with a default value. We reject duplicates as well as
>> empty
>>>> lists.
>>>> 
>>>> JR39. isEmptyAllowed and isNullAllowed are not public fields, right?
>> They
>>>> are only public in the anyNonDuplicateValues() method.
>>>> 
>>>> JR40. It would be useful to mention that if cleanup.policy is empty and
>>>> remote.storage.enable is true, the local log segments will be cleaned
>> based
>>>> on log.local.retention.bytes and log.local.retention.ms.
>>>> 
>>>> Jun
>>>> 
>>>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com>
>> wrote:
>>>> 
>>>>>> 
>>>>>> JR34: For these two configurations, setting an empty list feels a bit
>>>>>> unintuitive. If an empty list is
>>>>>> provided, the consumer will call the unsubscribe method, which doesn't
>>>>>> seem appropriate given
>>>>>> the documentation states: "Topic-partitions assigned to this task to
>>>>>> replicate."
>>>>>> 
>>>>> 
>>>>> agreed. Additionally, the source code [0] shows that it is not intended
>>>> to
>>>>> generate empty tps for the source task
>>>>> 
>>>>> [0]
>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang and Chia-Ping,

Thanks for the reply.

JR34. Sounds good.

JR36. Could you document that the default value for
sasl.oauthbearer.expected.audience and ssl.cipher.suites have been changed?

JR38. My concern wasn't quite addressed. "If the configuration is optional,
we will reject any duplicate values in the list." This implies that if a
config is optional, we allow empty lists. However, for plugin.path, we
reject empty lists.

JR39. We can remove the fields isEmptyAllowed and isNullAllowed since they
are not public facing.

Jun


On Tue, Jul 22, 2025 at 4:26 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun, Chia,
>
> Thanks for the reply,
>
> JR36: I’ve updated the following configurations:
> - sasl.oauthbearer.expected.audience
> - ssl.cipher.suites
> - ssl.enabled.protocols
> These now allow both empty and non-empty lists. Since the default values
> of
> sasl.oauthbearer.expected.audience and ssl.cipher.suites were previously
> null,
> I’ve updated their defaults to List.of() for consistency.
>
> JR37: A ClassCastException does not occur in MirrorClientConfig and it
> should be none.
> I’ve updated this in the KIP.
>
> JR38 & JR40: I’ve added a note below the table to indicate which
> configurations require special attention.
>
> JR39: Yes, these properties are package-private.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> >
> > JR34. Hmm, it seems that the code in taskConfigs() could return an empty
> > list for a task if knownSourceTopicPartitions is less than maxTasks,
> > Chia-Ping?
> >
> >
> > Yes, `MirrorSourceConnector#taskConfigs` could return an empty list,
> which
> > means no `MirrorSourceTask` can be created.
> >
> > By contrast, if any `MirrorSourceTask` is created, the configs used by
> > `MirrorSourceTask` must contain `task.assigned.partitions` with a
> non-empty
> > value.
> >
> > Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
> >
> >> Hi, Jiunn-Yang and Chia-Ping,
> >>
> >> Thanks for the reply.
> >>
> >> JR34. Hmm, it seems that the code in taskConfigs() could return an empty
> >> list for a task if knownSourceTopicPartitions is less than maxTasks,
> >> Chia-Ping?
> >>
> >> JR36. Sounds good. Could you update the KIP? Also, should we do the same
> >> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
> >>
> >> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
> >> ClassCastException for null? It seems that null could be casted to any
> >> class.
> >>
> >> JR38. "If the configuration is optional, we will reject any duplicate
> >> values in the list." Could you clarify this? For example, plugin.path is
> >> optional, but with a default value. We reject duplicates as well as
> empty
> >> lists.
> >>
> >> JR39. isEmptyAllowed and isNullAllowed are not public fields, right?
> They
> >> are only public in the anyNonDuplicateValues() method.
> >>
> >> JR40. It would be useful to mention that if cleanup.policy is empty and
> >> remote.storage.enable is true, the local log segments will be cleaned
> based
> >> on log.local.retention.bytes and log.local.retention.ms.
> >>
> >> Jun
> >>
> >> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >>
> >>>>
> >>>> JR34: For these two configurations, setting an empty list feels a bit
> >>>> unintuitive. If an empty list is
> >>>> provided, the consumer will call the unsubscribe method, which doesn't
> >>>> seem appropriate given
> >>>> the documentation states: "Topic-partitions assigned to this task to
> >>>> replicate."
> >>>>
> >>>
> >>> agreed. Additionally, the source code [0] shows that it is not intended
> >> to
> >>> generate empty tps for the source task
> >>>
> >>> [0]
> >>>
> >>>
> >>
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun, Chia,

Thanks for the reply,

JR36: I’ve updated the following configurations: 
- sasl.oauthbearer.expected.audience 
- ssl.cipher.suites 
- ssl.enabled.protocols 
These now allow both empty and non-empty lists. Since the default values of 
sasl.oauthbearer.expected.audience and ssl.cipher.suites were previously null, 
I’ve updated their defaults to List.of() for consistency. 

JR37: A ClassCastException does not occur in MirrorClientConfig and it should be none. 
I’ve updated this in the KIP.

JR38 & JR40: I’ve added a note below the table to indicate which configurations require special attention. 

JR39: Yes, these properties are package-private.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月22日 下午6:04 寫道：
> 
> JR34. Hmm, it seems that the code in taskConfigs() could return an empty
> list for a task if knownSourceTopicPartitions is less than maxTasks,
> Chia-Ping?
> 
> 
> Yes, `MirrorSourceConnector#taskConfigs` could return an empty list, which
> means no `MirrorSourceTask` can be created.
> 
> By contrast, if any `MirrorSourceTask` is created, the configs used by
> `MirrorSourceTask` must contain `task.assigned.partitions` with a non-empty
> value.
> 
> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：
> 
>> Hi, Jiunn-Yang and Chia-Ping,
>> 
>> Thanks for the reply.
>> 
>> JR34. Hmm, it seems that the code in taskConfigs() could return an empty
>> list for a task if knownSourceTopicPartitions is less than maxTasks,
>> Chia-Ping?
>> 
>> JR36. Sounds good. Could you update the KIP? Also, should we do the same
>> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
>> 
>> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
>> ClassCastException for null? It seems that null could be casted to any
>> class.
>> 
>> JR38. "If the configuration is optional, we will reject any duplicate
>> values in the list." Could you clarify this? For example, plugin.path is
>> optional, but with a default value. We reject duplicates as well as empty
>> lists.
>> 
>> JR39. isEmptyAllowed and isNullAllowed are not public fields, right? They
>> are only public in the anyNonDuplicateValues() method.
>> 
>> JR40. It would be useful to mention that if cleanup.policy is empty and
>> remote.storage.enable is true, the local log segments will be cleaned based
>> on log.local.retention.bytes and log.local.retention.ms.
>> 
>> Jun
>> 
>> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>> 
>>>> 
>>>> JR34: For these two configurations, setting an empty list feels a bit
>>>> unintuitive. If an empty list is
>>>> provided, the consumer will call the unsubscribe method, which doesn't
>>>> seem appropriate given
>>>> the documentation states: "Topic-partitions assigned to this task to
>>>> replicate."
>>>> 
>>> 
>>> agreed. Additionally, the source code [0] shows that it is not intended
>> to
>>> generate empty tps for the source task
>>> 
>>> [0]
>>> 
>>> 
>> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

JR34. Hmm, it seems that the code in taskConfigs() could return an empty
list for a task if knownSourceTopicPartitions is less than maxTasks,
Chia-Ping?


Yes, `MirrorSourceConnector#taskConfigs` could return an empty list, which
means no `MirrorSourceTask` can be created.

By contrast, if any `MirrorSourceTask` is created, the configs used by
`MirrorSourceTask` must contain `task.assigned.partitions` with a non-empty
value.

Jun Rao <ju...@confluent.io.invalid> 於 2025年7月22日 週二 上午5:14寫道：

> Hi, Jiunn-Yang and Chia-Ping,
>
> Thanks for the reply.
>
> JR34. Hmm, it seems that the code in taskConfigs() could return an empty
> list for a task if knownSourceTopicPartitions is less than maxTasks,
> Chia-Ping?
>
> JR36. Sounds good. Could you update the KIP? Also, should we do the same
> for ssl.cipher.suites and sasl.oauthbearer.expected.audience?
>
> JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
> ClassCastException for null? It seems that null could be casted to any
> class.
>
> JR38. "If the configuration is optional, we will reject any duplicate
> values in the list." Could you clarify this? For example, plugin.path is
> optional, but with a default value. We reject duplicates as well as empty
> lists.
>
> JR39. isEmptyAllowed and isNullAllowed are not public fields, right? They
> are only public in the anyNonDuplicateValues() method.
>
> JR40. It would be useful to mention that if cleanup.policy is empty and
> remote.storage.enable is true, the local log segments will be cleaned based
> on log.local.retention.bytes and log.local.retention.ms.
>
> Jun
>
> On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>
> > >
> > > JR34: For these two configurations, setting an empty list feels a bit
> > > unintuitive. If an empty list is
> > > provided, the consumer will call the unsubscribe method, which doesn't
> > > seem appropriate given
> > > the documentation states: "Topic-partitions assigned to this task to
> > > replicate."
> > >
> >
> > agreed. Additionally, the source code [0] shows that it is not intended
> to
> > generate empty tps for the source task
> >
> > [0]
> >
> >
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang and Chia-Ping,

Thanks for the reply.

JR34. Hmm, it seems that the code in taskConfigs() could return an empty
list for a task if knownSourceTopicPartitions is less than maxTasks,
Chia-Ping?

JR36. Sounds good. Could you update the KIP? Also, should we do the same
for ssl.cipher.suites and sasl.oauthbearer.expected.audience?

JR37. Is it true that MirrorClientConfig.bootstrap.servers returns
ClassCastException for null? It seems that null could be casted to any
class.

JR38. "If the configuration is optional, we will reject any duplicate
values in the list." Could you clarify this? For example, plugin.path is
optional, but with a default value. We reject duplicates as well as empty
lists.

JR39. isEmptyAllowed and isNullAllowed are not public fields, right? They
are only public in the anyNonDuplicateValues() method.

JR40. It would be useful to mention that if cleanup.policy is empty and
remote.storage.enable is true, the local log segments will be cleaned based
on log.local.retention.bytes and log.local.retention.ms.

Jun

On Fri, Jul 18, 2025 at 9:52 AM Chia-Ping Tsai <ch...@gmail.com> wrote:

> >
> > JR34: For these two configurations, setting an empty list feels a bit
> > unintuitive. If an empty list is
> > provided, the consumer will call the unsubscribe method, which doesn't
> > seem appropriate given
> > the documentation states: "Topic-partitions assigned to this task to
> > replicate."
> >
>
> agreed. Additionally, the source code [0] shows that it is not intended to
> generate empty tps for the source task
>
> [0]
>
> https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

>
> JR34: For these two configurations, setting an empty list feels a bit
> unintuitive. If an empty list is
> provided, the consumer will call the unsubscribe method, which doesn't
> seem appropriate given
> the documentation states: "Topic-partitions assigned to this task to
> replicate."
>

agreed. Additionally, the source code [0] shows that it is not intended to
generate empty tps for the source task

[0]
https://github.com/apache/kafka/blob/9b542b6ea21e84677a9292f250fc25f8b4162e6f/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L202

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

Thanks for the reply.

JR33, I’ve updated the validator to only allow non-empty lists. 
However, I have a question about the default value: It seems that in Streams, 
the config uses NO_DEFAULT_VALUE and includes a comment saying “required with no default value.” 
Should we also consider changing the default from NO_DEFAULT_VALUE to List.of()?

JR34: For these two configurations, setting an empty list feels a bit unintuitive. If an empty list is 
provided, the consumer will call the unsubscribe method, which doesn't seem appropriate given 
the documentation states: "Topic-partitions assigned to this task to replicate." 

JR35: AsyncConsumer doesn’t support this config, I updated it.

JR36: For ssl.enabled.protocols, the value is only accessed in DefaultSslEngineFactory[1], 
where null and an empty list result in the same behavior. 
Since users cannot set a null value through a configuration file, I believe we can safely disallow 
null values and allow empty lists instead.

[1]https://github.com/apache/kafka/blob/9572d19c5911f401a155741fcce61b69657aae79/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L146

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月18日 清晨6:32 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply.
> 
> JR31. This is fine then.
> 
> JR33. StreamsConfig: bootstrap.servers can't be empty.
> bootstrap.controllers is not available in Streams. It's only available in
> AdminClient.
> 
> JR34. If there are more MM instances than partitions, it seems that it's
> possible for task.assigned.partitions to be empty.
> Ditto for task.assigned.groups.
> 
> JR35. partition.assignment.strategy: It's only needed if group.protocol is
> classic. I guess that it's ok for it to be empty.
> 
> JR36. ssl.enabled.protocols: This is interesting. If this is null, it
> implies that all available protocols are enabled. However, since null is
> not the default value, a user can't set it to null through a config file. I
> am wondering if this is ever needed.
> 
> Jun
> 
> On Thu, Jul 17, 2025 at 7:23 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun, Chia-Ping,
>> 
>> Thanks for the reply.
>> 
>> JR28:
>> You're right. If the user passes a null value, it will be converted to an
>> empty list.
>> I’ve updated the KIP accordingly.
>> 
>> JR29:
>> For early.start.listeners, the exception is thrown during Kafka’s internal
>> validation,
>> and it’s not caused by a null or empty list. Therefore, I think we should
>> set the validator to none. I’ve updated the KIP.
>> 
>> JR30 / chia04:
>> Currently, passing a null value results in a NullPointerException. I’ve
>> updated
>> the KIP to reflect this behavior.
>> 
>> JR31:
>> Since the default value of plugin.path is null, it indicates that the user
>> has not set this configuration.
>> Changing it to an empty list might be confusing, as it implies the user
>> explicitly set the config but provided no paths.
>> However, if using an empty list is more appropriate for this config, I’m
>> fine with changing it.
>> We can set the default value to List.of() and disallow null values.
>> 
>> JR32:
>> An empty value for partition.assignment.strategy will result in an
>> IllegalStateException. I’ve updated the KIP.
>> 
>> chia00:
>> This config should not be null. I’ve updated the KIP to reflect that.
>> 
>> chia01 / chia02:
>> These configurations should be both non-null and non-empty.
>> I’ve updated them to only allow non-empty lists.
>> 
>> chia03:
>> Currently, passing a null value will throw a NullPointerException.
>> I’ve updated the KIP accordingly.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月17日 凌晨2:37 寫道：
>>> 
>>> hi Jiunn-Yang,
>>> 
>>> chia00. why could `bootstrap.servers` be null in StreamsConfig?
>>> 
>>> chia01. the empty `task.assigned.partitions` is a bit weird. That means
>>> consumer has nothing to seek/poll.
>>> 
>>> chia02. task.assigned.groups has analogous issue.
>>> 
>>> chia03. MirrorCheckpointConfig's `groups` and `groups.exclude` should not
>>> accept null, since they are complied by regular expression.
>>> 
>>> chia04. MirrorSourceConfig's `topics` and `topics.exclude` have similar
>>> issue.
>>> 
>>> Best,
>>> Chia-Ping
>>> 
>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月17日 週四 上午12:51寫道：
>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Thanks for the reply. A few more comments.
>>>> 
>>>> JR28. AdminClientConfig: Currently, bootstrap.servers doesn't throw
>>>> NullPointerException on null value. It's converted to an empty list.
>>>> 
>>>> JR29. The "Before exception" column: It would be useful to add which
>> value
>>>> causes an exception. For example, duplicated values currently don't
>>>> typically cause exceptions. Also, for early.start.listeners, which value
>>>> causes ConfigException currently?
>>>> 
>>>> JR30. Why do we allow the following to be null? In general, null only
>> makes
>>>> sense if it's the default value.
>>>> MirrorSourceConfig: topics, topics.exclude, config.properties.exclude
>>>> 
>>>> JR31. WorkerConfig: plugin.path It seems that we should allow it to be
>>>> empty, but not null?
>>>> 
>>>> JR32. partition.assignment.strategy shouldn't be empty, right?
>>>> 
>>>> Jun
>>>> 
>>>> On Wed, Jul 16, 2025 at 4:32 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hi Jun,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> JR26: I’ve addressed this change in the KIP.
>>>>> 
>>>>> JR27:
>>>>> I’ve documented the following configurations in the KIP:
>>>>> 
>>>>> ConsumerConfig
>>>>> partition.assignment.strategy
>>>>> 
>>>>> StreamsConfig
>>>>> bootstrap.servers
>>>>> 
>>>>> MirrorCheckpointConfig
>>>>> groups
>>>>> groups.exclude
>>>>> task.assigned.groups
>>>>> 
>>>>> MirrorSourceConfig
>>>>> topics
>>>>> topics.exclude
>>>>> config.properties.exclude
>>>>> 
>>>>> WorkerConfig
>>>>> plugin.path
>>>>> 
>>>>> RestServerConfig
>>>>> rest.extension.classes
>>>>> 
>>>>> Other configurations already have default values and corresponding
>>>>> validators:
>>>>> 
>>>>> LogConfig
>>>>> leader.replication.throttled.replicas: List.of() →
>>>>> ThrottledReplicaListValidator
>>>>> follower.replication.throttled.replicas: List.of() →
>>>>> ThrottledReplicaListValidator
>>>>> 
>>>>> StreamsConfig
>>>>> rack.aware.assignment.tags: List.of() → atMostOfSize
>>>>> 
>>>>> ConnectorConfig
>>>>> transforms: List.of() → aliasValidator
>>>>> predicates: List.of() → aliasValidator
>>>>> 
>>>>> SourceConnectorConfig
>>>>> topic.creation.groups: List.of() → ConfigDef.CompositeValidator
>>>>> 
>>>>> DistributedConfig
>>>>> inter.worker.verification.algorithms:
>>>>> defaultVerificationAlgorithms(crypto) → ConfigDef.LambdaValidator
>>>>> 
>>>>> RestServerConfig
>>>>> admin.listeners: null → AdminListenersValidator
>>>>> listeners: List.of("http://:8083") → ListenersValidator
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月16日 清晨7:14 寫道：
>>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply.
>>>>>> 
>>>>>> JR 26. early.start.listeners and interceptor.classes: It seems that
>>>> both
>>>>>> could be empty?
>>>>>> 
>>>>>> JR27. The following configs of type LIST are missing.
>>>>>> 
>>>>>> ConsumerConfig
>>>>>> partition.assignment.strategy
>>>>>> 
>>>>>> BrokerSecurityConfigs
>>>>>> 
>>>>>> QuorumConfig
>>>>>> 
>>>>>> LogConfig
>>>>>> leader.replication.throttled.replicas
>>>>>> follower.replication.throttled.replicas
>>>>>> 
>>>>>> StreamsConfig
>>>>>> rack.aware.assignment.tags
>>>>>> bootstrap.servers
>>>>>> 
>>>>>> MirrorCheckpointConfig
>>>>>> groups
>>>>>> groups.exclude
>>>>>> 
>>>>>> MirrorCheckpointConfig
>>>>>> task.assigned.groups
>>>>>> 
>>>>>> MirrorSourceConfig
>>>>>> topics
>>>>>> topics.exclude
>>>>>> config.properties.exclude
>>>>>> 
>>>>>> ConnectorConfig
>>>>>> transforms
>>>>>> predicates
>>>>>> 
>>>>>> SourceConnectorConfig
>>>>>> topic.creation.groups
>>>>>> 
>>>>>> WorkerConfig
>>>>>> plugin.path
>>>>>> 
>>>>>> DistributedConfig
>>>>>> inter.worker.verification.algorithms
>>>>>> 
>>>>>> RestServerConfig
>>>>>> rest.extension.classes
>>>>>> admin.listeners
>>>>>> listeners
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Tue, Jul 15, 2025 at 6:22 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hello Jun Chia,
>>>>>>> 
>>>>>>> Thanks for the reply,
>>>>>>> 
>>>>>>> JR20: This configuration should remain unchanged. I have updated the
>>>>> KIP.
>>>>>>> 
>>>>>>> JR21: For both Consumer and Producer, the default value is an empty
>>>>> list,
>>>>>>> and the
>>>>>>> validator used is NonNullValidator. I think we can change the
>>>> validator
>>>>> to
>>>>>>> ValidList.anyNonDuplicateValues
>>>>>>> with isNullAllowed = false and isEmptyAllowed = false.
>>>>>>> 
>>>>>>> JR23: I’ve addressed this change in the KIP. Although the default
>>>> value
>>>>> of
>>>>>>> early.start.listeners remains
>>>>>>> unchanged, the validator now disallows empty lists.
>>>>>>> 
>>>>>>> JR24: Since log.dirs does not change its default value, I used a dash
>>>>> (-)
>>>>>>> to indicate that the field remains unchanged.
>>>>>>> 
>>>>>>> JR25: I’ve updated the table and aligned similar validators as much
>> as
>>>>>>> possible for consistency.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月15日 下午2:12 寫道：
>>>>>>>> 
>>>>>>>>> JR20. controller.listener.names : It's required on the broker. Is
>> it
>>>>>>>> required on the controller?
>>>>>>>> 
>>>>>>>> there is a small "sugar" in the controller, which we will
>>>>>>>> create listener.security.protocol.map based on
>>>>> controller.listener.names
>>>>>>>> if listener.security.protocol.map is not explicitly set [0]
>>>>>>>> Additionally, `controller.listener.names` is also used to filter out
>>>>> the
>>>>>>>> advertised controller listeners [1]
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be
>>>> set
>>>>> to
>>>>>>>> null, right?
>>>>>>>> 
>>>>>>>> I don't think so. There is no null check after calling
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> `getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
>>>>>>>> which causes an NPE if getList returns null."
>>>>>>>> 
>>>>>>>> [0]
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
>>>>>>>> [1]
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
>>>>>>>> [2]
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
>>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：
>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Thanks for the updated KIP.
>>>>>>>>> 
>>>>>>>>> JR20. controller.listener.names : It's required on the broker. Is
>> it
>>>>>>>>> required on the controller?
>>>>>>>>> 
>>>>>>>>> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer)
>>>>>>> already
>>>>>>>>> require it to be non-null.
>>>>>>>>> 
>>>>>>>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be
>>>> set
>>>>> to
>>>>>>>>> null, right?
>>>>>>>>> 
>>>>>>>>> JR23. early.start.listeners: It allows default value of null, which
>>>>>>> means
>>>>>>>>> that all listeners included in controller.listener.names will also
>>>> be
>>>>>>> early
>>>>>>>>> start listeners.
>>>>>>>>> 
>>>>>>>>> JR24. log.dirs: It currently has a default value of null.
>>>>>>>>> 
>>>>>>>>> JR25. It seems there are quite a few more configs of type list. Do
>>>>> they
>>>>>>>>> need to be changed too?
>>>>>>>>> 
>>>>>>>>> AdminClientConfig: 4
>>>>>>>>> ConsumerConfig: 5
>>>>>>>>> ProducerConfig: 4
>>>>>>>>> SaslConfigs: 1
>>>>>>>>> SslConfigs: 2
>>>>>>>>> BrokerSecurityConfigs: 5
>>>>>>>>> MirrorClientConfig: 1
>>>>>>>>> org.apache.kafka.connect.mirror : 16
>>>>>>>>> org.apache.kafka.connect.runtime : 21
>>>>>>>>> GroupCoordinatorConfig: 3
>>>>>>>>> QuorumConfig: 2
>>>>>>>>> ServerConfigs: 1
>>>>>>>>> KRaftConfigs: 1
>>>>>>>>> ClientMetricsConfigs: 2
>>>>>>>>> MetricConfigs: 2
>>>>>>>>> LogConfig: 4
>>>>>>>>> StreamsConfig: 4
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Chia, Jun
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply,
>>>>>>>>>> I have updated the table in the KIP.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> Hi, Jun
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>> 
>>>>>>>>>>> I think the following configs could use null as the default value
>>>>> and
>>>>>>>>>> treat
>>>>>>>>>>> an "empty list" as an illegal value.
>>>>>>>>>>> 
>>>>>>>>>>> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
>>>>>>>>>>> `ssl.enabled.protocols`,
>>>>>>>>>>> and `log.dirs`
>>>>>>>>>>> 
>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Chia-Ping, Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>> 
>>>>>>>>>>>> As you said, if ssl.cipher.suites is empty, intuitively this
>>>> means
>>>>>>>>> that
>>>>>>>>>> no
>>>>>>>>>>>> cipher is allowed. So, this should be an illegal config. It just
>>>>>>>>> happens
>>>>>>>>>>>> that our implementation treats it as if the config is not set
>>>>>>> (meaning
>>>>>>>>>> the
>>>>>>>>>>>> value is null). So, it seems that a more intuitive convention
>> is:
>>>>> if
>>>>>>>>> the
>>>>>>>>>>>> config is not set, it defaults to all available cipher suites.
>> If
>>>>>>> it's
>>>>>>>>>>>> explicitly set, it can't be empty.
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Jun, Chia
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Also, it would be useful to add a compatibility column in the
>>>>>>>>> table.
>>>>>>>>>>>> If
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> value is formally disallowed, it would be useful to compare
>>>> the
>>>>>>> old
>>>>>>>>>>>> and
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>>>>>>>>>> I’ve updated the table to show which exceptions will be thrown
>>>> for
>>>>>>>>> each
>>>>>>>>>>>>> behavior.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I don't understand the point of "5.0". Could you please
>> share
>>>>>>> more
>>>>>>>>>>>>> details?
>>>>>>>>>>>>>>>> It seems to me changing the type or default value should be
>>>>> fine
>>>>>>>>> in
>>>>>>>>>>>>> minor
>>>>>>>>>>>>>>>> release if we don't break the existing property files.
>>>>>>>>>>>>> I don't think this requires any compatibility considerations,
>> so
>>>>>>> I’ve
>>>>>>>>>>>>> removed the Kafka 5.0 changes section.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type,
>>>> so
>>>>>>>>> using
>>>>>>>>>>>>> `LIST`
>>>>>>>>>>>>>>>> instead of `String` does not seems to make sense.
>>>>>>>>>>>>> I have removed the config from KIP
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> For ssl.cipher.suites, changing the default value from null
>> to
>>>>>>>>> empty
>>>>>>>>>>>> is
>>>>>>>>>>>>>>> actually a breaking change. According to the doc, null means
>>>>> that
>>>>>>>>> all
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> available cipher suites are supported. I am thinking that we
>>>>>>> should
>>>>>>>>>>>>>>> continue to allow null as the default since it could
>>>> represent a
>>>>>>>>>> state
>>>>>>>>>>>>>>> different from empty.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> "Empty list" seems to be a weird case. It appears to signify
>>>> that
>>>>>>> no
>>>>>>>>>>>>> cipher suites are accepted. However, in the codebase, an empty
>>>>> list
>>>>>>>>> is
>>>>>>>>>>>>> handled as if all available cipher suites are supported [0]. We
>>>>>>>>> should
>>>>>>>>>>>> not
>>>>>>>>>>>>> support the case of "no supported cipher suite," as it doesn't
>>>>> make
>>>>>>>>>>>> sense.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In short, changing the default value from null to empty does
>>>> not
>>>>>>>>> break
>>>>>>>>>>>>> the behavior.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [0]
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be
>>>> empty,
>>>>> we
>>>>>>>>>>>> will
>>>>>>>>>>>>>>> just continue to allow it in all future releases.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> For ssl.cipher.suites, changing the default value from null
>> to
>>>>>>>>> empty
>>>>>>>>>>>> is
>>>>>>>>>>>>>>> actually a breaking change. According to the doc, null means
>>>>> that
>>>>>>>>> all
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> available cipher suites are supported. I am thinking that we
>>>>>>> should
>>>>>>>>>>>>>>> continue to allow null as the default since it could
>>>> represent a
>>>>>>>>>> state
>>>>>>>>>>>>>>> different from empty.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Also, it would be useful to add a compatibility column in the
>>>>>>>>> table.
>>>>>>>>>>>> If
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> value is formally disallowed, it would be useful to compare
>>>> the
>>>>>>> old
>>>>>>>>>>>> and
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <
>>>>> chia7712@gmail.com
>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the explanation. I have updated the KIP
>>>> accordingly:
>>>>>>> in
>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>> 4.x, cleanup.policy
>>>>>>>>>>>>>>>>> can be set to an empty list with a warning message emitted;
>>>>>>>>>> however,
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> Kafka 5.0, setting an
>>>>>>>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If we agree to support "empty list", the approach of "none"
>>>>> could
>>>>>>>>> be
>>>>>>>>>>>>>>>> eliminated, right?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I don't understand the point of "5.0". Could you please
>> share
>>>>>>> more
>>>>>>>>>>>>> details?
>>>>>>>>>>>>>>>> It seems to me changing the type or default value should be
>>>>> fine
>>>>>>>>> in
>>>>>>>>>>>>> minor
>>>>>>>>>>>>>>>> release if we don't break the existing property files.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type,
>>>> so
>>>>>>>>> using
>>>>>>>>>>>>> `LIST`
>>>>>>>>>>>>>>>> instead of `String` does not seems to make sense.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Null could make sense as the default value, but it just
>>>>> means
>>>>>>>>>> that
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> config key is not set explicitly.
>>>>>>>>>>>>>>>>>>> Given that, it's probably simpler to just allow
>>>>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>>>>>>> Thanks for the explanation. I have updated the KIP
>>>>> accordingly:
>>>>>>>>> in
>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>> 4.x, cleanup.policy
>>>>>>>>>>>>>>>>> can be set to an empty list with a warning message emitted;
>>>>>>>>>> however,
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> Kafka 5.0, setting an
>>>>>>>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> For another, there is an additional issue which could be
>>>>>>> handled
>>>>>>>>>> by
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
>>>>> as a
>>>>>>>>>>>> LIST.
>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`.
>>>> Could
>>>>>>> we
>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to
>> make
>>>>>>> them
>>>>>>>>>>>>>>>>>> non-nullable.
>>>>>>>>>>>>>>>>> I have also updated the KIP to include the tables detailing
>>>>> the
>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>>> for LIST-type configuration
>>>>>>>>>>>>>>>>> default values and the migration from STRING-type to
>>>> LIST-type
>>>>>>>>>>>>>>>>> configurations.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06
>> 寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
>>>>>>>>> allows
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's
>>>> ok
>>>>>>> to
>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler
>>>> to
>>>>>>>>> just
>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> yes, that is what I meant before. Additionally, the `LIST`
>>>>> type
>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> accept "null". Instead, we should use an empty list as the
>>>>>>>>> default
>>>>>>>>>>>>>>>> value.
>>>>>>>>>>>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should
>>>>> never
>>>>>>>>> has
>>>>>>>>>>>>>>>> "null"
>>>>>>>>>>>>>>>>>> value. That could eliminate the null check.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> For another, there is an additional issue which could be
>>>>>>> handled
>>>>>>>>>> by
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
>>>>> as a
>>>>>>>>>>>> LIST.
>>>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`.
>>>> Could
>>>>>>> we
>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to
>> make
>>>>>>> them
>>>>>>>>>>>>>>>>>> non-nullable.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二
>>>> 上午5:56寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR10. Regarding supporting the null value, I am not sure
>>>> how
>>>>>>>>> one
>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>>>> a null value in a config file. For example, if you set
>> the
>>>>>>>>>>>> following
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> config file.
>>>>>>>>>>>>>>>>>>> configkey=
>>>>>>>>>>>>>>>>>>> The value of configkey will be an empty list, instead of
>>>>> null,
>>>>>>>>>>>>> right?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Null could make sense as the default value, but it just
>>>>> means
>>>>>>>>>> that
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> config key is not set explicitly.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes
>> that
>>>>>>>>> allows
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's
>>>> ok
>>>>>>> to
>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler
>>>> to
>>>>>>>>> just
>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is
>> optional,
>>>>>>> that
>>>>>>>>>> is
>>>>>>>>>>>>>>>>> right,
>>>>>>>>>>>>>>>>>>>> so I think
>>>>>>>>>>>>>>>>>>>> that we can allow null value, but not allow the empty
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> JR11: In this KIP, the goal is to address configurations
>>>>> that
>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> accept empty arrays.
>>>>>>>>>>>>>>>>>>>> We want to prevent users from passing in an empty list
>>>> when
>>>>>>>>> it's
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> valid. I believe there are two
>>>>>>>>>>>>>>>>>>>> main scenarios to consider:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 1. Nullable but not empty:
>>>>>>>>>>>>>>>>>>>> The parameter allows either a null value or a non-empty
>>>>> list.
>>>>>>>>> In
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> case,
>>>>>>>>>>>>>>>>>>>> if the user provides an empty list, we should throw a
>>>>>>>>>>>>>>>> ConfigException.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 2. Non-null and non-empty only: The parameter must
>> always
>>>>>>>>>>>> contain a
>>>>>>>>>>>>>>>>>>>> non-empty list — null is also
>>>>>>>>>>>>>>>>>>>> invalid. In this case, if the user provides either a
>> null
>>>>> or
>>>>>>>>> an
>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>> list, we should throw a ConfigException.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Therefore, I want to prevent scenarios where users can
>>>> pass
>>>>>>> in
>>>>>>>>>> an
>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
>>>>>>>>> allowNull
>>>>>>>>>>>>>>>>>>> property.
>>>>>>>>>>>>>>>>>>>> This allows us to specify
>>>>>>>>>>>>>>>>>>>> whether a configuration permits null values or not.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> JR13: I have updated the table
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30
>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply. A few more comments.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience
>>>> says
>>>>>>>>> that
>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>> optional. So an empty list seems to be valid.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support
>>>>> empty
>>>>>>>>>>>> lists
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> general, it's probably more consistent to just allow
>>>>>>>>>>>>>>>>> log.cleanup.policy
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> be empty as Luke suggested, instead of introducing a
>> new
>>>>>>>>> option
>>>>>>>>>>>>>>>> None.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be
>> null.
>>>>> It
>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
>>>>>>>>>> shouldn't
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> null.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> JR13. The table is very helpful. Could you also
>> describe
>>>>> the
>>>>>>>>>>>>> old/new
>>>>>>>>>>>>>>>>>>>>> behavior for each config with an empty list or
>> duplicate
>>>>>>>>> entry?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <
>> s7133700@gmail.com
>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I am manually bumping this thread. Any feedback would
>>>> be
>>>>>>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP and introduced a new
>> validator,
>>>>>>>>>>>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
>>>>>>>>>>>>>>>>>>>>>>> provided list is either null or a non-empty list
>>>> without
>>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>>>>>>>> entries. I’ve also listed the configurations that will
>>>>>>>>>>>>>>>>>>>>>>> be validated using this logic. Please feel free to
>>>> share
>>>>>>>>> any
>>>>>>>>>>>>>>>>> feedback
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日
>>>> 凌晨2:29
>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> It seems there are quite a few other configs of type
>>>>> list
>>>>>>>>>>>> (e.g.
>>>>>>>>>>>>>>>>>>>> several
>>>>>>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>>>>>>>>> related ones). It would be useful to understand if
>>>>> empty
>>>>>>>>>>>> lists
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>>>>>>>>> for them or not.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
>>>>>>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
>>>>>>> introduced?),
>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a
>> long
>>>>>>>>> time.
>>>>>>>>>> "
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> If you do a google search on "kafka infinite
>>>>> retention",
>>>>>>>>>> you
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>>> links like
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>>>>> both pointing to setting the retention time and
>>>>>>> retention
>>>>>>>>>>>> size
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> -1
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
>>>>>>>>>> intentionally
>>>>>>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>>>>>> retention, it's more likely they will use that
>>>>> approach
>>>>>>>>>>>>> instead
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> setting
>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our
>>>>>>> API/tools
>>>>>>>>>>>> make
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>>>>>>> for users to make a mistake by inadvertently
>> leaving
>>>>> the
>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
>>>>>>>>>>>> showuon@gmail.com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
>>>>>>>>> documented
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In
>> fact,
>>>>> if
>>>>>>>>> one
>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
>>>>> storage
>>>>>>>>>>>> since
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and
>> time
>>>>>>> too.
>>>>>>>>>>>> So,
>>>>>>>>>>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
>>>>>>>>> mistake
>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
>>>>> idea
>>>>>>>>>>>> behind
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
>>>>> aware
>>>>>>>>> of
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
>>>>> wanting
>>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
>>>>>>>>> implies
>>>>>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>>>>>>> retention, we also never documented (or failed)
>> the
>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
>>>>>>>>> behavior
>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>>>> existed
>>>>>>>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
>>>>>>>>> introduced?),
>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a
>>>> long
>>>>>>>>> time.
>>>>>>>>>>>> It
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>>>>>>>>>>>>> to break the backward compatibility in a minor
>>>>> release
>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>> (ex:
>>>>>>>>>>>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
>>>>>>>>>> existing
>>>>>>>>>>>>>>>> bug,
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>>> be a surprise to users.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>>>>>>> cleanup.policy
>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
>>>> never
>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in
>>>>> 4.x
>>>>>>>>> and
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> 5.0
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>>>>>>> exception directly to make it fail fast
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> This suggestion looks good to me!
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
>>>>>>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases,
>>>> it
>>>>> is
>>>>>>>>>>>> worth
>>>>>>>>>>>>>>>>>>>>>> considering
>>>>>>>>>>>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had
>> previously
>>>>>>>>>>>>> encountered
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>>>> instead.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
>>>>> never
>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make
>>>>> sense
>>>>>>>>> for
>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with
>>>> tiered
>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>>>>>>>>>>>> delete policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
>>>>>>>>> documented
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In
>>>> fact,
>>>>>>> if
>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
>>>>>>>>> storage
>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and
>>>> time
>>>>>>>>> too.
>>>>>>>>>>>>> So,
>>>>>>>>>>>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a
>> user
>>>>>>>>>> mistake
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption,
>> the
>>>>>>> idea
>>>>>>>>>>>>> behind
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
>>>>>>> aware
>>>>>>>>>> of
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
>>>>>>>>> wanting
>>>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
>>>>> never
>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
>>>>>>>>>>>>> showuon@gmail.com
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Comments:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty
>>>>> cleanup.policy
>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is
>>>> justified."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
>>>>>>>>>> trade-off
>>>>>>>>>>>>>>>>>>>> justified?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid
>>>> in
>>>>>>>>>>>> kafka,
>>>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this
>>>> KIP?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to
>>>> keep
>>>>>>> all
>>>>>>>>>>>>>>>> history
>>>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a cleanup policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast
>>>> for
>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
>>>>>>>>>>>>> "process.roles"
>>>>>>>>>>>>>>>>>>>> configs
>>>>>>>>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> they are invalid.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a
>>>>> persuasive
>>>>>>>>>>>>> reason
>>>>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to
>>>> configure
>>>>>>>>>> Kafka
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> retain
>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP,
>>>>> but
>>>>>>>>> to
>>>>>>>>>>>> me,
>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a
>>>> clear
>>>>>>>>> and
>>>>>>>>>>>>>>>> direct
>>>>>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really have to change the
>> "cleanup.policy"
>>>>>>>>>> config?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
>>>>>>>>>>>>>>>>>>> <jun@confluent.io.invalid
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to
>>>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
>>>> been
>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
>>>>>>>>>>>>>>>> Additionally,
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list
>> to
>>>>> the
>>>>>>>>>>>>>>>>> inNonEmpty
>>>>>>>>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>>>>> 2025年5月6日
>>>>>>>>>>>>> 凌晨12:46
>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users
>>>> pass
>>>>>>>>> empty
>>>>>>>>>>>>>>>>> config,
>>>>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw
>>>>>>> exception
>>>>>>>>>>>>>>>>> directly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>>>> 2025年5月6日
>>>>>>> 週二
>>>>>>>>>>>>>>>>>>> 上午12:28寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still
>>>>> references
>>>>>>>>> to
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be
>>>>> removed
>>>>>>>>> in
>>>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>>> 5.0"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
>>>>> been
>>>>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>>>>>>>>> 2025年5月3日
>>>>>>>>>>>>>>>> 凌晨2:03
>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
>>>>>>>>> Migration
>>>>>>>>>>>>>>>> Plan",
>>>>>>>>>>>>>>>>>>>>>>>>>>> could you
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
>>>>>>>>>> "clean.policy="
>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> replaced
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to
>>>>> disable
>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> compact.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>>>>>>> 2025年5月3日
>>>>>>>>>> 週六
>>>>>>>>>>>>>>>>>>>>>>>>>> 上午1:55寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in
>>>> for
>>>>>>>>> now.
>>>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references
>> to
>>>>>>>>>>>>>>>> deprecation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
>>>>>>>>>>>> ValidList#in
>>>>>>>>>>>>>>>>>>>>>>>>>> method,
>>>>>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
>>>>>>>>> useful
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces
>> non-empty
>>>>>>>>>> lists).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be
>>>> useful
>>>>> to
>>>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
>>>>> will
>>>>>>>>>>>> fail
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>>>> 2025年5月2日
>>>>>>>>>>>>>>>>>>> 凌晨12:50
>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be
>>>> useful
>>>>> to
>>>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
>>>>> will
>>>>>>>>>>>> fail
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we
>>>> need
>>>>>>> to
>>>>>>>>>>>>>>>> maintain
>>>>>>>>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
>>>>>>>>> isEmptyAllowed
>>>>>>>>>>>>> flag
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of
>>>>>>> isNonEmpty()
>>>>>>>>>> to
>>>>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID>
>> 於
>>>>>>>>>>>> 2025年5月1日
>>>>>>>>>>>>>>>>>>> 凌晨4:17
>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed?
>>>> It's
>>>>>>>>>>>> awkward
>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear
>> when
>>>>>>> it's
>>>>>>>>>>>> set
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> true.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽
>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
>>>>>>>>>> introduce
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is
>>>> that
>>>>>>> we
>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>> explicitly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
>>>>>>>>> format
>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>>>> phase
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
>>>>>>>>>> behavior.”
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <jun@confluent.io.INVALID
>>> 
>>>> 於
>>>>>>>>>>>>>>>> 2025年4月25日
>>>>>>>>>>>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
>>>>>>>>> introduced
>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be
>> thrown
>>>>>>>>> during
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
>>>>>>>>> startup."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating
>>>> the
>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>> group.coordinator.rebalance.protocols
>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM
>>>> 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this
>>>> change:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup
>>>> phase
>>>>> to
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an
>>>>> empty
>>>>>>>>>>>> list
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException
>> during
>>>>>>>>>> storage
>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
>> <jun@confluent.io.INVALID
>>>>> 
>>>>> 於
>>>>>>>>>>>>>>>>> 2025年4月24日
>>>>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:44
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty
>>>> right
>>>>>>>>> now.
>>>>>>>>>>>>> With
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and
>>>> more
>>>>>>>>>>>>> accurate)
>>>>>>>>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind
>>>> of
>>>>>>>>>> agree
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more
>>>>>>> likely
>>>>>>>>>> to
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as
>>>>> None,
>>>>>>>>>> the
>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception
>>>> will
>>>>>>>>> let
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>> Tsai
>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I
>>>>> understand
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> maintaining
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
>>>>>>>>>> documentation,
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence,
>>>> should
>>>>> we
>>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x?
>> This
>>>>>>>>> could
>>>>>>>>>>>>>>>>>>> streamline
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
>>>> <jun@confluent.io.invalid
>>>>>> 
>>>>>>> 於
>>>>>>>>>>>>>>>>>>> 2025年4月23日
>>>>>>>>>>>>>>>>>>>>>>>>>> 週三
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy
>> will
>>>>> not
>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>> compact
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We
>>>> won't
>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>> segments
>>>>>>>>>>>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
>>>>>>>>> log.retention.ms,
>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
>>>>>>>>>>>>>>>> log.local.retention.bytes/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk
>>>>>>> space
>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at
>> 9:45 AM
>>>>> Jun
>>>>>>>>>>>> Rao <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
>>>>>>>>> values
>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at
>>>> 7:29 AM
>>>>>>>>> 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
>>>>>>>>>>>>> accordingly
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> revised
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none
>>>> policy
>>>>>>>>>>>> cannot
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an
>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the
>>>>> preProcessParsedConfig
>>>>>>>>>>>> step.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform
>>>> users
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
>>>>>>> <jun@confluent.io.INVALID
>>>>>>>>>> 
>>>>>>>>>>>> 於
>>>>>>>>>>>>>>>>>>>>>>>>>> 2025年4月22日
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:52
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
>>>>>>>>> comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a
>>>>> new
>>>>>>>>>>>> value
>>>>>>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are
>>>>>>> valid.
>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would
>>>> be
>>>>>>>>>>>> useful
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior
>>>> during
>>>>>>>>>>>> upgrade
>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's
>>>> not
>>>>>>>>> clear
>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at
>>>>> 8:25 AM
>>>>>>>>> 黃竣陽
>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
>>>>>>>>> discussion
>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>> KIP-1161:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
>>>> improve
>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be
>>>>> left
>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
>>>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>>>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to
>>>>> validate
>>>>>>>>>>>>> whether
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none'
>>>>> value
>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply.

JR31. This is fine then.

JR33. StreamsConfig: bootstrap.servers can't be empty.
bootstrap.controllers is not available in Streams. It's only available in
AdminClient.

JR34. If there are more MM instances than partitions, it seems that it's
possible for task.assigned.partitions to be empty.
Ditto for task.assigned.groups.

JR35. partition.assignment.strategy: It's only needed if group.protocol is
classic. I guess that it's ok for it to be empty.

JR36. ssl.enabled.protocols: This is interesting. If this is null, it
implies that all available protocols are enabled. However, since null is
not the default value, a user can't set it to null through a config file. I
am wondering if this is ever needed.

Jun

On Thu, Jul 17, 2025 at 7:23 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun, Chia-Ping,
>
> Thanks for the reply.
>
> JR28:
> You're right. If the user passes a null value, it will be converted to an
> empty list.
> I’ve updated the KIP accordingly.
>
> JR29:
> For early.start.listeners, the exception is thrown during Kafka’s internal
> validation,
> and it’s not caused by a null or empty list. Therefore, I think we should
> set the validator to none. I’ve updated the KIP.
>
> JR30 / chia04:
> Currently, passing a null value results in a NullPointerException. I’ve
> updated
>  the KIP to reflect this behavior.
>
> JR31:
> Since the default value of plugin.path is null, it indicates that the user
> has not set this configuration.
> Changing it to an empty list might be confusing, as it implies the user
> explicitly set the config but provided no paths.
> However, if using an empty list is more appropriate for this config, I’m
> fine with changing it.
> We can set the default value to List.of() and disallow null values.
>
> JR32:
> An empty value for partition.assignment.strategy will result in an
> IllegalStateException. I’ve updated the KIP.
>
> chia00:
> This config should not be null. I’ve updated the KIP to reflect that.
>
> chia01 / chia02:
> These configurations should be both non-null and non-empty.
> I’ve updated them to only allow non-empty lists.
>
> chia03:
> Currently, passing a null value will throw a NullPointerException.
> I’ve updated the KIP accordingly.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月17日 凌晨2:37 寫道：
> >
> > hi Jiunn-Yang,
> >
> > chia00. why could `bootstrap.servers` be null in StreamsConfig?
> >
> > chia01. the empty `task.assigned.partitions` is a bit weird. That means
> > consumer has nothing to seek/poll.
> >
> > chia02. task.assigned.groups has analogous issue.
> >
> > chia03. MirrorCheckpointConfig's `groups` and `groups.exclude` should not
> > accept null, since they are complied by regular expression.
> >
> > chia04. MirrorSourceConfig's `topics` and `topics.exclude` have similar
> > issue.
> >
> > Best,
> > Chia-Ping
> >
> > Jun Rao <ju...@confluent.io.invalid> 於 2025年7月17日 週四 上午12:51寫道：
> >
> >> Hi, Jiunn-Yang,
> >>
> >> Thanks for the reply. A few more comments.
> >>
> >> JR28. AdminClientConfig: Currently, bootstrap.servers doesn't throw
> >> NullPointerException on null value. It's converted to an empty list.
> >>
> >> JR29. The "Before exception" column: It would be useful to add which
> value
> >> causes an exception. For example, duplicated values currently don't
> >> typically cause exceptions. Also, for early.start.listeners, which value
> >> causes ConfigException currently?
> >>
> >> JR30. Why do we allow the following to be null? In general, null only
> makes
> >> sense if it's the default value.
> >> MirrorSourceConfig: topics, topics.exclude, config.properties.exclude
> >>
> >> JR31. WorkerConfig: plugin.path It seems that we should allow it to be
> >> empty, but not null?
> >>
> >> JR32. partition.assignment.strategy shouldn't be empty, right?
> >>
> >> Jun
> >>
> >> On Wed, Jul 16, 2025 at 4:32 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hi Jun,
> >>>
> >>> Thanks for the reply.
> >>>
> >>> JR26: I’ve addressed this change in the KIP.
> >>>
> >>> JR27:
> >>> I’ve documented the following configurations in the KIP:
> >>>
> >>> ConsumerConfig
> >>> partition.assignment.strategy
> >>>
> >>> StreamsConfig
> >>> bootstrap.servers
> >>>
> >>> MirrorCheckpointConfig
> >>> groups
> >>> groups.exclude
> >>> task.assigned.groups
> >>>
> >>> MirrorSourceConfig
> >>> topics
> >>> topics.exclude
> >>> config.properties.exclude
> >>>
> >>> WorkerConfig
> >>> plugin.path
> >>>
> >>> RestServerConfig
> >>> rest.extension.classes
> >>>
> >>> Other configurations already have default values and corresponding
> >>> validators:
> >>>
> >>> LogConfig
> >>> leader.replication.throttled.replicas: List.of() →
> >>> ThrottledReplicaListValidator
> >>> follower.replication.throttled.replicas: List.of() →
> >>> ThrottledReplicaListValidator
> >>>
> >>> StreamsConfig
> >>> rack.aware.assignment.tags: List.of() → atMostOfSize
> >>>
> >>> ConnectorConfig
> >>> transforms: List.of() → aliasValidator
> >>> predicates: List.of() → aliasValidator
> >>>
> >>> SourceConnectorConfig
> >>> topic.creation.groups: List.of() → ConfigDef.CompositeValidator
> >>>
> >>> DistributedConfig
> >>> inter.worker.verification.algorithms:
> >>> defaultVerificationAlgorithms(crypto) → ConfigDef.LambdaValidator
> >>>
> >>> RestServerConfig
> >>> admin.listeners: null → AdminListenersValidator
> >>> listeners: List.of("http://:8083") → ListenersValidator
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月16日 清晨7:14 寫道：
> >>>>
> >>>> Hi, Jiunn-Yang,
> >>>>
> >>>> Thanks for the reply.
> >>>>
> >>>> JR 26. early.start.listeners and interceptor.classes: It seems that
> >> both
> >>>> could be empty?
> >>>>
> >>>> JR27. The following configs of type LIST are missing.
> >>>>
> >>>> ConsumerConfig
> >>>> partition.assignment.strategy
> >>>>
> >>>> BrokerSecurityConfigs
> >>>>
> >>>> QuorumConfig
> >>>>
> >>>> LogConfig
> >>>> leader.replication.throttled.replicas
> >>>> follower.replication.throttled.replicas
> >>>>
> >>>> StreamsConfig
> >>>> rack.aware.assignment.tags
> >>>> bootstrap.servers
> >>>>
> >>>> MirrorCheckpointConfig
> >>>> groups
> >>>> groups.exclude
> >>>>
> >>>> MirrorCheckpointConfig
> >>>> task.assigned.groups
> >>>>
> >>>> MirrorSourceConfig
> >>>> topics
> >>>> topics.exclude
> >>>> config.properties.exclude
> >>>>
> >>>> ConnectorConfig
> >>>> transforms
> >>>> predicates
> >>>>
> >>>> SourceConnectorConfig
> >>>> topic.creation.groups
> >>>>
> >>>> WorkerConfig
> >>>> plugin.path
> >>>>
> >>>> DistributedConfig
> >>>> inter.worker.verification.algorithms
> >>>>
> >>>> RestServerConfig
> >>>> rest.extension.classes
> >>>> admin.listeners
> >>>> listeners
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>> On Tue, Jul 15, 2025 at 6:22 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>
> >>>>> Hello Jun Chia,
> >>>>>
> >>>>> Thanks for the reply,
> >>>>>
> >>>>> JR20: This configuration should remain unchanged. I have updated the
> >>> KIP.
> >>>>>
> >>>>> JR21: For both Consumer and Producer, the default value is an empty
> >>> list,
> >>>>> and the
> >>>>> validator used is NonNullValidator. I think we can change the
> >> validator
> >>> to
> >>>>> ValidList.anyNonDuplicateValues
> >>>>> with isNullAllowed = false and isEmptyAllowed = false.
> >>>>>
> >>>>> JR23: I’ve addressed this change in the KIP. Although the default
> >> value
> >>> of
> >>>>> early.start.listeners remains
> >>>>> unchanged, the validator now disallows empty lists.
> >>>>>
> >>>>> JR24: Since log.dirs does not change its default value, I used a dash
> >>> (-)
> >>>>> to indicate that the field remains unchanged.
> >>>>>
> >>>>> JR25: I’ve updated the table and aligned similar validators as much
> as
> >>>>> possible for consistency.
> >>>>>
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月15日 下午2:12 寫道：
> >>>>>>
> >>>>>>> JR20. controller.listener.names : It's required on the broker. Is
> it
> >>>>>> required on the controller?
> >>>>>>
> >>>>>> there is a small "sugar" in the controller, which we will
> >>>>>> create listener.security.protocol.map based on
> >>> controller.listener.names
> >>>>>> if listener.security.protocol.map is not explicitly set [0]
> >>>>>> Additionally, `controller.listener.names` is also used to filter out
> >>> the
> >>>>>> advertised controller listeners [1]
> >>>>>>
> >>>>>>
> >>>>>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be
> >> set
> >>> to
> >>>>>> null, right?
> >>>>>>
> >>>>>> I don't think so. There is no null check after calling
> >>>>>>
> >>>>>
> >>>
> >>
> `getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
> >>>>>> which causes an NPE if getList returns null."
> >>>>>>
> >>>>>> [0]
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
> >>>>>> [1]
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
> >>>>>> [2]
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
> >>>>>>
> >>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：
> >>>>>>
> >>>>>>> Hi, Jiunn-Yang,
> >>>>>>>
> >>>>>>> Thanks for the updated KIP.
> >>>>>>>
> >>>>>>> JR20. controller.listener.names : It's required on the broker. Is
> it
> >>>>>>> required on the controller?
> >>>>>>>
> >>>>>>> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer)
> >>>>> already
> >>>>>>> require it to be non-null.
> >>>>>>>
> >>>>>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be
> >> set
> >>> to
> >>>>>>> null, right?
> >>>>>>>
> >>>>>>> JR23. early.start.listeners: It allows default value of null, which
> >>>>> means
> >>>>>>> that all listeners included in controller.listener.names will also
> >> be
> >>>>> early
> >>>>>>> start listeners.
> >>>>>>>
> >>>>>>> JR24. log.dirs: It currently has a default value of null.
> >>>>>>>
> >>>>>>> JR25. It seems there are quite a few more configs of type list. Do
> >>> they
> >>>>>>> need to be changed too?
> >>>>>>>
> >>>>>>> AdminClientConfig: 4
> >>>>>>> ConsumerConfig: 5
> >>>>>>> ProducerConfig: 4
> >>>>>>> SaslConfigs: 1
> >>>>>>> SslConfigs: 2
> >>>>>>> BrokerSecurityConfigs: 5
> >>>>>>> MirrorClientConfig: 1
> >>>>>>> org.apache.kafka.connect.mirror : 16
> >>>>>>> org.apache.kafka.connect.runtime : 21
> >>>>>>> GroupCoordinatorConfig: 3
> >>>>>>> QuorumConfig: 2
> >>>>>>> ServerConfigs: 1
> >>>>>>> KRaftConfigs: 1
> >>>>>>> ClientMetricsConfigs: 2
> >>>>>>> MetricConfigs: 2
> >>>>>>> LogConfig: 4
> >>>>>>> StreamsConfig: 4
> >>>>>>>
> >>>>>>> Jun
> >>>>>>>
> >>>>>>> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hi Chia, Jun
> >>>>>>>>
> >>>>>>>> Thanks for the reply,
> >>>>>>>> I have updated the table in the KIP.
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>>
> >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
> >>>>>>>>>
> >>>>>>>>> Hi, Jun
> >>>>>>>>>
> >>>>>>>>> Thanks for the reply.
> >>>>>>>>>
> >>>>>>>>> I think the following configs could use null as the default value
> >>> and
> >>>>>>>> treat
> >>>>>>>>> an "empty list" as an illegal value.
> >>>>>>>>>
> >>>>>>>>> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
> >>>>>>>>> `ssl.enabled.protocols`,
> >>>>>>>>> and `log.dirs`
> >>>>>>>>>
> >>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
> >>>>>>>>>
> >>>>>>>>>> Hi, Chia-Ping, Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>
> >>>>>>>>>> As you said, if ssl.cipher.suites is empty, intuitively this
> >> means
> >>>>>>> that
> >>>>>>>> no
> >>>>>>>>>> cipher is allowed. So, this should be an illegal config. It just
> >>>>>>> happens
> >>>>>>>>>> that our implementation treats it as if the config is not set
> >>>>> (meaning
> >>>>>>>> the
> >>>>>>>>>> value is null). So, it seems that a more intuitive convention
> is:
> >>> if
> >>>>>>> the
> >>>>>>>>>> config is not set, it defaults to all available cipher suites.
> If
> >>>>> it's
> >>>>>>>>>> explicitly set, it can't be empty.
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Jun, Chia
> >>>>>>>>>>>
> >>>>>>>>>>>>> Also, it would be useful to add a compatibility column in the
> >>>>>>> table.
> >>>>>>>>>> If
> >>>>>>>>>>> a
> >>>>>>>>>>>>> value is formally disallowed, it would be useful to compare
> >> the
> >>>>> old
> >>>>>>>>>> and
> >>>>>>>>>>> the
> >>>>>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
> >>>>>>>>>>> I’ve updated the table to show which exceptions will be thrown
> >> for
> >>>>>>> each
> >>>>>>>>>>> behavior.
> >>>>>>>>>>>
> >>>>>>>>>>>>>> I don't understand the point of "5.0". Could you please
> share
> >>>>> more
> >>>>>>>>>>> details?
> >>>>>>>>>>>>>> It seems to me changing the type or default value should be
> >>> fine
> >>>>>>> in
> >>>>>>>>>>> minor
> >>>>>>>>>>>>>> release if we don't break the existing property files.
> >>>>>>>>>>> I don't think this requires any compatibility considerations,
> so
> >>>>> I’ve
> >>>>>>>>>>> removed the Kafka 5.0 changes section.
> >>>>>>>>>>>
> >>>>>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type,
> >> so
> >>>>>>> using
> >>>>>>>>>>> `LIST`
> >>>>>>>>>>>>>> instead of `String` does not seems to make sense.
> >>>>>>>>>>> I have removed the config from KIP
> >>>>>>>>>>>
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>
> >>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> >>>>>>>>>>>>
> >>>>>>>>>>>>> For ssl.cipher.suites, changing the default value from null
> to
> >>>>>>> empty
> >>>>>>>>>> is
> >>>>>>>>>>>>> actually a breaking change. According to the doc, null means
> >>> that
> >>>>>>> all
> >>>>>>>>>>> the
> >>>>>>>>>>>>> available cipher suites are supported. I am thinking that we
> >>>>> should
> >>>>>>>>>>>>> continue to allow null as the default since it could
> >> represent a
> >>>>>>>> state
> >>>>>>>>>>>>> different from empty.
> >>>>>>>>>>>>
> >>>>>>>>>>>> "Empty list" seems to be a weird case. It appears to signify
> >> that
> >>>>> no
> >>>>>>>>>>> cipher suites are accepted. However, in the codebase, an empty
> >>> list
> >>>>>>> is
> >>>>>>>>>>> handled as if all available cipher suites are supported [0]. We
> >>>>>>> should
> >>>>>>>>>> not
> >>>>>>>>>>> support the case of "no supported cipher suite," as it doesn't
> >>> make
> >>>>>>>>>> sense.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In short, changing the default value from null to empty does
> >> not
> >>>>>>> break
> >>>>>>>>>>> the behavior.
> >>>>>>>>>>>>
> >>>>>>>>>>>> [0]
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be
> >> empty,
> >>> we
> >>>>>>>>>> will
> >>>>>>>>>>>>> just continue to allow it in all future releases.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For ssl.cipher.suites, changing the default value from null
> to
> >>>>>>> empty
> >>>>>>>>>> is
> >>>>>>>>>>>>> actually a breaking change. According to the doc, null means
> >>> that
> >>>>>>> all
> >>>>>>>>>>> the
> >>>>>>>>>>>>> available cipher suites are supported. I am thinking that we
> >>>>> should
> >>>>>>>>>>>>> continue to allow null as the default since it could
> >> represent a
> >>>>>>>> state
> >>>>>>>>>>>>> different from empty.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, it would be useful to add a compatibility column in the
> >>>>>>> table.
> >>>>>>>>>> If
> >>>>>>>>>>> a
> >>>>>>>>>>>>> value is formally disallowed, it would be useful to compare
> >> the
> >>>>> old
> >>>>>>>>>> and
> >>>>>>>>>>> the
> >>>>>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <
> >>> chia7712@gmail.com
> >>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the explanation. I have updated the KIP
> >> accordingly:
> >>>>> in
> >>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>> 4.x, cleanup.policy
> >>>>>>>>>>>>>>> can be set to an empty list with a warning message emitted;
> >>>>>>>> however,
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>> Kafka 5.0, setting an
> >>>>>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If we agree to support "empty list", the approach of "none"
> >>> could
> >>>>>>> be
> >>>>>>>>>>>>>> eliminated, right?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I don't understand the point of "5.0". Could you please
> share
> >>>>> more
> >>>>>>>>>>> details?
> >>>>>>>>>>>>>> It seems to me changing the type or default value should be
> >>> fine
> >>>>>>> in
> >>>>>>>>>>> minor
> >>>>>>>>>>>>>> release if we don't break the existing property files.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type,
> >> so
> >>>>>>> using
> >>>>>>>>>>> `LIST`
> >>>>>>>>>>>>>> instead of `String` does not seems to make sense.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Null could make sense as the default value, but it just
> >>> means
> >>>>>>>> that
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> config key is not set explicitly.
> >>>>>>>>>>>>>>>>> Given that, it's probably simpler to just allow
> >>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>>>>>>> Thanks for the explanation. I have updated the KIP
> >>> accordingly:
> >>>>>>> in
> >>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>> 4.x, cleanup.policy
> >>>>>>>>>>>>>>> can be set to an empty list with a warning message emitted;
> >>>>>>>> however,
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>> Kafka 5.0, setting an
> >>>>>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For another, there is an additional issue which could be
> >>>>> handled
> >>>>>>>> by
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
> >>> as a
> >>>>>>>>>> LIST.
> >>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`.
> >> Could
> >>>>> we
> >>>>>>>>>>> change
> >>>>>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to
> make
> >>>>> them
> >>>>>>>>>>>>>>>> non-nullable.
> >>>>>>>>>>>>>>> I have also updated the KIP to include the tables detailing
> >>> the
> >>>>>>>>>>> changes
> >>>>>>>>>>>>>>> for LIST-type configuration
> >>>>>>>>>>>>>>> default values and the migration from STRING-type to
> >> LIST-type
> >>>>>>>>>>>>>>> configurations.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06
> 寫道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> hi all,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
> >>>>>>> allows
> >>>>>>>>>> an
> >>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's
> >> ok
> >>>>> to
> >>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler
> >> to
> >>>>>>> just
> >>>>>>>>>>>>>> allow
> >>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> yes, that is what I meant before. Additionally, the `LIST`
> >>> type
> >>>>>>>>>>> should
> >>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>> accept "null". Instead, we should use an empty list as the
> >>>>>>> default
> >>>>>>>>>>>>>> value.
> >>>>>>>>>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should
> >>> never
> >>>>>>> has
> >>>>>>>>>>>>>> "null"
> >>>>>>>>>>>>>>>> value. That could eliminate the null check.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For another, there is an additional issue which could be
> >>>>> handled
> >>>>>>>> by
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
> >>> as a
> >>>>>>>>>> LIST.
> >>>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`.
> >> Could
> >>>>> we
> >>>>>>>>>>> change
> >>>>>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to
> make
> >>>>> them
> >>>>>>>>>>>>>>>> non-nullable.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二
> >> 上午5:56寫道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR10. Regarding supporting the null value, I am not sure
> >> how
> >>>>>>> one
> >>>>>>>>>>> could
> >>>>>>>>>>>>>>> set
> >>>>>>>>>>>>>>>>> a null value in a config file. For example, if you set
> the
> >>>>>>>>>> following
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> config file.
> >>>>>>>>>>>>>>>>> configkey=
> >>>>>>>>>>>>>>>>> The value of configkey will be an empty list, instead of
> >>> null,
> >>>>>>>>>>> right?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Null could make sense as the default value, but it just
> >>> means
> >>>>>>>> that
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> config key is not set explicitly.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes
> that
> >>>>>>> allows
> >>>>>>>>>> an
> >>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's
> >> ok
> >>>>> to
> >>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler
> >> to
> >>>>>>> just
> >>>>>>>>>>>>>> allow
> >>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is
> optional,
> >>>>> that
> >>>>>>>> is
> >>>>>>>>>>>>>>> right,
> >>>>>>>>>>>>>>>>>> so I think
> >>>>>>>>>>>>>>>>>> that we can allow null value, but not allow the empty
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> JR11: In this KIP, the goal is to address configurations
> >>> that
> >>>>>>>>>>> should
> >>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> accept empty arrays.
> >>>>>>>>>>>>>>>>>> We want to prevent users from passing in an empty list
> >> when
> >>>>>>> it's
> >>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> valid. I believe there are two
> >>>>>>>>>>>>>>>>>> main scenarios to consider:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1. Nullable but not empty:
> >>>>>>>>>>>>>>>>>> The parameter allows either a null value or a non-empty
> >>> list.
> >>>>>>> In
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> case,
> >>>>>>>>>>>>>>>>>> if the user provides an empty list, we should throw a
> >>>>>>>>>>>>>> ConfigException.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 2. Non-null and non-empty only: The parameter must
> always
> >>>>>>>>>> contain a
> >>>>>>>>>>>>>>>>>> non-empty list — null is also
> >>>>>>>>>>>>>>>>>> invalid. In this case, if the user provides either a
> null
> >>> or
> >>>>>>> an
> >>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>> list, we should throw a ConfigException.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Therefore, I want to prevent scenarios where users can
> >> pass
> >>>>> in
> >>>>>>>> an
> >>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
> >>>>>>> allowNull
> >>>>>>>>>>>>>>>>> property.
> >>>>>>>>>>>>>>>>>> This allows us to specify
> >>>>>>>>>>>>>>>>>> whether a configuration permits null values or not.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> JR13: I have updated the table
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30
> >> 寫道：
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Sorry for the late reply. A few more comments.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience
> >> says
> >>>>>>> that
> >>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>> optional. So an empty list seems to be valid.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support
> >>> empty
> >>>>>>>>>> lists
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> general, it's probably more consistent to just allow
> >>>>>>>>>>>>>>> log.cleanup.policy
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> be empty as Luke suggested, instead of introducing a
> new
> >>>>>>> option
> >>>>>>>>>>>>>> None.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be
> null.
> >>> It
> >>>>>>>>>> seems
> >>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
> >>>>>>>> shouldn't
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> null.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> JR13. The table is very helpful. Could you also
> describe
> >>> the
> >>>>>>>>>>> old/new
> >>>>>>>>>>>>>>>>>>> behavior for each config with an empty list or
> duplicate
> >>>>>>> entry?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <
> s7133700@gmail.com
> >>>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I am manually bumping this thread. Any feedback would
> >> be
> >>>>>>>>>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I have updated the KIP and introduced a new
> validator,
> >>>>>>>>>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
> >>>>>>>>>>>>>>>>>>>>> provided list is either null or a non-empty list
> >> without
> >>>>>>>>>>> duplicate
> >>>>>>>>>>>>>>>>>>>> entries. I’ve also listed the configurations that will
> >>>>>>>>>>>>>>>>>>>>> be validated using this logic. Please feel free to
> >> share
> >>>>>>> any
> >>>>>>>>>>>>>>> feedback
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> suggestions.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日
> >> 凌晨2:29
> >>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> It seems there are quite a few other configs of type
> >>> list
> >>>>>>>>>> (e.g.
> >>>>>>>>>>>>>>>>>> several
> >>>>>>>>>>>>>>>>>>>> SSL
> >>>>>>>>>>>>>>>>>>>>>> related ones). It would be useful to understand if
> >>> empty
> >>>>>>>>>> lists
> >>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> valid
> >>>>>>>>>>>>>>>>>>>>>> for them or not.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
> >>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
> >>>>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> >>>>> introduced?),
> >>>>>>>>>>> there
> >>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a
> long
> >>>>>>> time.
> >>>>>>>> "
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> If you do a google search on "kafka infinite
> >>> retention",
> >>>>>>>> you
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> get
> >>>>>>>>>>>>>>>>>>>>>>> links like
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> >>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> >>>>>>>>>>>>>>>>>>>> ,
> >>>>>>>>>>>>>>>>>>>>>>> both pointing to setting the retention time and
> >>>>> retention
> >>>>>>>>>> size
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> -1
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
> >>>>>>>> intentionally
> >>>>>>>>>>>>>> wants
> >>>>>>>>>>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>>>>>> retention, it's more likely they will use that
> >>> approach
> >>>>>>>>>>> instead
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>> setting
> >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our
> >>>>> API/tools
> >>>>>>>>>> make
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>> possible
> >>>>>>>>>>>>>>>>>>>>>>> for users to make a mistake by inadvertently
> leaving
> >>> the
> >>>>>>>>>>> config
> >>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
> >>>>>>>>>> showuon@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> >>>>>>> documented
> >>>>>>>>>>> that
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In
> fact,
> >>> if
> >>>>>>> one
> >>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
> >>> storage
> >>>>>>>>>> since
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>> stop
> >>>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and
> time
> >>>>> too.
> >>>>>>>>>> So,
> >>>>>>>>>>>>>>>>> leaving
> >>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> >>>>>>> mistake
> >>>>>>>>>>> than
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
> >>> idea
> >>>>>>>>>> behind
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
> >>> aware
> >>>>>>> of
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> likely
> >>>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
> >>> wanting
> >>>>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>>> retention
> >>>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
> >>>>>>> implies
> >>>>>>>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>>>>>>> retention, we also never documented (or failed)
> the
> >>>>>>> empty
> >>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
> >>>>>>> behavior
> >>>>>>>>>>> has
> >>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>>>>> existed
> >>>>>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> >>>>>>> introduced?),
> >>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a
> >> long
> >>>>>>> time.
> >>>>>>>>>> It
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> good
> >>>>>>>>>>>>>>>>>>>>>>>> to break the backward compatibility in a minor
> >>> release
> >>>>>>>>>>> version
> >>>>>>>>>>>>>>>>> (ex:
> >>>>>>>>>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
> >>>>>>>> existing
> >>>>>>>>>>>>>> bug,
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>>>> be a surprise to users.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >>>>>>> cleanup.policy
> >>>>>>>>>> by
> >>>>>>>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
> >> never
> >>>>>>>>>> realize
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in
> >>> 4.x
> >>>>>>> and
> >>>>>>>>>> in
> >>>>>>>>>>>>>> 5.0
> >>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>>>>> exception directly to make it fail fast
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> This suggestion looks good to me!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thank you.
> >>>>>>>>>>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> >>>>>>>>>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases,
> >> it
> >>> is
> >>>>>>>>>> worth
> >>>>>>>>>>>>>>>>>>>> considering
> >>>>>>>>>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had
> previously
> >>>>>>>>>>> encountered
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
> >>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>> instead.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >>>>>>>> cleanup.policy
> >>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
> >>> never
> >>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make
> >>> sense
> >>>>>>> for
> >>>>>>>>>>>>>> remote
> >>>>>>>>>>>>>>>>>>>>>>>> storage.
> >>>>>>>>>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with
> >> tiered
> >>>>>>>>>> storage
> >>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> valid
> >>>>>>>>>>>>>>>>>>>>>>>>> delete policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> >>>>>>> documented
> >>>>>>>>>>> that
> >>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In
> >> fact,
> >>>>> if
> >>>>>>>>>> one
> >>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
> >>>>>>> storage
> >>>>>>>>>>> since
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>> stop
> >>>>>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and
> >> time
> >>>>>>> too.
> >>>>>>>>>>> So,
> >>>>>>>>>>>>>>>>>> leaving
> >>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a
> user
> >>>>>>>> mistake
> >>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption,
> the
> >>>>> idea
> >>>>>>>>>>> behind
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
> >>>>> aware
> >>>>>>>> of
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> likely
> >>>>>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
> >>>>>>> wanting
> >>>>>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>>>>>>>> retention
> >>>>>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >>>>>>>> cleanup.policy
> >>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
> >>> never
> >>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
> >>>>>>>>>>> showuon@gmail.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Comments:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty
> >>> cleanup.policy
> >>>>>>> is
> >>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is
> >> justified."
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
> >>>>>>>> trade-off
> >>>>>>>>>>>>>>>>>> justified?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid
> >> in
> >>>>>>>>>> kafka,
> >>>>>>>>>>>>>> why
> >>>>>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>> provide
> >>>>>>>>>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this
> >> KIP?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to
> >> keep
> >>>>> all
> >>>>>>>>>>>>>> history
> >>>>>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>>>>>>>>>>>>> a cleanup policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast
> >> for
> >>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
> >>>>>>>>>>> "process.roles"
> >>>>>>>>>>>>>>>>>> configs
> >>>>>>>>>>>>>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>>>>>>>>>> they are invalid.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a
> >>> persuasive
> >>>>>>>>>>> reason
> >>>>>>>>>>>>>>> why
> >>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to
> >> configure
> >>>>>>>> Kafka
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> retain
> >>>>>>>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
> >>>>>>>>>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP,
> >>> but
> >>>>>>> to
> >>>>>>>>>> me,
> >>>>>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a
> >> clear
> >>>>>>> and
> >>>>>>>>>>>>>> direct
> >>>>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>>>>>>> kafka".
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really have to change the
> "cleanup.policy"
> >>>>>>>> config?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> >>>>>>>>>>>>>>>>> <jun@confluent.io.invalid
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to
> >> me.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
> >>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
> >> been
> >>>>>>>>>>> addressed
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
> >>>>>>>>>>>>>> Additionally,
> >>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list
> to
> >>> the
> >>>>>>>>>>>>>>> inNonEmpty
> >>>>>>>>>>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> >>> 2025年5月6日
> >>>>>>>>>>> 凌晨12:46
> >>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users
> >> pass
> >>>>>>> empty
> >>>>>>>>>>>>>>> config,
> >>>>>>>>>>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw
> >>>>> exception
> >>>>>>>>>>>>>>> directly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> >> 2025年5月6日
> >>>>> 週二
> >>>>>>>>>>>>>>>>> 上午12:28寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still
> >>> references
> >>>>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> deprecated
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be
> >>> removed
> >>>>>>> in
> >>>>>>>>>>> Kafka
> >>>>>>>>>>>>>>>>> 5.0"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
> >>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
> >>> been
> >>>>>>>>>>>>>> addressed
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> >>>>>>> 2025年5月3日
> >>>>>>>>>>>>>> 凌晨2:03
> >>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
> >>>>>>> Migration
> >>>>>>>>>>>>>> Plan",
> >>>>>>>>>>>>>>>>>>>>>>>>> could you
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> add
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
> >>>>>>>> "clean.policy="
> >>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> replaced
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to
> >>> disable
> >>>>>>>>>>> delete
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> compact.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> >>>>> 2025年5月3日
> >>>>>>>> 週六
> >>>>>>>>>>>>>>>>>>>>>>>> 上午1:55寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in
> >> for
> >>>>>>> now.
> >>>>>>>>>>>>>> Could
> >>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>>>>>> update
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references
> to
> >>>>>>>>>>>>>> deprecation.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> >>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
> >>>>>>>>>> ValidList#in
> >>>>>>>>>>>>>>>>>>>>>>>> method,
> >>>>>>>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
> >>>>>>> useful
> >>>>>>>>>> to
> >>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> methods:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces
> non-empty
> >>>>>>>> lists).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be
> >> useful
> >>> to
> >>>>>>>>>>>>>> document
> >>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
> >>> will
> >>>>>>>>>> fail
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>>>> 2025年5月2日
> >>>>>>>>>>>>>>>>> 凌晨12:50
> >>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be
> >> useful
> >>> to
> >>>>>>>>>>>>>> document
> >>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
> >>> will
> >>>>>>>>>> fail
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> >>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we
> >> need
> >>>>> to
> >>>>>>>>>>>>>> maintain
> >>>>>>>>>>>>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
> >>>>>>> isEmptyAllowed
> >>>>>>>>>>> flag
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of
> >>>>> isNonEmpty()
> >>>>>>>> to
> >>>>>>>>>>>>>>> remove
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID>
> 於
> >>>>>>>>>> 2025年5月1日
> >>>>>>>>>>>>>>>>> 凌晨4:17
> >>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed?
> >> It's
> >>>>>>>>>> awkward
> >>>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear
> when
> >>>>> it's
> >>>>>>>>>> set
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> true.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽
> <
> >>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
> >>>>>>>> introduce
> >>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is
> >> that
> >>>>> we
> >>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>> explicitly
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> throw a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
> >>>>>>> format
> >>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>>>>> phase
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
> >>>>>>>> behavior.”
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <jun@confluent.io.INVALID
> >
> >> 於
> >>>>>>>>>>>>>> 2025年4月25日
> >>>>>>>>>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
> >>>>>>> introduced
> >>>>>>>>>> by
> >>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be
> thrown
> >>>>>>> during
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
> >>>>>>> startup."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating
> >> the
> >>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>>> fails
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> group.coordinator.rebalance.protocols
> >>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM
> >> 黃竣陽 <
> >>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this
> >> change:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> >>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup
> >> phase
> >>> to
> >>>>>>>> the
> >>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an
> >>> empty
> >>>>>>>>>> list
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException
> during
> >>>>>>>> storage
> >>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
> <jun@confluent.io.INVALID
> >>>
> >>> 於
> >>>>>>>>>>>>>>> 2025年4月24日
> >>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:44
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> >>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty
> >> right
> >>>>>>> now.
> >>>>>>>>>>> With
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and
> >> more
> >>>>>>>>>>> accurate)
> >>>>>>>>>>>>>>>>>>>>>>>>> exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind
> >> of
> >>>>>>>> agree
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more
> >>>>> likely
> >>>>>>>> to
> >>>>>>>>>>> be
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>> mistake
> >>>>>>>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as
> >>> None,
> >>>>>>>> the
> >>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>> never
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception
> >> will
> >>>>>>> let
> >>>>>>>>>> the
> >>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
> >>>>>>>> Chia-Ping
> >>>>>>>>>>>>>> Tsai
> >>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I
> >>> understand
> >>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> maintaining
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
> >>>>>>>> documentation,
> >>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>>>>>> allow
> >>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence,
> >> should
> >>> we
> >>>>>>>>>>>>>> consider
> >>>>>>>>>>>>>>>>>>>>>>>>> throwing
> >>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x?
> This
> >>>>>>> could
> >>>>>>>>>>>>>>>>> streamline
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
> >> <jun@confluent.io.invalid
> >>>>
> >>>>> 於
> >>>>>>>>>>>>>>>>> 2025年4月23日
> >>>>>>>>>>>>>>>>>>>>>>>> 週三
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy
> will
> >>> not
> >>>>>>>>>>> delete
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>> compact
> >>>>>>>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We
> >> won't
> >>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>> segments
> >>>>>>>>>>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
> >>>>>>> log.retention.ms,
> >>>>>>>>>>> but
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> continue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> >>>>>>>>>>>>>> log.local.retention.bytes/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk
> >>>>> space
> >>>>>>>>>> when
> >>>>>>>>>>>>>>>>> remote
> >>>>>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at
> 9:45 AM
> >>> Jun
> >>>>>>>>>> Rao <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
> >>>>>>> values
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> group.coordinator.rebalance.protocols
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at
> >> 7:29 AM
> >>>>>>> 黃竣陽 <
> >>>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
> >>>>>>>>>>> accordingly
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> revised
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none
> >> policy
> >>>>>>>>>> cannot
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an
> >>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the
> >>> preProcessParsedConfig
> >>>>>>>>>> step.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform
> >> users
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
> >>>>> <jun@confluent.io.INVALID
> >>>>>>>>
> >>>>>>>>>> 於
> >>>>>>>>>>>>>>>>>>>>>>>> 2025年4月22日
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:52
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
> >>>>>>> comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a
> >>> new
> >>>>>>>>>> value
> >>>>>>>>>>>>>> None
> >>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are
> >>>>> valid.
> >>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>>>>>> example,
> >>>>>>>>>>>>>>>>>>>>>>>>> None
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would
> >> be
> >>>>>>>>>> useful
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> document
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior
> >> during
> >>>>>>>>>> upgrade
> >>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's
> >> not
> >>>>>>> clear
> >>>>>>>>>>> what
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> check
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at
> >>> 8:25 AM
> >>>>>>> 黃竣陽
> >>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> >>>>>>> discussion
> >>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>> KIP-1161:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
> >> improve
> >>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be
> >>> left
> >>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
> >>>>>>> proposed
> >>>>>>>>>>>>>>>>>>>>>>>> improvements:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to
> >>> validate
> >>>>>>>>>>> whether
> >>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none'
> >>> value
> >>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun, Chia-Ping,

Thanks for the reply.

JR28:
You're right. If the user passes a null value, it will be converted to an empty list. 
I’ve updated the KIP accordingly.

JR29:
For early.start.listeners, the exception is thrown during Kafka’s internal validation, 
and it’s not caused by a null or empty list. Therefore, I think we should 
set the validator to none. I’ve updated the KIP.

JR30 / chia04:
Currently, passing a null value results in a NullPointerException. I’ve updated
 the KIP to reflect this behavior.

JR31:
Since the default value of plugin.path is null, it indicates that the user has not set this configuration.
Changing it to an empty list might be confusing, as it implies the user explicitly set the config but provided no paths.
However, if using an empty list is more appropriate for this config, I’m fine with changing it.
We can set the default value to List.of() and disallow null values.

JR32:
An empty value for partition.assignment.strategy will result in an 
IllegalStateException. I’ve updated the KIP.

chia00:
This config should not be null. I’ve updated the KIP to reflect that.

chia01 / chia02:
These configurations should be both non-null and non-empty. 
I’ve updated them to only allow non-empty lists.

chia03:
Currently, passing a null value will throw a NullPointerException. 
I’ve updated the KIP accordingly.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月17日 凌晨2:37 寫道：
> 
> hi Jiunn-Yang,
> 
> chia00. why could `bootstrap.servers` be null in StreamsConfig?
> 
> chia01. the empty `task.assigned.partitions` is a bit weird. That means
> consumer has nothing to seek/poll.
> 
> chia02. task.assigned.groups has analogous issue.
> 
> chia03. MirrorCheckpointConfig's `groups` and `groups.exclude` should not
> accept null, since they are complied by regular expression.
> 
> chia04. MirrorSourceConfig's `topics` and `topics.exclude` have similar
> issue.
> 
> Best,
> Chia-Ping
> 
> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月17日 週四 上午12:51寫道：
> 
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the reply. A few more comments.
>> 
>> JR28. AdminClientConfig: Currently, bootstrap.servers doesn't throw
>> NullPointerException on null value. It's converted to an empty list.
>> 
>> JR29. The "Before exception" column: It would be useful to add which value
>> causes an exception. For example, duplicated values currently don't
>> typically cause exceptions. Also, for early.start.listeners, which value
>> causes ConfigException currently?
>> 
>> JR30. Why do we allow the following to be null? In general, null only makes
>> sense if it's the default value.
>> MirrorSourceConfig: topics, topics.exclude, config.properties.exclude
>> 
>> JR31. WorkerConfig: plugin.path It seems that we should allow it to be
>> empty, but not null?
>> 
>> JR32. partition.assignment.strategy shouldn't be empty, right?
>> 
>> Jun
>> 
>> On Wed, Jul 16, 2025 at 4:32 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hi Jun,
>>> 
>>> Thanks for the reply.
>>> 
>>> JR26: I’ve addressed this change in the KIP.
>>> 
>>> JR27:
>>> I’ve documented the following configurations in the KIP:
>>> 
>>> ConsumerConfig
>>> partition.assignment.strategy
>>> 
>>> StreamsConfig
>>> bootstrap.servers
>>> 
>>> MirrorCheckpointConfig
>>> groups
>>> groups.exclude
>>> task.assigned.groups
>>> 
>>> MirrorSourceConfig
>>> topics
>>> topics.exclude
>>> config.properties.exclude
>>> 
>>> WorkerConfig
>>> plugin.path
>>> 
>>> RestServerConfig
>>> rest.extension.classes
>>> 
>>> Other configurations already have default values and corresponding
>>> validators:
>>> 
>>> LogConfig
>>> leader.replication.throttled.replicas: List.of() →
>>> ThrottledReplicaListValidator
>>> follower.replication.throttled.replicas: List.of() →
>>> ThrottledReplicaListValidator
>>> 
>>> StreamsConfig
>>> rack.aware.assignment.tags: List.of() → atMostOfSize
>>> 
>>> ConnectorConfig
>>> transforms: List.of() → aliasValidator
>>> predicates: List.of() → aliasValidator
>>> 
>>> SourceConnectorConfig
>>> topic.creation.groups: List.of() → ConfigDef.CompositeValidator
>>> 
>>> DistributedConfig
>>> inter.worker.verification.algorithms:
>>> defaultVerificationAlgorithms(crypto) → ConfigDef.LambdaValidator
>>> 
>>> RestServerConfig
>>> admin.listeners: null → AdminListenersValidator
>>> listeners: List.of("http://:8083") → ListenersValidator
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月16日 清晨7:14 寫道：
>>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> JR 26. early.start.listeners and interceptor.classes: It seems that
>> both
>>>> could be empty?
>>>> 
>>>> JR27. The following configs of type LIST are missing.
>>>> 
>>>> ConsumerConfig
>>>> partition.assignment.strategy
>>>> 
>>>> BrokerSecurityConfigs
>>>> 
>>>> QuorumConfig
>>>> 
>>>> LogConfig
>>>> leader.replication.throttled.replicas
>>>> follower.replication.throttled.replicas
>>>> 
>>>> StreamsConfig
>>>> rack.aware.assignment.tags
>>>> bootstrap.servers
>>>> 
>>>> MirrorCheckpointConfig
>>>> groups
>>>> groups.exclude
>>>> 
>>>> MirrorCheckpointConfig
>>>> task.assigned.groups
>>>> 
>>>> MirrorSourceConfig
>>>> topics
>>>> topics.exclude
>>>> config.properties.exclude
>>>> 
>>>> ConnectorConfig
>>>> transforms
>>>> predicates
>>>> 
>>>> SourceConnectorConfig
>>>> topic.creation.groups
>>>> 
>>>> WorkerConfig
>>>> plugin.path
>>>> 
>>>> DistributedConfig
>>>> inter.worker.verification.algorithms
>>>> 
>>>> RestServerConfig
>>>> rest.extension.classes
>>>> admin.listeners
>>>> listeners
>>>> 
>>>> Thanks,
>>>> 
>>>> Jun
>>>> 
>>>> On Tue, Jul 15, 2025 at 6:22 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hello Jun Chia,
>>>>> 
>>>>> Thanks for the reply,
>>>>> 
>>>>> JR20: This configuration should remain unchanged. I have updated the
>>> KIP.
>>>>> 
>>>>> JR21: For both Consumer and Producer, the default value is an empty
>>> list,
>>>>> and the
>>>>> validator used is NonNullValidator. I think we can change the
>> validator
>>> to
>>>>> ValidList.anyNonDuplicateValues
>>>>> with isNullAllowed = false and isEmptyAllowed = false.
>>>>> 
>>>>> JR23: I’ve addressed this change in the KIP. Although the default
>> value
>>> of
>>>>> early.start.listeners remains
>>>>> unchanged, the validator now disallows empty lists.
>>>>> 
>>>>> JR24: Since log.dirs does not change its default value, I used a dash
>>> (-)
>>>>> to indicate that the field remains unchanged.
>>>>> 
>>>>> JR25: I’ve updated the table and aligned similar validators as much as
>>>>> possible for consistency.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月15日 下午2:12 寫道：
>>>>>> 
>>>>>>> JR20. controller.listener.names : It's required on the broker. Is it
>>>>>> required on the controller?
>>>>>> 
>>>>>> there is a small "sugar" in the controller, which we will
>>>>>> create listener.security.protocol.map based on
>>> controller.listener.names
>>>>>> if listener.security.protocol.map is not explicitly set [0]
>>>>>> Additionally, `controller.listener.names` is also used to filter out
>>> the
>>>>>> advertised controller listeners [1]
>>>>>> 
>>>>>> 
>>>>>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be
>> set
>>> to
>>>>>> null, right?
>>>>>> 
>>>>>> I don't think so. There is no null check after calling
>>>>>> 
>>>>> 
>>> 
>> `getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
>>>>>> which causes an NPE if getList returns null."
>>>>>> 
>>>>>> [0]
>>>>>> 
>>>>> 
>>> 
>> https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
>>>>>> [1]
>>>>>> 
>>>>> 
>>> 
>> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
>>>>>> [2]
>>>>>> 
>>>>> 
>>> 
>> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
>>>>>> 
>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：
>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the updated KIP.
>>>>>>> 
>>>>>>> JR20. controller.listener.names : It's required on the broker. Is it
>>>>>>> required on the controller?
>>>>>>> 
>>>>>>> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer)
>>>>> already
>>>>>>> require it to be non-null.
>>>>>>> 
>>>>>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be
>> set
>>> to
>>>>>>> null, right?
>>>>>>> 
>>>>>>> JR23. early.start.listeners: It allows default value of null, which
>>>>> means
>>>>>>> that all listeners included in controller.listener.names will also
>> be
>>>>> early
>>>>>>> start listeners.
>>>>>>> 
>>>>>>> JR24. log.dirs: It currently has a default value of null.
>>>>>>> 
>>>>>>> JR25. It seems there are quite a few more configs of type list. Do
>>> they
>>>>>>> need to be changed too?
>>>>>>> 
>>>>>>> AdminClientConfig: 4
>>>>>>> ConsumerConfig: 5
>>>>>>> ProducerConfig: 4
>>>>>>> SaslConfigs: 1
>>>>>>> SslConfigs: 2
>>>>>>> BrokerSecurityConfigs: 5
>>>>>>> MirrorClientConfig: 1
>>>>>>> org.apache.kafka.connect.mirror : 16
>>>>>>> org.apache.kafka.connect.runtime : 21
>>>>>>> GroupCoordinatorConfig: 3
>>>>>>> QuorumConfig: 2
>>>>>>> ServerConfigs: 1
>>>>>>> KRaftConfigs: 1
>>>>>>> ClientMetricsConfigs: 2
>>>>>>> MetricConfigs: 2
>>>>>>> LogConfig: 4
>>>>>>> StreamsConfig: 4
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hi Chia, Jun
>>>>>>>> 
>>>>>>>> Thanks for the reply,
>>>>>>>> I have updated the table in the KIP.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
>>>>>>>>> 
>>>>>>>>> Hi, Jun
>>>>>>>>> 
>>>>>>>>> Thanks for the reply.
>>>>>>>>> 
>>>>>>>>> I think the following configs could use null as the default value
>>> and
>>>>>>>> treat
>>>>>>>>> an "empty list" as an illegal value.
>>>>>>>>> 
>>>>>>>>> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
>>>>>>>>> `ssl.enabled.protocols`,
>>>>>>>>> and `log.dirs`
>>>>>>>>> 
>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
>>>>>>>>> 
>>>>>>>>>> Hi, Chia-Ping, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply.
>>>>>>>>>> 
>>>>>>>>>> As you said, if ssl.cipher.suites is empty, intuitively this
>> means
>>>>>>> that
>>>>>>>> no
>>>>>>>>>> cipher is allowed. So, this should be an illegal config. It just
>>>>>>> happens
>>>>>>>>>> that our implementation treats it as if the config is not set
>>>>> (meaning
>>>>>>>> the
>>>>>>>>>> value is null). So, it seems that a more intuitive convention is:
>>> if
>>>>>>> the
>>>>>>>>>> config is not set, it defaults to all available cipher suites. If
>>>>> it's
>>>>>>>>>> explicitly set, it can't be empty.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Jun, Chia
>>>>>>>>>>> 
>>>>>>>>>>>>> Also, it would be useful to add a compatibility column in the
>>>>>>> table.
>>>>>>>>>> If
>>>>>>>>>>> a
>>>>>>>>>>>>> value is formally disallowed, it would be useful to compare
>> the
>>>>> old
>>>>>>>>>> and
>>>>>>>>>>> the
>>>>>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>>>>>>>> I’ve updated the table to show which exceptions will be thrown
>> for
>>>>>>> each
>>>>>>>>>>> behavior.
>>>>>>>>>>> 
>>>>>>>>>>>>>> I don't understand the point of "5.0". Could you please share
>>>>> more
>>>>>>>>>>> details?
>>>>>>>>>>>>>> It seems to me changing the type or default value should be
>>> fine
>>>>>>> in
>>>>>>>>>>> minor
>>>>>>>>>>>>>> release if we don't break the existing property files.
>>>>>>>>>>> I don't think this requires any compatibility considerations, so
>>>>> I’ve
>>>>>>>>>>> removed the Kafka 5.0 changes section.
>>>>>>>>>>> 
>>>>>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type,
>> so
>>>>>>> using
>>>>>>>>>>> `LIST`
>>>>>>>>>>>>>> instead of `String` does not seems to make sense.
>>>>>>>>>>> I have removed the config from KIP
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>>> For ssl.cipher.suites, changing the default value from null to
>>>>>>> empty
>>>>>>>>>> is
>>>>>>>>>>>>> actually a breaking change. According to the doc, null means
>>> that
>>>>>>> all
>>>>>>>>>>> the
>>>>>>>>>>>>> available cipher suites are supported. I am thinking that we
>>>>> should
>>>>>>>>>>>>> continue to allow null as the default since it could
>> represent a
>>>>>>>> state
>>>>>>>>>>>>> different from empty.
>>>>>>>>>>>> 
>>>>>>>>>>>> "Empty list" seems to be a weird case. It appears to signify
>> that
>>>>> no
>>>>>>>>>>> cipher suites are accepted. However, in the codebase, an empty
>>> list
>>>>>>> is
>>>>>>>>>>> handled as if all available cipher suites are supported [0]. We
>>>>>>> should
>>>>>>>>>> not
>>>>>>>>>>> support the case of "no supported cipher suite," as it doesn't
>>> make
>>>>>>>>>> sense.
>>>>>>>>>>>> 
>>>>>>>>>>>> In short, changing the default value from null to empty does
>> not
>>>>>>> break
>>>>>>>>>>> the behavior.
>>>>>>>>>>>> 
>>>>>>>>>>>> [0]
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
>>>>>>>>>>>> 
>>>>>>>>>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be
>> empty,
>>> we
>>>>>>>>>> will
>>>>>>>>>>>>> just continue to allow it in all future releases.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> For ssl.cipher.suites, changing the default value from null to
>>>>>>> empty
>>>>>>>>>> is
>>>>>>>>>>>>> actually a breaking change. According to the doc, null means
>>> that
>>>>>>> all
>>>>>>>>>>> the
>>>>>>>>>>>>> available cipher suites are supported. I am thinking that we
>>>>> should
>>>>>>>>>>>>> continue to allow null as the default since it could
>> represent a
>>>>>>>> state
>>>>>>>>>>>>> different from empty.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Also, it would be useful to add a compatibility column in the
>>>>>>> table.
>>>>>>>>>> If
>>>>>>>>>>> a
>>>>>>>>>>>>> value is formally disallowed, it would be useful to compare
>> the
>>>>> old
>>>>>>>>>> and
>>>>>>>>>>> the
>>>>>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <
>>> chia7712@gmail.com
>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the explanation. I have updated the KIP
>> accordingly:
>>>>> in
>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>> 4.x, cleanup.policy
>>>>>>>>>>>>>>> can be set to an empty list with a warning message emitted;
>>>>>>>> however,
>>>>>>>>>>> in
>>>>>>>>>>>>>>> Kafka 5.0, setting an
>>>>>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If we agree to support "empty list", the approach of "none"
>>> could
>>>>>>> be
>>>>>>>>>>>>>> eliminated, right?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I don't understand the point of "5.0". Could you please share
>>>>> more
>>>>>>>>>>> details?
>>>>>>>>>>>>>> It seems to me changing the type or default value should be
>>> fine
>>>>>>> in
>>>>>>>>>>> minor
>>>>>>>>>>>>>> release if we don't break the existing property files.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type,
>> so
>>>>>>> using
>>>>>>>>>>> `LIST`
>>>>>>>>>>>>>> instead of `String` does not seems to make sense.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Null could make sense as the default value, but it just
>>> means
>>>>>>>> that
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> config key is not set explicitly.
>>>>>>>>>>>>>>>>> Given that, it's probably simpler to just allow
>>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>>>>> Thanks for the explanation. I have updated the KIP
>>> accordingly:
>>>>>>> in
>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>> 4.x, cleanup.policy
>>>>>>>>>>>>>>> can be set to an empty list with a warning message emitted;
>>>>>>>> however,
>>>>>>>>>>> in
>>>>>>>>>>>>>>> Kafka 5.0, setting an
>>>>>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> For another, there is an additional issue which could be
>>>>> handled
>>>>>>>> by
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
>>> as a
>>>>>>>>>> LIST.
>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`.
>> Could
>>>>> we
>>>>>>>>>>> change
>>>>>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
>>>>> them
>>>>>>>>>>>>>>>> non-nullable.
>>>>>>>>>>>>>>> I have also updated the KIP to include the tables detailing
>>> the
>>>>>>>>>>> changes
>>>>>>>>>>>>>>> for LIST-type configuration
>>>>>>>>>>>>>>> default values and the migration from STRING-type to
>> LIST-type
>>>>>>>>>>>>>>> configurations.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
>>>>>>> allows
>>>>>>>>>> an
>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's
>> ok
>>>>> to
>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler
>> to
>>>>>>> just
>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> yes, that is what I meant before. Additionally, the `LIST`
>>> type
>>>>>>>>>>> should
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>> accept "null". Instead, we should use an empty list as the
>>>>>>> default
>>>>>>>>>>>>>> value.
>>>>>>>>>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should
>>> never
>>>>>>> has
>>>>>>>>>>>>>> "null"
>>>>>>>>>>>>>>>> value. That could eliminate the null check.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> For another, there is an additional issue which could be
>>>>> handled
>>>>>>>> by
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
>>> as a
>>>>>>>>>> LIST.
>>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`.
>> Could
>>>>> we
>>>>>>>>>>> change
>>>>>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
>>>>> them
>>>>>>>>>>>>>>>> non-nullable.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二
>> 上午5:56寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR10. Regarding supporting the null value, I am not sure
>> how
>>>>>>> one
>>>>>>>>>>> could
>>>>>>>>>>>>>>> set
>>>>>>>>>>>>>>>>> a null value in a config file. For example, if you set the
>>>>>>>>>> following
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> config file.
>>>>>>>>>>>>>>>>> configkey=
>>>>>>>>>>>>>>>>> The value of configkey will be an empty list, instead of
>>> null,
>>>>>>>>>>> right?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Null could make sense as the default value, but it just
>>> means
>>>>>>>> that
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> config key is not set explicitly.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
>>>>>>> allows
>>>>>>>>>> an
>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's
>> ok
>>>>> to
>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler
>> to
>>>>>>> just
>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional,
>>>>> that
>>>>>>>> is
>>>>>>>>>>>>>>> right,
>>>>>>>>>>>>>>>>>> so I think
>>>>>>>>>>>>>>>>>> that we can allow null value, but not allow the empty
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> JR11: In this KIP, the goal is to address configurations
>>> that
>>>>>>>>>>> should
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> accept empty arrays.
>>>>>>>>>>>>>>>>>> We want to prevent users from passing in an empty list
>> when
>>>>>>> it's
>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> valid. I believe there are two
>>>>>>>>>>>>>>>>>> main scenarios to consider:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 1. Nullable but not empty:
>>>>>>>>>>>>>>>>>> The parameter allows either a null value or a non-empty
>>> list.
>>>>>>> In
>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> case,
>>>>>>>>>>>>>>>>>> if the user provides an empty list, we should throw a
>>>>>>>>>>>>>> ConfigException.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
>>>>>>>>>> contain a
>>>>>>>>>>>>>>>>>> non-empty list — null is also
>>>>>>>>>>>>>>>>>> invalid. In this case, if the user provides either a null
>>> or
>>>>>>> an
>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>> list, we should throw a ConfigException.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Therefore, I want to prevent scenarios where users can
>> pass
>>>>> in
>>>>>>>> an
>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
>>>>>>> allowNull
>>>>>>>>>>>>>>>>> property.
>>>>>>>>>>>>>>>>>> This allows us to specify
>>>>>>>>>>>>>>>>>> whether a configuration permits null values or not.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> JR13: I have updated the table
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30
>> 寫道：
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Sorry for the late reply. A few more comments.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience
>> says
>>>>>>> that
>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>> optional. So an empty list seems to be valid.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support
>>> empty
>>>>>>>>>> lists
>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> general, it's probably more consistent to just allow
>>>>>>>>>>>>>>> log.cleanup.policy
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> be empty as Luke suggested, instead of introducing a new
>>>>>>> option
>>>>>>>>>>>>>> None.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null.
>>> It
>>>>>>>>>> seems
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
>>>>>>>> shouldn't
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> null.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> JR13. The table is very helpful. Could you also describe
>>> the
>>>>>>>>>>> old/new
>>>>>>>>>>>>>>>>>>> behavior for each config with an empty list or duplicate
>>>>>>> entry?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7133700@gmail.com
>>> 
>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I am manually bumping this thread. Any feedback would
>> be
>>>>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
>>>>>>>>>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
>>>>>>>>>>>>>>>>>>>>> provided list is either null or a non-empty list
>> without
>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>>>>>> entries. I’ve also listed the configurations that will
>>>>>>>>>>>>>>>>>>>>> be validated using this logic. Please feel free to
>> share
>>>>>>> any
>>>>>>>>>>>>>>> feedback
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日
>> 凌晨2:29
>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> It seems there are quite a few other configs of type
>>> list
>>>>>>>>>> (e.g.
>>>>>>>>>>>>>>>>>> several
>>>>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>>>>>>> related ones). It would be useful to understand if
>>> empty
>>>>>>>>>> lists
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>>>>>>> for them or not.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
>>>>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
>>>>> introduced?),
>>>>>>>>>>> there
>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
>>>>>>> time.
>>>>>>>> "
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> If you do a google search on "kafka infinite
>>> retention",
>>>>>>>> you
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>> links like
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>>> both pointing to setting the retention time and
>>>>> retention
>>>>>>>>>> size
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> -1
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
>>>>>>>> intentionally
>>>>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>>>> retention, it's more likely they will use that
>>> approach
>>>>>>>>>>> instead
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>> setting
>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our
>>>>> API/tools
>>>>>>>>>> make
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving
>>> the
>>>>>>>>>>> config
>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
>>>>>>>>>> showuon@gmail.com
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
>>>>>>> documented
>>>>>>>>>>> that
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact,
>>> if
>>>>>>> one
>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
>>> storage
>>>>>>>>>> since
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
>>>>> too.
>>>>>>>>>> So,
>>>>>>>>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
>>>>>>> mistake
>>>>>>>>>>> than
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
>>> idea
>>>>>>>>>> behind
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
>>> aware
>>>>>>> of
>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
>>> wanting
>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
>>>>>>> implies
>>>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>>>>> retention, we also never documented (or failed) the
>>>>>>> empty
>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
>>>>>>> behavior
>>>>>>>>>>> has
>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>>>>> existed
>>>>>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
>>>>>>> introduced?),
>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a
>> long
>>>>>>> time.
>>>>>>>>>> It
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>>>>>>>>>>> to break the backward compatibility in a minor
>>> release
>>>>>>>>>>> version
>>>>>>>>>>>>>>>>> (ex:
>>>>>>>>>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
>>>>>>>> existing
>>>>>>>>>>>>>> bug,
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>> be a surprise to users.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>>>>> cleanup.policy
>>>>>>>>>> by
>>>>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
>> never
>>>>>>>>>> realize
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in
>>> 4.x
>>>>>>> and
>>>>>>>>>> in
>>>>>>>>>>>>>> 5.0
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>>>>> exception directly to make it fail fast
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> This suggestion looks good to me!
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
>>>>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases,
>> it
>>> is
>>>>>>>>>> worth
>>>>>>>>>>>>>>>>>>>> considering
>>>>>>>>>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
>>>>>>>>>>> encountered
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>> instead.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>>>>>> cleanup.policy
>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
>>> never
>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make
>>> sense
>>>>>>> for
>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with
>> tiered
>>>>>>>>>> storage
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>>>>>>>>>> delete policy.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
>>>>>>> documented
>>>>>>>>>>> that
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In
>> fact,
>>>>> if
>>>>>>>>>> one
>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
>>>>>>> storage
>>>>>>>>>>> since
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and
>> time
>>>>>>> too.
>>>>>>>>>>> So,
>>>>>>>>>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
>>>>>>>> mistake
>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
>>>>> idea
>>>>>>>>>>> behind
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
>>>>> aware
>>>>>>>> of
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
>>>>>>> wanting
>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>>>>>> cleanup.policy
>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
>>> never
>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
>>>>>>>>>>> showuon@gmail.com
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Comments:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty
>>> cleanup.policy
>>>>>>> is
>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is
>> justified."
>>>>>>>>>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
>>>>>>>> trade-off
>>>>>>>>>>>>>>>>>> justified?
>>>>>>>>>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid
>> in
>>>>>>>>>> kafka,
>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this
>> KIP?
>>>>>>>>>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to
>> keep
>>>>> all
>>>>>>>>>>>>>> history
>>>>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>>>>>>> a cleanup policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast
>> for
>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
>>>>>>>>>>> "process.roles"
>>>>>>>>>>>>>>>>>> configs
>>>>>>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>>>>>>> they are invalid.
>>>>>>>>>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a
>>> persuasive
>>>>>>>>>>> reason
>>>>>>>>>>>>>>> why
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
>>>>>>>>>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to
>> configure
>>>>>>>> Kafka
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> retain
>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
>>>>>>>>>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP,
>>> but
>>>>>>> to
>>>>>>>>>> me,
>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a
>> clear
>>>>>>> and
>>>>>>>>>>>>>> direct
>>>>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>>>>> kafka".
>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy"
>>>>>>>> config?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
>>>>>>>>>>>>>>>>> <jun@confluent.io.invalid
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to
>> me.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
>>>>>>>> s7133700@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
>> been
>>>>>>>>>>> addressed
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
>>>>>>>>>>>>>> Additionally,
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to
>>> the
>>>>>>>>>>>>>>> inNonEmpty
>>>>>>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>>> 2025年5月6日
>>>>>>>>>>> 凌晨12:46
>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users
>> pass
>>>>>>> empty
>>>>>>>>>>>>>>> config,
>>>>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw
>>>>> exception
>>>>>>>>>>>>>>> directly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>> 2025年5月6日
>>>>> 週二
>>>>>>>>>>>>>>>>> 上午12:28寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still
>>> references
>>>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be
>>> removed
>>>>>>> in
>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>> 5.0"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
>>> been
>>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>>>>>>> 2025年5月3日
>>>>>>>>>>>>>> 凌晨2:03
>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
>>>>>>> Migration
>>>>>>>>>>>>>> Plan",
>>>>>>>>>>>>>>>>>>>>>>>>> could you
>>>>>>>>>>>>>>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
>>>>>>>> "clean.policy="
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> replaced
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to
>>> disable
>>>>>>>>>>> delete
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> compact.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>>>>> 2025年5月3日
>>>>>>>> 週六
>>>>>>>>>>>>>>>>>>>>>>>> 上午1:55寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in
>> for
>>>>>>> now.
>>>>>>>>>>>>>> Could
>>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
>>>>>>>>>>>>>> deprecation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
>>>>>>>>>> ValidList#in
>>>>>>>>>>>>>>>>>>>>>>>> method,
>>>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
>>>>>>> useful
>>>>>>>>>> to
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty
>>>>>>>> lists).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be
>> useful
>>> to
>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
>>> will
>>>>>>>>>> fail
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>> 2025年5月2日
>>>>>>>>>>>>>>>>> 凌晨12:50
>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be
>> useful
>>> to
>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
>>> will
>>>>>>>>>> fail
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we
>> need
>>>>> to
>>>>>>>>>>>>>> maintain
>>>>>>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
>>>>>>> isEmptyAllowed
>>>>>>>>>>> flag
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of
>>>>> isNonEmpty()
>>>>>>>> to
>>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> boolean
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>>>> 2025年5月1日
>>>>>>>>>>>>>>>>> 凌晨4:17
>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed?
>> It's
>>>>>>>>>> awkward
>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when
>>>>> it's
>>>>>>>>>> set
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> true.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
>>>>>>>> introduce
>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is
>> that
>>>>> we
>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>> explicitly
>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
>>>>>>> format
>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>> phase
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
>>>>>>>> behavior.”
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID>
>> 於
>>>>>>>>>>>>>> 2025年4月25日
>>>>>>>>>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
>>>>>>> introduced
>>>>>>>>>> by
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown
>>>>>>> during
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
>>>>>>> startup."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating
>> the
>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>> group.coordinator.rebalance.protocols
>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM
>> 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this
>> change:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup
>> phase
>>> to
>>>>>>>> the
>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an
>>> empty
>>>>>>>>>> list
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during
>>>>>>>> storage
>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <jun@confluent.io.INVALID
>>> 
>>> 於
>>>>>>>>>>>>>>> 2025年4月24日
>>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:44
>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty
>> right
>>>>>>> now.
>>>>>>>>>>> With
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and
>> more
>>>>>>>>>>> accurate)
>>>>>>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind
>> of
>>>>>>>> agree
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more
>>>>> likely
>>>>>>>> to
>>>>>>>>>>> be
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> mistake
>>>>>>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as
>>> None,
>>>>>>>> the
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception
>> will
>>>>>>> let
>>>>>>>>>> the
>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
>>>>>>>> Chia-Ping
>>>>>>>>>>>>>> Tsai
>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I
>>> understand
>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> maintaining
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
>>>>>>>> documentation,
>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence,
>> should
>>> we
>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This
>>>>>>> could
>>>>>>>>>>>>>>>>> streamline
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
>> <jun@confluent.io.invalid
>>>> 
>>>>> 於
>>>>>>>>>>>>>>>>> 2025年4月23日
>>>>>>>>>>>>>>>>>>>>>>>> 週三
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will
>>> not
>>>>>>>>>>> delete
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>> compact
>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We
>> won't
>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>> segments
>>>>>>>>>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
>>>>>>> log.retention.ms,
>>>>>>>>>>> but
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
>>>>>>>>>>>>>> log.local.retention.bytes/
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk
>>>>> space
>>>>>>>>>> when
>>>>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM
>>> Jun
>>>>>>>>>> Rao <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
>>>>>>> values
>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at
>> 7:29 AM
>>>>>>> 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
>>>>>>>>>>> accordingly
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> revised
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none
>> policy
>>>>>>>>>> cannot
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an
>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy,
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the
>>> preProcessParsedConfig
>>>>>>>>>> step.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Additionally, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform
>> users
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
>>>>> <jun@confluent.io.INVALID
>>>>>>>> 
>>>>>>>>>> 於
>>>>>>>>>>>>>>>>>>>>>>>> 2025年4月22日
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:52
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
>>>>>>> comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a
>>> new
>>>>>>>>>> value
>>>>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are
>>>>> valid.
>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would
>> be
>>>>>>>>>> useful
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior
>> during
>>>>>>>>>> upgrade
>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's
>> not
>>>>>>> clear
>>>>>>>>>>> what
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at
>>> 8:25 AM
>>>>>>> 黃竣陽
>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
>>>>>>> discussion
>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>> KIP-1161:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
>> improve
>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be
>>> left
>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
>>>>>>> proposed
>>>>>>>>>>>>>>>>>>>>>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to
>>> validate
>>>>>>>>>>> whether
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none'
>>> value
>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

hi Jiunn-Yang,

chia00. why could `bootstrap.servers` be null in StreamsConfig?

chia01. the empty `task.assigned.partitions` is a bit weird. That means
consumer has nothing to seek/poll.

chia02. task.assigned.groups has analogous issue.

chia03. MirrorCheckpointConfig's `groups` and `groups.exclude` should not
accept null, since they are complied by regular expression.

chia04. MirrorSourceConfig's `topics` and `topics.exclude` have similar
issue.

Best,
Chia-Ping

Jun Rao <ju...@confluent.io.invalid> 於 2025年7月17日 週四 上午12:51寫道：

> Hi, Jiunn-Yang,
>
> Thanks for the reply. A few more comments.
>
> JR28. AdminClientConfig: Currently, bootstrap.servers doesn't throw
> NullPointerException on null value. It's converted to an empty list.
>
> JR29. The "Before exception" column: It would be useful to add which value
> causes an exception. For example, duplicated values currently don't
> typically cause exceptions. Also, for early.start.listeners, which value
> causes ConfigException currently?
>
> JR30. Why do we allow the following to be null? In general, null only makes
> sense if it's the default value.
> MirrorSourceConfig: topics, topics.exclude, config.properties.exclude
>
> JR31. WorkerConfig: plugin.path It seems that we should allow it to be
> empty, but not null?
>
> JR32. partition.assignment.strategy shouldn't be empty, right?
>
> Jun
>
> On Wed, Jul 16, 2025 at 4:32 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > Thanks for the reply.
> >
> > JR26: I’ve addressed this change in the KIP.
> >
> > JR27:
> > I’ve documented the following configurations in the KIP:
> >
> > ConsumerConfig
> > partition.assignment.strategy
> >
> > StreamsConfig
> > bootstrap.servers
> >
> > MirrorCheckpointConfig
> > groups
> > groups.exclude
> > task.assigned.groups
> >
> > MirrorSourceConfig
> > topics
> > topics.exclude
> > config.properties.exclude
> >
> > WorkerConfig
> > plugin.path
> >
> > RestServerConfig
> > rest.extension.classes
> >
> > Other configurations already have default values and corresponding
> > validators:
> >
> > LogConfig
> > leader.replication.throttled.replicas: List.of() →
> > ThrottledReplicaListValidator
> > follower.replication.throttled.replicas: List.of() →
> > ThrottledReplicaListValidator
> >
> > StreamsConfig
> > rack.aware.assignment.tags: List.of() → atMostOfSize
> >
> > ConnectorConfig
> > transforms: List.of() → aliasValidator
> > predicates: List.of() → aliasValidator
> >
> > SourceConnectorConfig
> > topic.creation.groups: List.of() → ConfigDef.CompositeValidator
> >
> > DistributedConfig
> > inter.worker.verification.algorithms:
> > defaultVerificationAlgorithms(crypto) → ConfigDef.LambdaValidator
> >
> > RestServerConfig
> > admin.listeners: null → AdminListenersValidator
> > listeners: List.of("http://:8083") → ListenersValidator
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月16日 清晨7:14 寫道：
> > >
> > > Hi, Jiunn-Yang,
> > >
> > > Thanks for the reply.
> > >
> > > JR 26. early.start.listeners and interceptor.classes: It seems that
> both
> > > could be empty?
> > >
> > > JR27. The following configs of type LIST are missing.
> > >
> > > ConsumerConfig
> > > partition.assignment.strategy
> > >
> > > BrokerSecurityConfigs
> > >
> > > QuorumConfig
> > >
> > > LogConfig
> > > leader.replication.throttled.replicas
> > > follower.replication.throttled.replicas
> > >
> > > StreamsConfig
> > > rack.aware.assignment.tags
> > > bootstrap.servers
> > >
> > > MirrorCheckpointConfig
> > > groups
> > > groups.exclude
> > >
> > > MirrorCheckpointConfig
> > > task.assigned.groups
> > >
> > > MirrorSourceConfig
> > > topics
> > > topics.exclude
> > > config.properties.exclude
> > >
> > > ConnectorConfig
> > > transforms
> > > predicates
> > >
> > > SourceConnectorConfig
> > > topic.creation.groups
> > >
> > > WorkerConfig
> > > plugin.path
> > >
> > > DistributedConfig
> > > inter.worker.verification.algorithms
> > >
> > > RestServerConfig
> > > rest.extension.classes
> > > admin.listeners
> > > listeners
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Jul 15, 2025 at 6:22 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >
> > >> Hello Jun Chia,
> > >>
> > >> Thanks for the reply,
> > >>
> > >> JR20: This configuration should remain unchanged. I have updated the
> > KIP.
> > >>
> > >> JR21: For both Consumer and Producer, the default value is an empty
> > list,
> > >> and the
> > >> validator used is NonNullValidator. I think we can change the
> validator
> > to
> > >> ValidList.anyNonDuplicateValues
> > >> with isNullAllowed = false and isEmptyAllowed = false.
> > >>
> > >> JR23: I’ve addressed this change in the KIP. Although the default
> value
> > of
> > >> early.start.listeners remains
> > >> unchanged, the validator now disallows empty lists.
> > >>
> > >> JR24: Since log.dirs does not change its default value, I used a dash
> > (-)
> > >> to indicate that the field remains unchanged.
> > >>
> > >> JR25: I’ve updated the table and aligned similar validators as much as
> > >> possible for consistency.
> > >>
> > >> Best Regards,
> > >> Jiunn-Yang
> > >>
> > >>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月15日 下午2:12 寫道：
> > >>>
> > >>>> JR20. controller.listener.names : It's required on the broker. Is it
> > >>> required on the controller?
> > >>>
> > >>> there is a small "sugar" in the controller, which we will
> > >>> create listener.security.protocol.map based on
> > controller.listener.names
> > >>> if listener.security.protocol.map is not explicitly set [0]
> > >>> Additionally, `controller.listener.names` is also used to filter out
> > the
> > >>> advertised controller listeners [1]
> > >>>
> > >>>
> > >>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be
> set
> > to
> > >>> null, right?
> > >>>
> > >>> I don't think so. There is no null check after calling
> > >>>
> > >>
> >
> `getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
> > >>> which causes an NPE if getList returns null."
> > >>>
> > >>> [0]
> > >>>
> > >>
> >
> https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
> > >>> [1]
> > >>>
> > >>
> >
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
> > >>> [2]
> > >>>
> > >>
> >
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
> > >>>
> > >>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：
> > >>>
> > >>>> Hi, Jiunn-Yang,
> > >>>>
> > >>>> Thanks for the updated KIP.
> > >>>>
> > >>>> JR20. controller.listener.names : It's required on the broker. Is it
> > >>>> required on the controller?
> > >>>>
> > >>>> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer)
> > >> already
> > >>>> require it to be non-null.
> > >>>>
> > >>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be
> set
> > to
> > >>>> null, right?
> > >>>>
> > >>>> JR23. early.start.listeners: It allows default value of null, which
> > >> means
> > >>>> that all listeners included in controller.listener.names will also
> be
> > >> early
> > >>>> start listeners.
> > >>>>
> > >>>> JR24. log.dirs: It currently has a default value of null.
> > >>>>
> > >>>> JR25. It seems there are quite a few more configs of type list. Do
> > they
> > >>>> need to be changed too?
> > >>>>
> > >>>> AdminClientConfig: 4
> > >>>> ConsumerConfig: 5
> > >>>> ProducerConfig: 4
> > >>>> SaslConfigs: 1
> > >>>> SslConfigs: 2
> > >>>> BrokerSecurityConfigs: 5
> > >>>> MirrorClientConfig: 1
> > >>>> org.apache.kafka.connect.mirror : 16
> > >>>> org.apache.kafka.connect.runtime : 21
> > >>>> GroupCoordinatorConfig: 3
> > >>>> QuorumConfig: 2
> > >>>> ServerConfigs: 1
> > >>>> KRaftConfigs: 1
> > >>>> ClientMetricsConfigs: 2
> > >>>> MetricConfigs: 2
> > >>>> LogConfig: 4
> > >>>> StreamsConfig: 4
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>
> > >>>>> Hi Chia, Jun
> > >>>>>
> > >>>>> Thanks for the reply,
> > >>>>> I have updated the table in the KIP.
> > >>>>>
> > >>>>> Best Regards,
> > >>>>> Jiunn-Yang
> > >>>>>
> > >>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
> > >>>>>>
> > >>>>>> Hi, Jun
> > >>>>>>
> > >>>>>> Thanks for the reply.
> > >>>>>>
> > >>>>>> I think the following configs could use null as the default value
> > and
> > >>>>> treat
> > >>>>>> an "empty list" as an illegal value.
> > >>>>>>
> > >>>>>> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
> > >>>>>> `ssl.enabled.protocols`,
> > >>>>>> and `log.dirs`
> > >>>>>>
> > >>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
> > >>>>>>
> > >>>>>>> Hi, Chia-Ping, Jiunn-Yang,
> > >>>>>>>
> > >>>>>>> Thanks for the reply.
> > >>>>>>>
> > >>>>>>> As you said, if ssl.cipher.suites is empty, intuitively this
> means
> > >>>> that
> > >>>>> no
> > >>>>>>> cipher is allowed. So, this should be an illegal config. It just
> > >>>> happens
> > >>>>>>> that our implementation treats it as if the config is not set
> > >> (meaning
> > >>>>> the
> > >>>>>>> value is null). So, it seems that a more intuitive convention is:
> > if
> > >>>> the
> > >>>>>>> config is not set, it defaults to all available cipher suites. If
> > >> it's
> > >>>>>>> explicitly set, it can't be empty.
> > >>>>>>>
> > >>>>>>> Jun
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> Hi Jun, Chia
> > >>>>>>>>
> > >>>>>>>>>> Also, it would be useful to add a compatibility column in the
> > >>>> table.
> > >>>>>>> If
> > >>>>>>>> a
> > >>>>>>>>>> value is formally disallowed, it would be useful to compare
> the
> > >> old
> > >>>>>>> and
> > >>>>>>>> the
> > >>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
> > >>>>>>>> I’ve updated the table to show which exceptions will be thrown
> for
> > >>>> each
> > >>>>>>>> behavior.
> > >>>>>>>>
> > >>>>>>>>>>> I don't understand the point of "5.0". Could you please share
> > >> more
> > >>>>>>>> details?
> > >>>>>>>>>>> It seems to me changing the type or default value should be
> > fine
> > >>>> in
> > >>>>>>>> minor
> > >>>>>>>>>>> release if we don't break the existing property files.
> > >>>>>>>> I don't think this requires any compatibility considerations, so
> > >> I’ve
> > >>>>>>>> removed the Kafka 5.0 changes section.
> > >>>>>>>>
> > >>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type,
> so
> > >>>> using
> > >>>>>>>> `LIST`
> > >>>>>>>>>>> instead of `String` does not seems to make sense.
> > >>>>>>>> I have removed the config from KIP
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Jiunn-Yang
> > >>>>>>>>
> > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> > >>>>>>>>>
> > >>>>>>>>>> For ssl.cipher.suites, changing the default value from null to
> > >>>> empty
> > >>>>>>> is
> > >>>>>>>>>> actually a breaking change. According to the doc, null means
> > that
> > >>>> all
> > >>>>>>>> the
> > >>>>>>>>>> available cipher suites are supported. I am thinking that we
> > >> should
> > >>>>>>>>>> continue to allow null as the default since it could
> represent a
> > >>>>> state
> > >>>>>>>>>> different from empty.
> > >>>>>>>>>
> > >>>>>>>>> "Empty list" seems to be a weird case. It appears to signify
> that
> > >> no
> > >>>>>>>> cipher suites are accepted. However, in the codebase, an empty
> > list
> > >>>> is
> > >>>>>>>> handled as if all available cipher suites are supported [0]. We
> > >>>> should
> > >>>>>>> not
> > >>>>>>>> support the case of "no supported cipher suite," as it doesn't
> > make
> > >>>>>>> sense.
> > >>>>>>>>>
> > >>>>>>>>> In short, changing the default value from null to empty does
> not
> > >>>> break
> > >>>>>>>> the behavior.
> > >>>>>>>>>
> > >>>>>>>>> [0]
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> > >>>>>>>>>
> > >>>>>>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
> > >>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>
> > >>>>>>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be
> empty,
> > we
> > >>>>>>> will
> > >>>>>>>>>> just continue to allow it in all future releases.
> > >>>>>>>>>>
> > >>>>>>>>>> For ssl.cipher.suites, changing the default value from null to
> > >>>> empty
> > >>>>>>> is
> > >>>>>>>>>> actually a breaking change. According to the doc, null means
> > that
> > >>>> all
> > >>>>>>>> the
> > >>>>>>>>>> available cipher suites are supported. I am thinking that we
> > >> should
> > >>>>>>>>>> continue to allow null as the default since it could
> represent a
> > >>>>> state
> > >>>>>>>>>> different from empty.
> > >>>>>>>>>>
> > >>>>>>>>>> Also, it would be useful to add a compatibility column in the
> > >>>> table.
> > >>>>>>> If
> > >>>>>>>> a
> > >>>>>>>>>> value is formally disallowed, it would be useful to compare
> the
> > >> old
> > >>>>>>> and
> > >>>>>>>> the
> > >>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <
> > chia7712@gmail.com
> > >>>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> hi Jiunn
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the explanation. I have updated the KIP
> accordingly:
> > >> in
> > >>>>>>>> Kafka
> > >>>>>>>>>>>> 4.x, cleanup.policy
> > >>>>>>>>>>>> can be set to an empty list with a warning message emitted;
> > >>>>> however,
> > >>>>>>>> in
> > >>>>>>>>>>>> Kafka 5.0, setting an
> > >>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> If we agree to support "empty list", the approach of "none"
> > could
> > >>>> be
> > >>>>>>>>>>> eliminated, right?
> > >>>>>>>>>>>
> > >>>>>>>>>>> I don't understand the point of "5.0". Could you please share
> > >> more
> > >>>>>>>> details?
> > >>>>>>>>>>> It seems to me changing the type or default value should be
> > fine
> > >>>> in
> > >>>>>>>> minor
> > >>>>>>>>>>> release if we don't break the existing property files.
> > >>>>>>>>>>>
> > >>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type,
> so
> > >>>> using
> > >>>>>>>> `LIST`
> > >>>>>>>>>>> instead of `String` does not seems to make sense.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>> Null could make sense as the default value, but it just
> > means
> > >>>>> that
> > >>>>>>>> the
> > >>>>>>>>>>>>>> config key is not set explicitly.
> > >>>>>>>>>>>>>> Given that, it's probably simpler to just allow
> > >>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>>>>>>>>>> Thanks for the explanation. I have updated the KIP
> > accordingly:
> > >>>> in
> > >>>>>>>> Kafka
> > >>>>>>>>>>>> 4.x, cleanup.policy
> > >>>>>>>>>>>> can be set to an empty list with a warning message emitted;
> > >>>>> however,
> > >>>>>>>> in
> > >>>>>>>>>>>> Kafka 5.0, setting an
> > >>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> For another, there is an additional issue which could be
> > >> handled
> > >>>>> by
> > >>>>>>>>>>> this
> > >>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
> > as a
> > >>>>>>> LIST.
> > >>>>>>>>>>> For
> > >>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`.
> Could
> > >> we
> > >>>>>>>> change
> > >>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
> > >> them
> > >>>>>>>>>>>>> non-nullable.
> > >>>>>>>>>>>> I have also updated the KIP to include the tables detailing
> > the
> > >>>>>>>> changes
> > >>>>>>>>>>>> for LIST-type configuration
> > >>>>>>>>>>>> default values and the migration from STRING-type to
> LIST-type
> > >>>>>>>>>>>> configurations.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> hi all,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
> > >>>> allows
> > >>>>>>> an
> > >>>>>>>>>>>> empty
> > >>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's
> ok
> > >> to
> > >>>>>>>>>>> support
> > >>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler
> to
> > >>>> just
> > >>>>>>>>>>> allow
> > >>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> yes, that is what I meant before. Additionally, the `LIST`
> > type
> > >>>>>>>> should
> > >>>>>>>>>>>> not
> > >>>>>>>>>>>>> accept "null". Instead, we should use an empty list as the
> > >>>> default
> > >>>>>>>>>>> value.
> > >>>>>>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should
> > never
> > >>>> has
> > >>>>>>>>>>> "null"
> > >>>>>>>>>>>>> value. That could eliminate the null check.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> For another, there is an additional issue which could be
> > >> handled
> > >>>>> by
> > >>>>>>>>>>> this
> > >>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
> > as a
> > >>>>>>> LIST.
> > >>>>>>>>>>> For
> > >>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`.
> Could
> > >> we
> > >>>>>>>> change
> > >>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
> > >> them
> > >>>>>>>>>>>>> non-nullable.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二
> 上午5:56寫道：
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> JR10. Regarding supporting the null value, I am not sure
> how
> > >>>> one
> > >>>>>>>> could
> > >>>>>>>>>>>> set
> > >>>>>>>>>>>>>> a null value in a config file. For example, if you set the
> > >>>>>>> following
> > >>>>>>>>>>> in
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>> config file.
> > >>>>>>>>>>>>>> configkey=
> > >>>>>>>>>>>>>> The value of configkey will be an empty list, instead of
> > null,
> > >>>>>>>> right?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Null could make sense as the default value, but it just
> > means
> > >>>>> that
> > >>>>>>>> the
> > >>>>>>>>>>>>>> config key is not set explicitly.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
> > >>>> allows
> > >>>>>>> an
> > >>>>>>>>>>>> empty
> > >>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's
> ok
> > >> to
> > >>>>>>>>>>> support
> > >>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler
> to
> > >>>> just
> > >>>>>>>>>>> allow
> > >>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com>
> > >> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional,
> > >> that
> > >>>>> is
> > >>>>>>>>>>>> right,
> > >>>>>>>>>>>>>>> so I think
> > >>>>>>>>>>>>>>> that we can allow null value, but not allow the empty
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> JR11: In this KIP, the goal is to address configurations
> > that
> > >>>>>>>> should
> > >>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> accept empty arrays.
> > >>>>>>>>>>>>>>> We want to prevent users from passing in an empty list
> when
> > >>>> it's
> > >>>>>>>> not
> > >>>>>>>>>>>>>>> valid. I believe there are two
> > >>>>>>>>>>>>>>> main scenarios to consider:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. Nullable but not empty:
> > >>>>>>>>>>>>>>> The parameter allows either a null value or a non-empty
> > list.
> > >>>> In
> > >>>>>>>> this
> > >>>>>>>>>>>>>>> case,
> > >>>>>>>>>>>>>>> if the user provides an empty list, we should throw a
> > >>>>>>>>>>> ConfigException.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
> > >>>>>>> contain a
> > >>>>>>>>>>>>>>> non-empty list — null is also
> > >>>>>>>>>>>>>>> invalid. In this case, if the user provides either a null
> > or
> > >>>> an
> > >>>>>>>> empty
> > >>>>>>>>>>>>>>> list, we should throw a ConfigException.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Therefore, I want to prevent scenarios where users can
> pass
> > >> in
> > >>>>> an
> > >>>>>>>>>>> empty
> > >>>>>>>>>>>>>>> list.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
> > >>>> allowNull
> > >>>>>>>>>>>>>> property.
> > >>>>>>>>>>>>>>> This allows us to specify
> > >>>>>>>>>>>>>>> whether a configuration permits null values or not.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> JR13: I have updated the table
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30
> 寫道：
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Sorry for the late reply. A few more comments.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience
> says
> > >>>> that
> > >>>>>>>> it's
> > >>>>>>>>>>>>>>>> optional. So an empty list seems to be valid.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support
> > empty
> > >>>>>>> lists
> > >>>>>>>> in
> > >>>>>>>>>>>>>>>> general, it's probably more consistent to just allow
> > >>>>>>>>>>>> log.cleanup.policy
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> be empty as Luke suggested, instead of introducing a new
> > >>>> option
> > >>>>>>>>>>> None.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null.
> > It
> > >>>>>>> seems
> > >>>>>>>>>>> that
> > >>>>>>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
> > >>>>> shouldn't
> > >>>>>>>> be
> > >>>>>>>>>>>>>> null.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> JR13. The table is very helpful. Could you also describe
> > the
> > >>>>>>>> old/new
> > >>>>>>>>>>>>>>>> behavior for each config with an empty list or duplicate
> > >>>> entry?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7133700@gmail.com
> >
> > >>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I am manually bumping this thread. Any feedback would
> be
> > >>>>>>>>>>> appreciated.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
> > >>>>>>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
> > >>>>>>>>>>>>>>>>>> provided list is either null or a non-empty list
> without
> > >>>>>>>> duplicate
> > >>>>>>>>>>>>>>>>> entries. I’ve also listed the configurations that will
> > >>>>>>>>>>>>>>>>>> be validated using this logic. Please feel free to
> share
> > >>>> any
> > >>>>>>>>>>>> feedback
> > >>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>> suggestions.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日
> 凌晨2:29
> > >>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> It seems there are quite a few other configs of type
> > list
> > >>>>>>> (e.g.
> > >>>>>>>>>>>>>>> several
> > >>>>>>>>>>>>>>>>> SSL
> > >>>>>>>>>>>>>>>>>>> related ones). It would be useful to understand if
> > empty
> > >>>>>>> lists
> > >>>>>>>>>>> are
> > >>>>>>>>>>>>>>> valid
> > >>>>>>>>>>>>>>>>>>> for them or not.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
> > >>>> jun@confluent.io>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi, Luke,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
> > >>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> > >> introduced?),
> > >>>>>>>> there
> > >>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
> > >>>> time.
> > >>>>> "
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> If you do a google search on "kafka infinite
> > retention",
> > >>>>> you
> > >>>>>>>>>>> will
> > >>>>>>>>>>>>>> get
> > >>>>>>>>>>>>>>>>>>>> links like
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> > >>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> > >>>>>>>>>>>>>>>>> ,
> > >>>>>>>>>>>>>>>>>>>> both pointing to setting the retention time and
> > >> retention
> > >>>>>>> size
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>> -1
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
> > >>>>> intentionally
> > >>>>>>>>>>> wants
> > >>>>>>>>>>>>>>>>> infinite
> > >>>>>>>>>>>>>>>>>>>> retention, it's more likely they will use that
> > approach
> > >>>>>>>> instead
> > >>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>> setting
> > >>>>>>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our
> > >> API/tools
> > >>>>>>> make
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>> possible
> > >>>>>>>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving
> > the
> > >>>>>>>> config
> > >>>>>>>>>>>>>> value
> > >>>>>>>>>>>>>>>>> empty.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
> > >>>>>>> showuon@gmail.com
> > >>>>>>>>>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> > >>>> documented
> > >>>>>>>> that
> > >>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact,
> > if
> > >>>> one
> > >>>>>>>>>>> leaves
> > >>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
> > storage
> > >>>>>>> since
> > >>>>>>>>>>> it
> > >>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>> stop
> > >>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
> > >> too.
> > >>>>>>> So,
> > >>>>>>>>>>>>>> leaving
> > >>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> > >>>> mistake
> > >>>>>>>> than
> > >>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
> > idea
> > >>>>>>> behind
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
> > aware
> > >>>> of
> > >>>>>>>> this
> > >>>>>>>>>>>>>>> likely
> > >>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
> > wanting
> > >>>>>>>>>>> infinite
> > >>>>>>>>>>>>>>>>> retention
> > >>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
> > >>>> implies
> > >>>>>>>>>>>>>> infinite
> > >>>>>>>>>>>>>>>>>>>>> retention, we also never documented (or failed) the
> > >>>> empty
> > >>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
> > >>>> behavior
> > >>>>>>>> has
> > >>>>>>>>>>>>>> been
> > >>>>>>>>>>>>>>>>>>>>> existed
> > >>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> > >>>> introduced?),
> > >>>>>>>> there
> > >>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a
> long
> > >>>> time.
> > >>>>>>> It
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>> good
> > >>>>>>>>>>>>>>>>>>>>> to break the backward compatibility in a minor
> > release
> > >>>>>>>> version
> > >>>>>>>>>>>>>> (ex:
> > >>>>>>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
> > >>>>> existing
> > >>>>>>>>>>> bug,
> > >>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>>> be a surprise to users.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> > >>>> cleanup.policy
> > >>>>>>> by
> > >>>>>>>>>>>>>> fixing
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
> never
> > >>>>>>> realize
> > >>>>>>>>>>> this
> > >>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>> it's
> > >>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Chia-Ping,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in
> > 4.x
> > >>>> and
> > >>>>>>> in
> > >>>>>>>>>>> 5.0
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>> throw
> > >>>>>>>>>>>>>>>>>>>>> exception directly to make it fail fast
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> This suggestion looks good to me!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thank you.
> > >>>>>>>>>>>>>>>>>>>>> Luke
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> > >>>>>>>>>>>>>> chia7712@apache.org
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> hi all,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases,
> it
> > is
> > >>>>>>> worth
> > >>>>>>>>>>>>>>>>> considering
> > >>>>>>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
> > >>>>>>>> encountered
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> use
> > >>>>>>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
> > >>>>>>>> configurations
> > >>>>>>>>>>>>>>>>> instead.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> > >>>>> cleanup.policy
> > >>>>>>>> by
> > >>>>>>>>>>>>>>> fixing
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
> > never
> > >>>>>>>> realize
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>> it's a
> > >>>>>>>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make
> > sense
> > >>>> for
> > >>>>>>>>>>> remote
> > >>>>>>>>>>>>>>>>>>>>> storage.
> > >>>>>>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with
> tiered
> > >>>>>>> storage
> > >>>>>>>>>>> use
> > >>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>> valid
> > >>>>>>>>>>>>>>>>>>>>>> delete policy.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> > >>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> > >>>> documented
> > >>>>>>>> that
> > >>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In
> fact,
> > >> if
> > >>>>>>> one
> > >>>>>>>>>>>>>> leaves
> > >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
> > >>>> storage
> > >>>>>>>> since
> > >>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>> stop
> > >>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and
> time
> > >>>> too.
> > >>>>>>>> So,
> > >>>>>>>>>>>>>>> leaving
> > >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> > >>>>> mistake
> > >>>>>>>>>>> than
> > >>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
> > >> idea
> > >>>>>>>> behind
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
> > >> aware
> > >>>>> of
> > >>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> likely
> > >>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
> > >>>> wanting
> > >>>>>>>>>>>> infinite
> > >>>>>>>>>>>>>>>>>>>>>> retention
> > >>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> > >>>>> cleanup.policy
> > >>>>>>>> by
> > >>>>>>>>>>>>>>> fixing
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
> > never
> > >>>>>>>> realize
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>> it's a
> > >>>>>>>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
> > >>>>>>>> showuon@gmail.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Comments:
> > >>>>>>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty
> > cleanup.policy
> > >>>> is
> > >>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is
> justified."
> > >>>>>>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
> > >>>>> trade-off
> > >>>>>>>>>>>>>>> justified?
> > >>>>>>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid
> in
> > >>>>>>> kafka,
> > >>>>>>>>>>> why
> > >>>>>>>>>>>>>> do
> > >>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>> provide
> > >>>>>>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this
> KIP?
> > >>>>>>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to
> keep
> > >> all
> > >>>>>>>>>>> history
> > >>>>>>>>>>>>>>> data
> > >>>>>>>>>>>>>>>>>>>>>> without
> > >>>>>>>>>>>>>>>>>>>>>>>> a cleanup policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast
> for
> > >>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
> > >>>>>>>> "process.roles"
> > >>>>>>>>>>>>>>> configs
> > >>>>>>>>>>>>>>>>>>>>>> since
> > >>>>>>>>>>>>>>>>>>>>>>>> they are invalid.
> > >>>>>>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a
> > persuasive
> > >>>>>>>> reason
> > >>>>>>>>>>>> why
> > >>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
> > >>>>>>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to
> configure
> > >>>>> Kafka
> > >>>>>>>> to
> > >>>>>>>>>>>>>>> retain
> > >>>>>>>>>>>>>>>>>>>>> all
> > >>>>>>>>>>>>>>>>>>>>>> log
> > >>>>>>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
> > >>>>>>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP,
> > but
> > >>>> to
> > >>>>>>> me,
> > >>>>>>>>>>>>>>> backward
> > >>>>>>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a
> clear
> > >>>> and
> > >>>>>>>>>>> direct
> > >>>>>>>>>>>>>> way
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> config
> > >>>>>>>>>>>>>>>>>>>>>>>> kafka".
> > >>>>>>>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy"
> > >>>>> config?
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks.
> > >>>>>>>>>>>>>>>>>>>>>>>> Luke
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> > >>>>>>>>>>>>>> <jun@confluent.io.invalid
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to
> me.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
> > >>>>> s7133700@gmail.com
> > >>>>>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
> been
> > >>>>>>>> addressed
> > >>>>>>>>>>> in
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
> > >>>>>>>>>>> Additionally,
> > >>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to
> > the
> > >>>>>>>>>>>> inNonEmpty
> > >>>>>>>>>>>>>>>>>>>>>> method.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > 2025年5月6日
> > >>>>>>>> 凌晨12:46
> > >>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users
> pass
> > >>>> empty
> > >>>>>>>>>>>> config,
> > >>>>>>>>>>>>>>>>>>>>> so
> > >>>>>>>>>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw
> > >> exception
> > >>>>>>>>>>>> directly.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> 2025年5月6日
> > >> 週二
> > >>>>>>>>>>>>>> 上午12:28寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still
> > references
> > >>>> to
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> deprecated
> > >>>>>>>>>>>>>>>>>>>>>>>>>> method.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be
> > removed
> > >>>> in
> > >>>>>>>> Kafka
> > >>>>>>>>>>>>>> 5.0"
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
> > >>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
> > been
> > >>>>>>>>>>> addressed
> > >>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > >>>> 2025年5月3日
> > >>>>>>>>>>> 凌晨2:03
> > >>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
> > >>>> Migration
> > >>>>>>>>>>> Plan",
> > >>>>>>>>>>>>>>>>>>>>>> could you
> > >>>>>>>>>>>>>>>>>>>>>>>>> add
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
> > >>>>> "clean.policy="
> > >>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>> replaced
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to
> > disable
> > >>>>>>>> delete
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> compact.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> > >> 2025年5月3日
> > >>>>> 週六
> > >>>>>>>>>>>>>>>>>>>>> 上午1:55寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in
> for
> > >>>> now.
> > >>>>>>>>>>> Could
> > >>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>>>>>>>>> update
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
> > >>>>>>>>>>> deprecation.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> > >>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
> > >>>>>>> ValidList#in
> > >>>>>>>>>>>>>>>>>>>>> method,
> > >>>>>>>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
> > >>>> useful
> > >>>>>>> to
> > >>>>>>>>>>> have
> > >>>>>>>>>>>>>>>>>>>>> both
> > >>>>>>>>>>>>>>>>>>>>>>>>> methods:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty
> > >>>>> lists).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be
> useful
> > to
> > >>>>>>>>>>> document
> > >>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
> > will
> > >>>>>>> fail
> > >>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> start.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >>>>> 2025年5月2日
> > >>>>>>>>>>>>>> 凌晨12:50
> > >>>>>>>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be
> useful
> > to
> > >>>>>>>>>>> document
> > >>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
> > will
> > >>>>>>> fail
> > >>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> start.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> > >>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we
> need
> > >> to
> > >>>>>>>>>>> maintain
> > >>>>>>>>>>>>>>>>>>>>>> backward
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
> > >>>> isEmptyAllowed
> > >>>>>>>> flag
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>> necessary.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of
> > >> isNonEmpty()
> > >>>>> to
> > >>>>>>>>>>>> remove
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> boolean
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >>>>>>> 2025年5月1日
> > >>>>>>>>>>>>>> 凌晨4:17
> > >>>>>>>>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed?
> It's
> > >>>>>>> awkward
> > >>>>>>>>>>>> since
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when
> > >> it's
> > >>>>>>> set
> > >>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> true.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> > >>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
> > >>>>> introduce
> > >>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is
> that
> > >> we
> > >>>>>>> now
> > >>>>>>>>>>>>>>>>>>>>> explicitly
> > >>>>>>>>>>>>>>>>>>>>>>>>> throw a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
> > >>>> format
> > >>>>>>>>>>>>>> validation
> > >>>>>>>>>>>>>>>>>>>>>> phase
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
> > >>>>> behavior.”
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID>
> 於
> > >>>>>>>>>>> 2025年4月25日
> > >>>>>>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
> > >>>> introduced
> > >>>>>>> by
> > >>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>> change
> > >>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown
> > >>>> during
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> storage
> > >>>>>>>>>>>>>>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
> > >>>> startup."
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating
> the
> > >>>>>>> storage
> > >>>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>> fails
> > >>>>>>>>>>>>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> group.coordinator.rebalance.protocols
> > >>>> and
> > >>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM
> 黃竣陽 <
> > >>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this
> change:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> > >>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup
> phase
> > to
> > >>>>> the
> > >>>>>>>>>>>>>> storage
> > >>>>>>>>>>>>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an
> > empty
> > >>>>>>> list
> > >>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during
> > >>>>> storage
> > >>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <jun@confluent.io.INVALID
> >
> > 於
> > >>>>>>>>>>>> 2025年4月24日
> > >>>>>>>>>>>>>>>>>>>>>> 凌晨4:44
> > >>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> > >>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> > >>>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty
> right
> > >>>> now.
> > >>>>>>>> With
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>> KIP,
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and
> more
> > >>>>>>>> accurate)
> > >>>>>>>>>>>>>>>>>>>>>> exception.
> > >>>>>>>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind
> of
> > >>>>> agree
> > >>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping.
> > >>>>>>>>>>>>>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more
> > >> likely
> > >>>>> to
> > >>>>>>>> be
> > >>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>> mistake
> > >>>>>>>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as
> > None,
> > >>>>> the
> > >>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>> never
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> realize
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception
> will
> > >>>> let
> > >>>>>>> the
> > >>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>> realize
> > >>>>>>>>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
> > >>>>> Chia-Ping
> > >>>>>>>>>>> Tsai
> > >>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I
> > understand
> > >>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>> maintaining
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
> > >>>>> documentation,
> > >>>>>>>> we
> > >>>>>>>>>>>>>> don't
> > >>>>>>>>>>>>>>>>>>>>>> allow
> > >>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence,
> should
> > we
> > >>>>>>>>>>> consider
> > >>>>>>>>>>>>>>>>>>>>>> throwing
> > >>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This
> > >>>> could
> > >>>>>>>>>>>>>> streamline
> > >>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> code
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
> <jun@confluent.io.invalid
> > >
> > >> 於
> > >>>>>>>>>>>>>> 2025年4月23日
> > >>>>>>>>>>>>>>>>>>>>> 週三
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will
> > not
> > >>>>>>>> delete
> > >>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>> compact
> > >>>>>>>>>>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We
> won't
> > >>>>>>> delete
> > >>>>>>>>>>>>>>>>>>>>> segments
> > >>>>>>>>>>>>>>>>>>>>>> based
> > >>>>>>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
> > >>>> log.retention.ms,
> > >>>>>>>> but
> > >>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> continue
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> > >>>>>>>>>>> log.local.retention.bytes/
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk
> > >> space
> > >>>>>>> when
> > >>>>>>>>>>>>>> remote
> > >>>>>>>>>>>>>>>>>>>>>>>> storage
> > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM
> > Jun
> > >>>>>>> Rao <
> > >>>>>>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
> > >>>> values
> > >>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>> group.coordinator.rebalance.protocols
> > >>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at
> 7:29 AM
> > >>>> 黃竣陽 <
> > >>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
> > >>>>>>>> accordingly
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>> revised
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none
> policy
> > >>>>>>> cannot
> > >>>>>>>> be
> > >>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an
> > >> empty
> > >>>>>>>>>>>>>>>>>>>>>> cleanup.policy,
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the
> > preProcessParsedConfig
> > >>>>>>> step.
> > >>>>>>>>>>>>>>>>>>>>>>>> Additionally, a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform
> users
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
> > >> <jun@confluent.io.INVALID
> > >>>>>
> > >>>>>>> 於
> > >>>>>>>>>>>>>>>>>>>>> 2025年4月22日
> > >>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:52
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
> > >>>> comments.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a
> > new
> > >>>>>>> value
> > >>>>>>>>>>> None
> > >>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are
> > >> valid.
> > >>>>>>> For
> > >>>>>>>>>>>>>>>>>>>>> example,
> > >>>>>>>>>>>>>>>>>>>>>> None
> > >>>>>>>>>>>>>>>>>>>>>>>>>> can't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would
> be
> > >>>>>>> useful
> > >>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> document
> > >>>>>>>>>>>>>>>>>>>>>>>>> that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior
> during
> > >>>>>>> upgrade
> > >>>>>>>>>>> when
> > >>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>> existing
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's
> not
> > >>>> clear
> > >>>>>>>> what
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>> check
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at
> > 8:25 AM
> > >>>> 黃竣陽
> > >>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> > >>>> discussion
> > >>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>> KIP-1161:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
> improve
> > >>>> the
> > >>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be
> > left
> > >>>>>>>> empty.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
> > >>>> proposed
> > >>>>>>>>>>>>>>>>>>>>> improvements:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to
> > validate
> > >>>>>>>> whether
> > >>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>> list
> > >>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none'
> > value
> > >>>>> for
> > >>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply. A few more comments.

JR28. AdminClientConfig: Currently, bootstrap.servers doesn't throw
NullPointerException on null value. It's converted to an empty list.

JR29. The "Before exception" column: It would be useful to add which value
causes an exception. For example, duplicated values currently don't
typically cause exceptions. Also, for early.start.listeners, which value
causes ConfigException currently?

JR30. Why do we allow the following to be null? In general, null only makes
sense if it's the default value.
MirrorSourceConfig: topics, topics.exclude, config.properties.exclude

JR31. WorkerConfig: plugin.path It seems that we should allow it to be
empty, but not null?

JR32. partition.assignment.strategy shouldn't be empty, right?

Jun

On Wed, Jul 16, 2025 at 4:32 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hi Jun,
>
> Thanks for the reply.
>
> JR26: I’ve addressed this change in the KIP.
>
> JR27:
> I’ve documented the following configurations in the KIP:
>
> ConsumerConfig
> partition.assignment.strategy
>
> StreamsConfig
> bootstrap.servers
>
> MirrorCheckpointConfig
> groups
> groups.exclude
> task.assigned.groups
>
> MirrorSourceConfig
> topics
> topics.exclude
> config.properties.exclude
>
> WorkerConfig
> plugin.path
>
> RestServerConfig
> rest.extension.classes
>
> Other configurations already have default values and corresponding
> validators:
>
> LogConfig
> leader.replication.throttled.replicas: List.of() →
> ThrottledReplicaListValidator
> follower.replication.throttled.replicas: List.of() →
> ThrottledReplicaListValidator
>
> StreamsConfig
> rack.aware.assignment.tags: List.of() → atMostOfSize
>
> ConnectorConfig
> transforms: List.of() → aliasValidator
> predicates: List.of() → aliasValidator
>
> SourceConnectorConfig
> topic.creation.groups: List.of() → ConfigDef.CompositeValidator
>
> DistributedConfig
> inter.worker.verification.algorithms:
> defaultVerificationAlgorithms(crypto) → ConfigDef.LambdaValidator
>
> RestServerConfig
> admin.listeners: null → AdminListenersValidator
> listeners: List.of("http://:8083") → ListenersValidator
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月16日 清晨7:14 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the reply.
> >
> > JR 26. early.start.listeners and interceptor.classes: It seems that both
> > could be empty?
> >
> > JR27. The following configs of type LIST are missing.
> >
> > ConsumerConfig
> > partition.assignment.strategy
> >
> > BrokerSecurityConfigs
> >
> > QuorumConfig
> >
> > LogConfig
> > leader.replication.throttled.replicas
> > follower.replication.throttled.replicas
> >
> > StreamsConfig
> > rack.aware.assignment.tags
> > bootstrap.servers
> >
> > MirrorCheckpointConfig
> > groups
> > groups.exclude
> >
> > MirrorCheckpointConfig
> > task.assigned.groups
> >
> > MirrorSourceConfig
> > topics
> > topics.exclude
> > config.properties.exclude
> >
> > ConnectorConfig
> > transforms
> > predicates
> >
> > SourceConnectorConfig
> > topic.creation.groups
> >
> > WorkerConfig
> > plugin.path
> >
> > DistributedConfig
> > inter.worker.verification.algorithms
> >
> > RestServerConfig
> > rest.extension.classes
> > admin.listeners
> > listeners
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Jul 15, 2025 at 6:22 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello Jun Chia,
> >>
> >> Thanks for the reply,
> >>
> >> JR20: This configuration should remain unchanged. I have updated the
> KIP.
> >>
> >> JR21: For both Consumer and Producer, the default value is an empty
> list,
> >> and the
> >> validator used is NonNullValidator. I think we can change the validator
> to
> >> ValidList.anyNonDuplicateValues
> >> with isNullAllowed = false and isEmptyAllowed = false.
> >>
> >> JR23: I’ve addressed this change in the KIP. Although the default value
> of
> >> early.start.listeners remains
> >> unchanged, the validator now disallows empty lists.
> >>
> >> JR24: Since log.dirs does not change its default value, I used a dash
> (-)
> >> to indicate that the field remains unchanged.
> >>
> >> JR25: I’ve updated the table and aligned similar validators as much as
> >> possible for consistency.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月15日 下午2:12 寫道：
> >>>
> >>>> JR20. controller.listener.names : It's required on the broker. Is it
> >>> required on the controller?
> >>>
> >>> there is a small "sugar" in the controller, which we will
> >>> create listener.security.protocol.map based on
> controller.listener.names
> >>> if listener.security.protocol.map is not explicitly set [0]
> >>> Additionally, `controller.listener.names` is also used to filter out
> the
> >>> advertised controller listeners [1]
> >>>
> >>>
> >>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be set
> to
> >>> null, right?
> >>>
> >>> I don't think so. There is no null check after calling
> >>>
> >>
> `getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
> >>> which causes an NPE if getList returns null."
> >>>
> >>> [0]
> >>>
> >>
> https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
> >>> [1]
> >>>
> >>
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
> >>> [2]
> >>>
> >>
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
> >>>
> >>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：
> >>>
> >>>> Hi, Jiunn-Yang,
> >>>>
> >>>> Thanks for the updated KIP.
> >>>>
> >>>> JR20. controller.listener.names : It's required on the broker. Is it
> >>>> required on the controller?
> >>>>
> >>>> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer)
> >> already
> >>>> require it to be non-null.
> >>>>
> >>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be set
> to
> >>>> null, right?
> >>>>
> >>>> JR23. early.start.listeners: It allows default value of null, which
> >> means
> >>>> that all listeners included in controller.listener.names will also be
> >> early
> >>>> start listeners.
> >>>>
> >>>> JR24. log.dirs: It currently has a default value of null.
> >>>>
> >>>> JR25. It seems there are quite a few more configs of type list. Do
> they
> >>>> need to be changed too?
> >>>>
> >>>> AdminClientConfig: 4
> >>>> ConsumerConfig: 5
> >>>> ProducerConfig: 4
> >>>> SaslConfigs: 1
> >>>> SslConfigs: 2
> >>>> BrokerSecurityConfigs: 5
> >>>> MirrorClientConfig: 1
> >>>> org.apache.kafka.connect.mirror : 16
> >>>> org.apache.kafka.connect.runtime : 21
> >>>> GroupCoordinatorConfig: 3
> >>>> QuorumConfig: 2
> >>>> ServerConfigs: 1
> >>>> KRaftConfigs: 1
> >>>> ClientMetricsConfigs: 2
> >>>> MetricConfigs: 2
> >>>> LogConfig: 4
> >>>> StreamsConfig: 4
> >>>>
> >>>> Jun
> >>>>
> >>>> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>
> >>>>> Hi Chia, Jun
> >>>>>
> >>>>> Thanks for the reply,
> >>>>> I have updated the table in the KIP.
> >>>>>
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
> >>>>>>
> >>>>>> Hi, Jun
> >>>>>>
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> I think the following configs could use null as the default value
> and
> >>>>> treat
> >>>>>> an "empty list" as an illegal value.
> >>>>>>
> >>>>>> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
> >>>>>> `ssl.enabled.protocols`,
> >>>>>> and `log.dirs`
> >>>>>>
> >>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
> >>>>>>
> >>>>>>> Hi, Chia-Ping, Jiunn-Yang,
> >>>>>>>
> >>>>>>> Thanks for the reply.
> >>>>>>>
> >>>>>>> As you said, if ssl.cipher.suites is empty, intuitively this means
> >>>> that
> >>>>> no
> >>>>>>> cipher is allowed. So, this should be an illegal config. It just
> >>>> happens
> >>>>>>> that our implementation treats it as if the config is not set
> >> (meaning
> >>>>> the
> >>>>>>> value is null). So, it seems that a more intuitive convention is:
> if
> >>>> the
> >>>>>>> config is not set, it defaults to all available cipher suites. If
> >> it's
> >>>>>>> explicitly set, it can't be empty.
> >>>>>>>
> >>>>>>> Jun
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hi Jun, Chia
> >>>>>>>>
> >>>>>>>>>> Also, it would be useful to add a compatibility column in the
> >>>> table.
> >>>>>>> If
> >>>>>>>> a
> >>>>>>>>>> value is formally disallowed, it would be useful to compare the
> >> old
> >>>>>>> and
> >>>>>>>> the
> >>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
> >>>>>>>> I’ve updated the table to show which exceptions will be thrown for
> >>>> each
> >>>>>>>> behavior.
> >>>>>>>>
> >>>>>>>>>>> I don't understand the point of "5.0". Could you please share
> >> more
> >>>>>>>> details?
> >>>>>>>>>>> It seems to me changing the type or default value should be
> fine
> >>>> in
> >>>>>>>> minor
> >>>>>>>>>>> release if we don't break the existing property files.
> >>>>>>>> I don't think this requires any compatibility considerations, so
> >> I’ve
> >>>>>>>> removed the Kafka 5.0 changes section.
> >>>>>>>>
> >>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
> >>>> using
> >>>>>>>> `LIST`
> >>>>>>>>>>> instead of `String` does not seems to make sense.
> >>>>>>>> I have removed the config from KIP
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>>
> >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> >>>>>>>>>
> >>>>>>>>>> For ssl.cipher.suites, changing the default value from null to
> >>>> empty
> >>>>>>> is
> >>>>>>>>>> actually a breaking change. According to the doc, null means
> that
> >>>> all
> >>>>>>>> the
> >>>>>>>>>> available cipher suites are supported. I am thinking that we
> >> should
> >>>>>>>>>> continue to allow null as the default since it could represent a
> >>>>> state
> >>>>>>>>>> different from empty.
> >>>>>>>>>
> >>>>>>>>> "Empty list" seems to be a weird case. It appears to signify that
> >> no
> >>>>>>>> cipher suites are accepted. However, in the codebase, an empty
> list
> >>>> is
> >>>>>>>> handled as if all available cipher suites are supported [0]. We
> >>>> should
> >>>>>>> not
> >>>>>>>> support the case of "no supported cipher suite," as it doesn't
> make
> >>>>>>> sense.
> >>>>>>>>>
> >>>>>>>>> In short, changing the default value from null to empty does not
> >>>> break
> >>>>>>>> the behavior.
> >>>>>>>>>
> >>>>>>>>> [0]
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> >>>>>>>>>
> >>>>>>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
> >>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be empty,
> we
> >>>>>>> will
> >>>>>>>>>> just continue to allow it in all future releases.
> >>>>>>>>>>
> >>>>>>>>>> For ssl.cipher.suites, changing the default value from null to
> >>>> empty
> >>>>>>> is
> >>>>>>>>>> actually a breaking change. According to the doc, null means
> that
> >>>> all
> >>>>>>>> the
> >>>>>>>>>> available cipher suites are supported. I am thinking that we
> >> should
> >>>>>>>>>> continue to allow null as the default since it could represent a
> >>>>> state
> >>>>>>>>>> different from empty.
> >>>>>>>>>>
> >>>>>>>>>> Also, it would be useful to add a compatibility column in the
> >>>> table.
> >>>>>>> If
> >>>>>>>> a
> >>>>>>>>>> value is formally disallowed, it would be useful to compare the
> >> old
> >>>>>>> and
> >>>>>>>> the
> >>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <
> chia7712@gmail.com
> >>>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the explanation. I have updated the KIP accordingly:
> >> in
> >>>>>>>> Kafka
> >>>>>>>>>>>> 4.x, cleanup.policy
> >>>>>>>>>>>> can be set to an empty list with a warning message emitted;
> >>>>> however,
> >>>>>>>> in
> >>>>>>>>>>>> Kafka 5.0, setting an
> >>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> If we agree to support "empty list", the approach of "none"
> could
> >>>> be
> >>>>>>>>>>> eliminated, right?
> >>>>>>>>>>>
> >>>>>>>>>>> I don't understand the point of "5.0". Could you please share
> >> more
> >>>>>>>> details?
> >>>>>>>>>>> It seems to me changing the type or default value should be
> fine
> >>>> in
> >>>>>>>> minor
> >>>>>>>>>>> release if we don't break the existing property files.
> >>>>>>>>>>>
> >>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
> >>>> using
> >>>>>>>> `LIST`
> >>>>>>>>>>> instead of `String` does not seems to make sense.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> Null could make sense as the default value, but it just
> means
> >>>>> that
> >>>>>>>> the
> >>>>>>>>>>>>>> config key is not set explicitly.
> >>>>>>>>>>>>>> Given that, it's probably simpler to just allow
> >>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>>>> Thanks for the explanation. I have updated the KIP
> accordingly:
> >>>> in
> >>>>>>>> Kafka
> >>>>>>>>>>>> 4.x, cleanup.policy
> >>>>>>>>>>>> can be set to an empty list with a warning message emitted;
> >>>>> however,
> >>>>>>>> in
> >>>>>>>>>>>> Kafka 5.0, setting an
> >>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> For another, there is an additional issue which could be
> >> handled
> >>>>> by
> >>>>>>>>>>> this
> >>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
> as a
> >>>>>>> LIST.
> >>>>>>>>>>> For
> >>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`. Could
> >> we
> >>>>>>>> change
> >>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
> >> them
> >>>>>>>>>>>>> non-nullable.
> >>>>>>>>>>>> I have also updated the KIP to include the tables detailing
> the
> >>>>>>>> changes
> >>>>>>>>>>>> for LIST-type configuration
> >>>>>>>>>>>> default values and the migration from STRING-type to LIST-type
> >>>>>>>>>>>> configurations.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hi all,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
> >>>> allows
> >>>>>>> an
> >>>>>>>>>>>> empty
> >>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok
> >> to
> >>>>>>>>>>> support
> >>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler to
> >>>> just
> >>>>>>>>>>> allow
> >>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> yes, that is what I meant before. Additionally, the `LIST`
> type
> >>>>>>>> should
> >>>>>>>>>>>> not
> >>>>>>>>>>>>> accept "null". Instead, we should use an empty list as the
> >>>> default
> >>>>>>>>>>> value.
> >>>>>>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should
> never
> >>>> has
> >>>>>>>>>>> "null"
> >>>>>>>>>>>>> value. That could eliminate the null check.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For another, there is an additional issue which could be
> >> handled
> >>>>> by
> >>>>>>>>>>> this
> >>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled
> as a
> >>>>>>> LIST.
> >>>>>>>>>>> For
> >>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`. Could
> >> we
> >>>>>>>> change
> >>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
> >> them
> >>>>>>>>>>>>> non-nullable.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR10. Regarding supporting the null value, I am not sure how
> >>>> one
> >>>>>>>> could
> >>>>>>>>>>>> set
> >>>>>>>>>>>>>> a null value in a config file. For example, if you set the
> >>>>>>> following
> >>>>>>>>>>> in
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>> config file.
> >>>>>>>>>>>>>> configkey=
> >>>>>>>>>>>>>> The value of configkey will be an empty list, instead of
> null,
> >>>>>>>> right?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Null could make sense as the default value, but it just
> means
> >>>>> that
> >>>>>>>> the
> >>>>>>>>>>>>>> config key is not set explicitly.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
> >>>> allows
> >>>>>>> an
> >>>>>>>>>>>> empty
> >>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok
> >> to
> >>>>>>>>>>> support
> >>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler to
> >>>> just
> >>>>>>>>>>> allow
> >>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com>
> >> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional,
> >> that
> >>>>> is
> >>>>>>>>>>>> right,
> >>>>>>>>>>>>>>> so I think
> >>>>>>>>>>>>>>> that we can allow null value, but not allow the empty
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR11: In this KIP, the goal is to address configurations
> that
> >>>>>>>> should
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>> accept empty arrays.
> >>>>>>>>>>>>>>> We want to prevent users from passing in an empty list when
> >>>> it's
> >>>>>>>> not
> >>>>>>>>>>>>>>> valid. I believe there are two
> >>>>>>>>>>>>>>> main scenarios to consider:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. Nullable but not empty:
> >>>>>>>>>>>>>>> The parameter allows either a null value or a non-empty
> list.
> >>>> In
> >>>>>>>> this
> >>>>>>>>>>>>>>> case,
> >>>>>>>>>>>>>>> if the user provides an empty list, we should throw a
> >>>>>>>>>>> ConfigException.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
> >>>>>>> contain a
> >>>>>>>>>>>>>>> non-empty list — null is also
> >>>>>>>>>>>>>>> invalid. In this case, if the user provides either a null
> or
> >>>> an
> >>>>>>>> empty
> >>>>>>>>>>>>>>> list, we should throw a ConfigException.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Therefore, I want to prevent scenarios where users can pass
> >> in
> >>>>> an
> >>>>>>>>>>> empty
> >>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
> >>>> allowNull
> >>>>>>>>>>>>>> property.
> >>>>>>>>>>>>>>> This allows us to specify
> >>>>>>>>>>>>>>> whether a configuration permits null values or not.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> JR13: I have updated the table
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Sorry for the late reply. A few more comments.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says
> >>>> that
> >>>>>>>> it's
> >>>>>>>>>>>>>>>> optional. So an empty list seems to be valid.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support
> empty
> >>>>>>> lists
> >>>>>>>> in
> >>>>>>>>>>>>>>>> general, it's probably more consistent to just allow
> >>>>>>>>>>>> log.cleanup.policy
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> be empty as Luke suggested, instead of introducing a new
> >>>> option
> >>>>>>>>>>> None.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null.
> It
> >>>>>>> seems
> >>>>>>>>>>> that
> >>>>>>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
> >>>>> shouldn't
> >>>>>>>> be
> >>>>>>>>>>>>>> null.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> JR13. The table is very helpful. Could you also describe
> the
> >>>>>>>> old/new
> >>>>>>>>>>>>>>>> behavior for each config with an empty list or duplicate
> >>>> entry?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I am manually bumping this thread. Any feedback would be
> >>>>>>>>>>> appreciated.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
> >>>>>>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
> >>>>>>>>>>>>>>>>>> provided list is either null or a non-empty list without
> >>>>>>>> duplicate
> >>>>>>>>>>>>>>>>> entries. I’ve also listed the configurations that will
> >>>>>>>>>>>>>>>>>> be validated using this logic. Please feel free to share
> >>>> any
> >>>>>>>>>>>> feedback
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> suggestions.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29
> >>>> 寫道：
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> It seems there are quite a few other configs of type
> list
> >>>>>>> (e.g.
> >>>>>>>>>>>>>>> several
> >>>>>>>>>>>>>>>>> SSL
> >>>>>>>>>>>>>>>>>>> related ones). It would be useful to understand if
> empty
> >>>>>>> lists
> >>>>>>>>>>> are
> >>>>>>>>>>>>>>> valid
> >>>>>>>>>>>>>>>>>>> for them or not.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
> >>>> jun@confluent.io>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
> >>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> >> introduced?),
> >>>>>>>> there
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
> >>>> time.
> >>>>> "
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> If you do a google search on "kafka infinite
> retention",
> >>>>> you
> >>>>>>>>>>> will
> >>>>>>>>>>>>>> get
> >>>>>>>>>>>>>>>>>>>> links like
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> >>>>>>>>>>>>>>>>> ,
> >>>>>>>>>>>>>>>>>>>> both pointing to setting the retention time and
> >> retention
> >>>>>>> size
> >>>>>>>>>>> to
> >>>>>>>>>>>>>> -1
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
> >>>>> intentionally
> >>>>>>>>>>> wants
> >>>>>>>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>>> retention, it's more likely they will use that
> approach
> >>>>>>>> instead
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> setting
> >>>>>>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our
> >> API/tools
> >>>>>>> make
> >>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> possible
> >>>>>>>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving
> the
> >>>>>>>> config
> >>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
> >>>>>>> showuon@gmail.com
> >>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> >>>> documented
> >>>>>>>> that
> >>>>>>>>>>> an
> >>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact,
> if
> >>>> one
> >>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
> storage
> >>>>>>> since
> >>>>>>>>>>> it
> >>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> stop
> >>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
> >> too.
> >>>>>>> So,
> >>>>>>>>>>>>>> leaving
> >>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> >>>> mistake
> >>>>>>>> than
> >>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
> idea
> >>>>>>> behind
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
> aware
> >>>> of
> >>>>>>>> this
> >>>>>>>>>>>>>>> likely
> >>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
> wanting
> >>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>> retention
> >>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
> >>>> implies
> >>>>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>>>> retention, we also never documented (or failed) the
> >>>> empty
> >>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
> >>>> behavior
> >>>>>>>> has
> >>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>>>> existed
> >>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> >>>> introduced?),
> >>>>>>>> there
> >>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
> >>>> time.
> >>>>>>> It
> >>>>>>>>>>> is
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>> good
> >>>>>>>>>>>>>>>>>>>>> to break the backward compatibility in a minor
> release
> >>>>>>>> version
> >>>>>>>>>>>>>> (ex:
> >>>>>>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
> >>>>> existing
> >>>>>>>>>>> bug,
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>> be a surprise to users.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >>>> cleanup.policy
> >>>>>>> by
> >>>>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> >>>>>>> realize
> >>>>>>>>>>> this
> >>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in
> 4.x
> >>>> and
> >>>>>>> in
> >>>>>>>>>>> 5.0
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>> exception directly to make it fail fast
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This suggestion looks good to me!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thank you.
> >>>>>>>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> >>>>>>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> hi all,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it
> is
> >>>>>>> worth
> >>>>>>>>>>>>>>>>> considering
> >>>>>>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
> >>>>>>>> encountered
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
> >>>>>>>> configurations
> >>>>>>>>>>>>>>>>> instead.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >>>>> cleanup.policy
> >>>>>>>> by
> >>>>>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
> never
> >>>>>>>> realize
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make
> sense
> >>>> for
> >>>>>>>>>>> remote
> >>>>>>>>>>>>>>>>>>>>> storage.
> >>>>>>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered
> >>>>>>> storage
> >>>>>>>>>>> use
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> valid
> >>>>>>>>>>>>>>>>>>>>>> delete policy.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> >>>> documented
> >>>>>>>> that
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact,
> >> if
> >>>>>>> one
> >>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
> >>>> storage
> >>>>>>>> since
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>> stop
> >>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
> >>>> too.
> >>>>>>>> So,
> >>>>>>>>>>>>>>> leaving
> >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> >>>>> mistake
> >>>>>>>>>>> than
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
> >> idea
> >>>>>>>> behind
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
> >> aware
> >>>>> of
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> likely
> >>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
> >>>> wanting
> >>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>>>>> retention
> >>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >>>>> cleanup.policy
> >>>>>>>> by
> >>>>>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may
> never
> >>>>>>>> realize
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
> >>>>>>>> showuon@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Comments:
> >>>>>>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty
> cleanup.policy
> >>>> is
> >>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> >>>>>>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
> >>>>> trade-off
> >>>>>>>>>>>>>>> justified?
> >>>>>>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in
> >>>>>>> kafka,
> >>>>>>>>>>> why
> >>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>> provide
> >>>>>>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
> >>>>>>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep
> >> all
> >>>>>>>>>>> history
> >>>>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>>>>>>>>>> a cleanup policy.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for
> >>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
> >>>>>>>> "process.roles"
> >>>>>>>>>>>>>>> configs
> >>>>>>>>>>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>>>>>>> they are invalid.
> >>>>>>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a
> persuasive
> >>>>>>>> reason
> >>>>>>>>>>>> why
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
> >>>>>>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to configure
> >>>>> Kafka
> >>>>>>>> to
> >>>>>>>>>>>>>>> retain
> >>>>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
> >>>>>>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP,
> but
> >>>> to
> >>>>>>> me,
> >>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a clear
> >>>> and
> >>>>>>>>>>> direct
> >>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>>>> kafka".
> >>>>>>>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy"
> >>>>> config?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> >>>>>>>>>>>>>> <jun@confluent.io.invalid
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
> >>>>> s7133700@gmail.com
> >>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> >>>>>>>> addressed
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
> >>>>>>>>>>> Additionally,
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
> >>>>>>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to
> the
> >>>>>>>>>>>> inNonEmpty
> >>>>>>>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> 2025年5月6日
> >>>>>>>> 凌晨12:46
> >>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass
> >>>> empty
> >>>>>>>>>>>> config,
> >>>>>>>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw
> >> exception
> >>>>>>>>>>>> directly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日
> >> 週二
> >>>>>>>>>>>>>> 上午12:28寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still
> references
> >>>> to
> >>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> deprecated
> >>>>>>>>>>>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be
> removed
> >>>> in
> >>>>>>>> Kafka
> >>>>>>>>>>>>>> 5.0"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
> >>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all
> been
> >>>>>>>>>>> addressed
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> >>>> 2025年5月3日
> >>>>>>>>>>> 凌晨2:03
> >>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
> >>>> Migration
> >>>>>>>>>>> Plan",
> >>>>>>>>>>>>>>>>>>>>>> could you
> >>>>>>>>>>>>>>>>>>>>>>>>> add
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
> >>>>> "clean.policy="
> >>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> replaced
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to
> disable
> >>>>>>>> delete
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> compact.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> >> 2025年5月3日
> >>>>> 週六
> >>>>>>>>>>>>>>>>>>>>> 上午1:55寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for
> >>>> now.
> >>>>>>>>>>> Could
> >>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>>>> update
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
> >>>>>>>>>>> deprecation.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> >>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
> >>>>>>> ValidList#in
> >>>>>>>>>>>>>>>>>>>>> method,
> >>>>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
> >>>> useful
> >>>>>>> to
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>>>> methods:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty
> >>>>> lists).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful
> to
> >>>>>>>>>>> document
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
> will
> >>>>>>> fail
> >>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>> 2025年5月2日
> >>>>>>>>>>>>>> 凌晨12:50
> >>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful
> to
> >>>>>>>>>>> document
> >>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker
> will
> >>>>>>> fail
> >>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> >>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need
> >> to
> >>>>>>>>>>> maintain
> >>>>>>>>>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
> >>>> isEmptyAllowed
> >>>>>>>> flag
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of
> >> isNonEmpty()
> >>>>> to
> >>>>>>>>>>>> remove
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> boolean
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>>> 2025年5月1日
> >>>>>>>>>>>>>> 凌晨4:17
> >>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's
> >>>>>>> awkward
> >>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when
> >> it's
> >>>>>>> set
> >>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> true.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> >>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
> >>>>> introduce
> >>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that
> >> we
> >>>>>>> now
> >>>>>>>>>>>>>>>>>>>>> explicitly
> >>>>>>>>>>>>>>>>>>>>>>>>> throw a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
> >>>> format
> >>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>> phase
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
> >>>>> behavior.”
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>>>>>>> 2025年4月25日
> >>>>>>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
> >>>> introduced
> >>>>>>> by
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown
> >>>> during
> >>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
> >>>> startup."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the
> >>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>> fails
> >>>>>>>>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> >>>> and
> >>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> >>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> >>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase
> to
> >>>>> the
> >>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an
> empty
> >>>>>>> list
> >>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during
> >>>>> storage
> >>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID>
> 於
> >>>>>>>>>>>> 2025年4月24日
> >>>>>>>>>>>>>>>>>>>>>> 凌晨4:44
> >>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> >>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> >>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right
> >>>> now.
> >>>>>>>> With
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
> >>>>>>>> accurate)
> >>>>>>>>>>>>>>>>>>>>>> exception.
> >>>>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of
> >>>>> agree
> >>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping.
> >>>>>>>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more
> >> likely
> >>>>> to
> >>>>>>>> be
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>> mistake
> >>>>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as
> None,
> >>>>> the
> >>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>> never
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will
> >>>> let
> >>>>>>> the
> >>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
> >>>>> Chia-Ping
> >>>>>>>>>>> Tsai
> >>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I
> understand
> >>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>> maintaining
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
> >>>>> documentation,
> >>>>>>>> we
> >>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>>> allow
> >>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should
> we
> >>>>>>>>>>> consider
> >>>>>>>>>>>>>>>>>>>>>> throwing
> >>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This
> >>>> could
> >>>>>>>>>>>>>> streamline
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <jun@confluent.io.invalid
> >
> >> 於
> >>>>>>>>>>>>>> 2025年4月23日
> >>>>>>>>>>>>>>>>>>>>> 週三
> >>>>>>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will
> not
> >>>>>>>> delete
> >>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>> compact
> >>>>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't
> >>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>> segments
> >>>>>>>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
> >>>> log.retention.ms,
> >>>>>>>> but
> >>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> continue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> >>>>>>>>>>> log.local.retention.bytes/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk
> >> space
> >>>>>>> when
> >>>>>>>>>>>>>> remote
> >>>>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM
> Jun
> >>>>>>> Rao <
> >>>>>>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
> >>>> values
> >>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>> group.coordinator.rebalance.protocols
> >>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM
> >>>> 黃竣陽 <
> >>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
> >>>>>>>> accordingly
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> revised
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy
> >>>>>>> cannot
> >>>>>>>> be
> >>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an
> >> empty
> >>>>>>>>>>>>>>>>>>>>>> cleanup.policy,
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the
> preProcessParsedConfig
> >>>>>>> step.
> >>>>>>>>>>>>>>>>>>>>>>>> Additionally, a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
> >> <jun@confluent.io.INVALID
> >>>>>
> >>>>>>> 於
> >>>>>>>>>>>>>>>>>>>>> 2025年4月22日
> >>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:52
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
> >>>> comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a
> new
> >>>>>>> value
> >>>>>>>>>>> None
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are
> >> valid.
> >>>>>>> For
> >>>>>>>>>>>>>>>>>>>>> example,
> >>>>>>>>>>>>>>>>>>>>>> None
> >>>>>>>>>>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be
> >>>>>>> useful
> >>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> document
> >>>>>>>>>>>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during
> >>>>>>> upgrade
> >>>>>>>>>>> when
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not
> >>>> clear
> >>>>>>>> what
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>> check
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at
> 8:25 AM
> >>>> 黃竣陽
> >>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> >>>> discussion
> >>>>>>> on
> >>>>>>>>>>>>>>>>>>>>> KIP-1161:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve
> >>>> the
> >>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be
> left
> >>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
> >>>> proposed
> >>>>>>>>>>>>>>>>>>>>> improvements:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to
> validate
> >>>>>>>> whether
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none'
> value
> >>>>> for
> >>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi Jun, 

Thanks for the reply. 

JR26: I’ve addressed this change in the KIP. 

JR27: 
I’ve documented the following configurations in the KIP: 

ConsumerConfig 
partition.assignment.strategy 

StreamsConfig 
bootstrap.servers

MirrorCheckpointConfig 
groups 
groups.exclude
task.assigned.groups 

MirrorSourceConfig 
topics 
topics.exclude 
config.properties.exclude

WorkerConfig 
plugin.path 

RestServerConfig 
rest.extension.classes 

Other configurations already have default values and corresponding validators: 

LogConfig 
leader.replication.throttled.replicas: List.of() → ThrottledReplicaListValidator 
follower.replication.throttled.replicas: List.of() → ThrottledReplicaListValidator 

StreamsConfig 
rack.aware.assignment.tags: List.of() → atMostOfSize 

ConnectorConfig 
transforms: List.of() → aliasValidator 
predicates: List.of() → aliasValidator 

SourceConnectorConfig 
topic.creation.groups: List.of() → ConfigDef.CompositeValidator 

DistributedConfig 
inter.worker.verification.algorithms: defaultVerificationAlgorithms(crypto) → ConfigDef.LambdaValidator 

RestServerConfig 
admin.listeners: null → AdminListenersValidator 
listeners: List.of("http://:8083") → ListenersValidator

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月16日 清晨7:14 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply.
> 
> JR 26. early.start.listeners and interceptor.classes: It seems that both
> could be empty?
> 
> JR27. The following configs of type LIST are missing.
> 
> ConsumerConfig
> partition.assignment.strategy
> 
> BrokerSecurityConfigs
> 
> QuorumConfig
> 
> LogConfig
> leader.replication.throttled.replicas
> follower.replication.throttled.replicas
> 
> StreamsConfig
> rack.aware.assignment.tags
> bootstrap.servers
> 
> MirrorCheckpointConfig
> groups
> groups.exclude
> 
> MirrorCheckpointConfig
> task.assigned.groups
> 
> MirrorSourceConfig
> topics
> topics.exclude
> config.properties.exclude
> 
> ConnectorConfig
> transforms
> predicates
> 
> SourceConnectorConfig
> topic.creation.groups
> 
> WorkerConfig
> plugin.path
> 
> DistributedConfig
> inter.worker.verification.algorithms
> 
> RestServerConfig
> rest.extension.classes
> admin.listeners
> listeners
> 
> Thanks,
> 
> Jun
> 
> On Tue, Jul 15, 2025 at 6:22 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun Chia,
>> 
>> Thanks for the reply,
>> 
>> JR20: This configuration should remain unchanged. I have updated the KIP.
>> 
>> JR21: For both Consumer and Producer, the default value is an empty list,
>> and the
>> validator used is NonNullValidator. I think we can change the validator to
>> ValidList.anyNonDuplicateValues
>> with isNullAllowed = false and isEmptyAllowed = false.
>> 
>> JR23: I’ve addressed this change in the KIP. Although the default value of
>> early.start.listeners remains
>> unchanged, the validator now disallows empty lists.
>> 
>> JR24: Since log.dirs does not change its default value, I used a dash (-)
>> to indicate that the field remains unchanged.
>> 
>> JR25: I’ve updated the table and aligned similar validators as much as
>> possible for consistency.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月15日 下午2:12 寫道：
>>> 
>>>> JR20. controller.listener.names : It's required on the broker. Is it
>>> required on the controller?
>>> 
>>> there is a small "sugar" in the controller, which we will
>>> create listener.security.protocol.map based on controller.listener.names
>>> if listener.security.protocol.map is not explicitly set [0]
>>> Additionally, `controller.listener.names` is also used to filter out the
>>> advertised controller listeners [1]
>>> 
>>> 
>>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
>>> null, right?
>>> 
>>> I don't think so. There is no null check after calling
>>> 
>> `getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
>>> which causes an NPE if getList returns null."
>>> 
>>> [0]
>>> 
>> https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
>>> [1]
>>> 
>> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
>>> [2]
>>> 
>> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
>>> 
>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：
>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Thanks for the updated KIP.
>>>> 
>>>> JR20. controller.listener.names : It's required on the broker. Is it
>>>> required on the controller?
>>>> 
>>>> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer)
>> already
>>>> require it to be non-null.
>>>> 
>>>> JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
>>>> null, right?
>>>> 
>>>> JR23. early.start.listeners: It allows default value of null, which
>> means
>>>> that all listeners included in controller.listener.names will also be
>> early
>>>> start listeners.
>>>> 
>>>> JR24. log.dirs: It currently has a default value of null.
>>>> 
>>>> JR25. It seems there are quite a few more configs of type list. Do they
>>>> need to be changed too?
>>>> 
>>>> AdminClientConfig: 4
>>>> ConsumerConfig: 5
>>>> ProducerConfig: 4
>>>> SaslConfigs: 1
>>>> SslConfigs: 2
>>>> BrokerSecurityConfigs: 5
>>>> MirrorClientConfig: 1
>>>> org.apache.kafka.connect.mirror : 16
>>>> org.apache.kafka.connect.runtime : 21
>>>> GroupCoordinatorConfig: 3
>>>> QuorumConfig: 2
>>>> ServerConfigs: 1
>>>> KRaftConfigs: 1
>>>> ClientMetricsConfigs: 2
>>>> MetricConfigs: 2
>>>> LogConfig: 4
>>>> StreamsConfig: 4
>>>> 
>>>> Jun
>>>> 
>>>> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hi Chia, Jun
>>>>> 
>>>>> Thanks for the reply,
>>>>> I have updated the table in the KIP.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
>>>>>> 
>>>>>> Hi, Jun
>>>>>> 
>>>>>> Thanks for the reply.
>>>>>> 
>>>>>> I think the following configs could use null as the default value and
>>>>> treat
>>>>>> an "empty list" as an illegal value.
>>>>>> 
>>>>>> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
>>>>>> `ssl.enabled.protocols`,
>>>>>> and `log.dirs`
>>>>>> 
>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
>>>>>> 
>>>>>>> Hi, Chia-Ping, Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the reply.
>>>>>>> 
>>>>>>> As you said, if ssl.cipher.suites is empty, intuitively this means
>>>> that
>>>>> no
>>>>>>> cipher is allowed. So, this should be an illegal config. It just
>>>> happens
>>>>>>> that our implementation treats it as if the config is not set
>> (meaning
>>>>> the
>>>>>>> value is null). So, it seems that a more intuitive convention is: if
>>>> the
>>>>>>> config is not set, it defaults to all available cipher suites. If
>> it's
>>>>>>> explicitly set, it can't be empty.
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hi Jun, Chia
>>>>>>>> 
>>>>>>>>>> Also, it would be useful to add a compatibility column in the
>>>> table.
>>>>>>> If
>>>>>>>> a
>>>>>>>>>> value is formally disallowed, it would be useful to compare the
>> old
>>>>>>> and
>>>>>>>> the
>>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>>>>> I’ve updated the table to show which exceptions will be thrown for
>>>> each
>>>>>>>> behavior.
>>>>>>>> 
>>>>>>>>>>> I don't understand the point of "5.0". Could you please share
>> more
>>>>>>>> details?
>>>>>>>>>>> It seems to me changing the type or default value should be fine
>>>> in
>>>>>>>> minor
>>>>>>>>>>> release if we don't break the existing property files.
>>>>>>>> I don't think this requires any compatibility considerations, so
>> I’ve
>>>>>>>> removed the Kafka 5.0 changes section.
>>>>>>>> 
>>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
>>>> using
>>>>>>>> `LIST`
>>>>>>>>>>> instead of `String` does not seems to make sense.
>>>>>>>> I have removed the config from KIP
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
>>>>>>>>> 
>>>>>>>>>> For ssl.cipher.suites, changing the default value from null to
>>>> empty
>>>>>>> is
>>>>>>>>>> actually a breaking change. According to the doc, null means that
>>>> all
>>>>>>>> the
>>>>>>>>>> available cipher suites are supported. I am thinking that we
>> should
>>>>>>>>>> continue to allow null as the default since it could represent a
>>>>> state
>>>>>>>>>> different from empty.
>>>>>>>>> 
>>>>>>>>> "Empty list" seems to be a weird case. It appears to signify that
>> no
>>>>>>>> cipher suites are accepted. However, in the codebase, an empty list
>>>> is
>>>>>>>> handled as if all available cipher suites are supported [0]. We
>>>> should
>>>>>>> not
>>>>>>>> support the case of "no supported cipher suite," as it doesn't make
>>>>>>> sense.
>>>>>>>>> 
>>>>>>>>> In short, changing the default value from null to empty does not
>>>> break
>>>>>>>> the behavior.
>>>>>>>>> 
>>>>>>>>> [0]
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
>>>>>>>>> 
>>>>>>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we
>>>>>>> will
>>>>>>>>>> just continue to allow it in all future releases.
>>>>>>>>>> 
>>>>>>>>>> For ssl.cipher.suites, changing the default value from null to
>>>> empty
>>>>>>> is
>>>>>>>>>> actually a breaking change. According to the doc, null means that
>>>> all
>>>>>>>> the
>>>>>>>>>> available cipher suites are supported. I am thinking that we
>> should
>>>>>>>>>> continue to allow null as the default since it could represent a
>>>>> state
>>>>>>>>>> different from empty.
>>>>>>>>>> 
>>>>>>>>>> Also, it would be useful to add a compatibility column in the
>>>> table.
>>>>>>> If
>>>>>>>> a
>>>>>>>>>> value is formally disallowed, it would be useful to compare the
>> old
>>>>>>> and
>>>>>>>> the
>>>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <chia7712@gmail.com
>>> 
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> hi Jiunn
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the explanation. I have updated the KIP accordingly:
>> in
>>>>>>>> Kafka
>>>>>>>>>>>> 4.x, cleanup.policy
>>>>>>>>>>>> can be set to an empty list with a warning message emitted;
>>>>> however,
>>>>>>>> in
>>>>>>>>>>>> Kafka 5.0, setting an
>>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> If we agree to support "empty list", the approach of "none" could
>>>> be
>>>>>>>>>>> eliminated, right?
>>>>>>>>>>> 
>>>>>>>>>>> I don't understand the point of "5.0". Could you please share
>> more
>>>>>>>> details?
>>>>>>>>>>> It seems to me changing the type or default value should be fine
>>>> in
>>>>>>>> minor
>>>>>>>>>>> release if we don't break the existing property files.
>>>>>>>>>>> 
>>>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
>>>> using
>>>>>>>> `LIST`
>>>>>>>>>>> instead of `String` does not seems to make sense.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
>>>>>>>>>>> 
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>> 
>>>>>>>>>>>>>> Null could make sense as the default value, but it just means
>>>>> that
>>>>>>>> the
>>>>>>>>>>>>>> config key is not set explicitly.
>>>>>>>>>>>>>> Given that, it's probably simpler to just allow
>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>> Thanks for the explanation. I have updated the KIP accordingly:
>>>> in
>>>>>>>> Kafka
>>>>>>>>>>>> 4.x, cleanup.policy
>>>>>>>>>>>> can be set to an empty list with a warning message emitted;
>>>>> however,
>>>>>>>> in
>>>>>>>>>>>> Kafka 5.0, setting an
>>>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>>>>>>> 
>>>>>>>>>>>>> For another, there is an additional issue which could be
>> handled
>>>>> by
>>>>>>>>>>> this
>>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled as a
>>>>>>> LIST.
>>>>>>>>>>> For
>>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`. Could
>> we
>>>>>>>> change
>>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
>> them
>>>>>>>>>>>>> non-nullable.
>>>>>>>>>>>> I have also updated the KIP to include the tables detailing the
>>>>>>>> changes
>>>>>>>>>>>> for LIST-type configuration
>>>>>>>>>>>> default values and the migration from STRING-type to LIST-type
>>>>>>>>>>>> configurations.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
>>>> allows
>>>>>>> an
>>>>>>>>>>>> empty
>>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok
>> to
>>>>>>>>>>> support
>>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler to
>>>> just
>>>>>>>>>>> allow
>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> yes, that is what I meant before. Additionally, the `LIST` type
>>>>>>>> should
>>>>>>>>>>>> not
>>>>>>>>>>>>> accept "null". Instead, we should use an empty list as the
>>>> default
>>>>>>>>>>> value.
>>>>>>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should never
>>>> has
>>>>>>>>>>> "null"
>>>>>>>>>>>>> value. That could eliminate the null check.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> For another, there is an additional issue which could be
>> handled
>>>>> by
>>>>>>>>>>> this
>>>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled as a
>>>>>>> LIST.
>>>>>>>>>>> For
>>>>>>>>>>>>> example, `early.start.listeners`, `security.providers`. Could
>> we
>>>>>>>> change
>>>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
>> them
>>>>>>>>>>>>> non-nullable.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR10. Regarding supporting the null value, I am not sure how
>>>> one
>>>>>>>> could
>>>>>>>>>>>> set
>>>>>>>>>>>>>> a null value in a config file. For example, if you set the
>>>>>>> following
>>>>>>>>>>> in
>>>>>>>>>>>> a
>>>>>>>>>>>>>> config file.
>>>>>>>>>>>>>> configkey=
>>>>>>>>>>>>>> The value of configkey will be an empty list, instead of null,
>>>>>>>> right?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Null could make sense as the default value, but it just means
>>>>> that
>>>>>>>> the
>>>>>>>>>>>>>> config key is not set explicitly.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
>>>> allows
>>>>>>> an
>>>>>>>>>>>> empty
>>>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok
>> to
>>>>>>>>>>> support
>>>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler to
>>>> just
>>>>>>>>>>> allow
>>>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional,
>> that
>>>>> is
>>>>>>>>>>>> right,
>>>>>>>>>>>>>>> so I think
>>>>>>>>>>>>>>> that we can allow null value, but not allow the empty
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR11: In this KIP, the goal is to address configurations that
>>>>>>>> should
>>>>>>>>>>>> not
>>>>>>>>>>>>>>> accept empty arrays.
>>>>>>>>>>>>>>> We want to prevent users from passing in an empty list when
>>>> it's
>>>>>>>> not
>>>>>>>>>>>>>>> valid. I believe there are two
>>>>>>>>>>>>>>> main scenarios to consider:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. Nullable but not empty:
>>>>>>>>>>>>>>> The parameter allows either a null value or a non-empty list.
>>>> In
>>>>>>>> this
>>>>>>>>>>>>>>> case,
>>>>>>>>>>>>>>> if the user provides an empty list, we should throw a
>>>>>>>>>>> ConfigException.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
>>>>>>> contain a
>>>>>>>>>>>>>>> non-empty list — null is also
>>>>>>>>>>>>>>> invalid. In this case, if the user provides either a null or
>>>> an
>>>>>>>> empty
>>>>>>>>>>>>>>> list, we should throw a ConfigException.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Therefore, I want to prevent scenarios where users can pass
>> in
>>>>> an
>>>>>>>>>>> empty
>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
>>>> allowNull
>>>>>>>>>>>>>> property.
>>>>>>>>>>>>>>> This allows us to specify
>>>>>>>>>>>>>>> whether a configuration permits null values or not.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> JR13: I have updated the table
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Sorry for the late reply. A few more comments.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says
>>>> that
>>>>>>>> it's
>>>>>>>>>>>>>>>> optional. So an empty list seems to be valid.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support empty
>>>>>>> lists
>>>>>>>> in
>>>>>>>>>>>>>>>> general, it's probably more consistent to just allow
>>>>>>>>>>>> log.cleanup.policy
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be empty as Luke suggested, instead of introducing a new
>>>> option
>>>>>>>>>>> None.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It
>>>>>>> seems
>>>>>>>>>>> that
>>>>>>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
>>>>> shouldn't
>>>>>>>> be
>>>>>>>>>>>>>> null.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> JR13. The table is very helpful. Could you also describe the
>>>>>>>> old/new
>>>>>>>>>>>>>>>> behavior for each config with an empty list or duplicate
>>>> entry?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I am manually bumping this thread. Any feedback would be
>>>>>>>>>>> appreciated.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
>>>>>>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
>>>>>>>>>>>>>>>>>> provided list is either null or a non-empty list without
>>>>>>>> duplicate
>>>>>>>>>>>>>>>>> entries. I’ve also listed the configurations that will
>>>>>>>>>>>>>>>>>> be validated using this logic. Please feel free to share
>>>> any
>>>>>>>>>>>> feedback
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29
>>>> 寫道：
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> It seems there are quite a few other configs of type list
>>>>>>> (e.g.
>>>>>>>>>>>>>>> several
>>>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>>>> related ones). It would be useful to understand if empty
>>>>>>> lists
>>>>>>>>>>> are
>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>>>> for them or not.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
>>>> jun@confluent.io>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
>> introduced?),
>>>>>>>> there
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
>>>> time.
>>>>> "
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> If you do a google search on "kafka infinite retention",
>>>>> you
>>>>>>>>>>> will
>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>> links like
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>> both pointing to setting the retention time and
>> retention
>>>>>>> size
>>>>>>>>>>> to
>>>>>>>>>>>>>> -1
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
>>>>> intentionally
>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>> retention, it's more likely they will use that approach
>>>>>>>> instead
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> setting
>>>>>>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our
>> API/tools
>>>>>>> make
>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving the
>>>>>>>> config
>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
>>>>>>> showuon@gmail.com
>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
>>>> documented
>>>>>>>> that
>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
>>>> one
>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
>>>>>>> since
>>>>>>>>>>> it
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
>> too.
>>>>>>> So,
>>>>>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
>>>> mistake
>>>>>>>> than
>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
>>>>>>> behind
>>>>>>>>>>> the
>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware
>>>> of
>>>>>>>> this
>>>>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
>>>> implies
>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>> retention, we also never documented (or failed) the
>>>> empty
>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
>>>> behavior
>>>>>>>> has
>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>>> existed
>>>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
>>>> introduced?),
>>>>>>>> there
>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
>>>> time.
>>>>>>> It
>>>>>>>>>>> is
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>>>>>>>> to break the backward compatibility in a minor release
>>>>>>>> version
>>>>>>>>>>>>>> (ex:
>>>>>>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
>>>>> existing
>>>>>>>>>>> bug,
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> be a surprise to users.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>> cleanup.policy
>>>>>>> by
>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>>>>>>> realize
>>>>>>>>>>> this
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x
>>>> and
>>>>>>> in
>>>>>>>>>>> 5.0
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>> exception directly to make it fail fast
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> This suggestion looks good to me!
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is
>>>>>>> worth
>>>>>>>>>>>>>>>>> considering
>>>>>>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
>>>>>>>> encountered
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
>>>>>>>> configurations
>>>>>>>>>>>>>>>>> instead.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>>> cleanup.policy
>>>>>>>> by
>>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>>>>>>>> realize
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense
>>>> for
>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered
>>>>>>> storage
>>>>>>>>>>> use
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>>>>>>> delete policy.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
>>>> documented
>>>>>>>> that
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact,
>> if
>>>>>>> one
>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
>>>> storage
>>>>>>>> since
>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
>>>> too.
>>>>>>>> So,
>>>>>>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
>>>>> mistake
>>>>>>>>>>> than
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
>> idea
>>>>>>>> behind
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
>> aware
>>>>> of
>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
>>>> wanting
>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>>>> cleanup.policy
>>>>>>>> by
>>>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>>>>>>>> realize
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
>>>>>>>> showuon@gmail.com
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Comments:
>>>>>>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy
>>>> is
>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
>>>>>>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
>>>>> trade-off
>>>>>>>>>>>>>>> justified?
>>>>>>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in
>>>>>>> kafka,
>>>>>>>>>>> why
>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
>>>>>>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep
>> all
>>>>>>>>>>> history
>>>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>>>> a cleanup policy.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for
>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
>>>>>>>> "process.roles"
>>>>>>>>>>>>>>> configs
>>>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>>>> they are invalid.
>>>>>>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive
>>>>>>>> reason
>>>>>>>>>>>> why
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
>>>>>>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to configure
>>>>> Kafka
>>>>>>>> to
>>>>>>>>>>>>>>> retain
>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
>>>>>>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but
>>>> to
>>>>>>> me,
>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a clear
>>>> and
>>>>>>>>>>> direct
>>>>>>>>>>>>>> way
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>> kafka".
>>>>>>>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy"
>>>>> config?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
>>>>>>>>>>>>>> <jun@confluent.io.invalid
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
>>>>> s7133700@gmail.com
>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
>>>>>>>> addressed
>>>>>>>>>>> in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
>>>>>>>>>>> Additionally,
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
>>>>>>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
>>>>>>>>>>>> inNonEmpty
>>>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日
>>>>>>>> 凌晨12:46
>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass
>>>> empty
>>>>>>>>>>>> config,
>>>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw
>> exception
>>>>>>>>>>>> directly.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日
>> 週二
>>>>>>>>>>>>>> 上午12:28寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references
>>>> to
>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed
>>>> in
>>>>>>>> Kafka
>>>>>>>>>>>>>> 5.0"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
>>>>>>>>>>> addressed
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>>>> 2025年5月3日
>>>>>>>>>>> 凌晨2:03
>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
>>>> Migration
>>>>>>>>>>> Plan",
>>>>>>>>>>>>>>>>>>>>>> could you
>>>>>>>>>>>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
>>>>> "clean.policy="
>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> replaced
>>>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable
>>>>>>>> delete
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> compact.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>> 2025年5月3日
>>>>> 週六
>>>>>>>>>>>>>>>>>>>>> 上午1:55寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for
>>>> now.
>>>>>>>>>>> Could
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
>>>>>>>>>>> deprecation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
>>>>>>> ValidList#in
>>>>>>>>>>>>>>>>>>>>> method,
>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
>>>> useful
>>>>>>> to
>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>>>> methods:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty
>>>>> lists).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
>>>>>>> fail
>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>> 2025年5月2日
>>>>>>>>>>>>>> 凌晨12:50
>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
>>>>>>> fail
>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need
>> to
>>>>>>>>>>> maintain
>>>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
>>>> isEmptyAllowed
>>>>>>>> flag
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of
>> isNonEmpty()
>>>>> to
>>>>>>>>>>>> remove
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> boolean
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>> 2025年5月1日
>>>>>>>>>>>>>> 凌晨4:17
>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's
>>>>>>> awkward
>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when
>> it's
>>>>>>> set
>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> true.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
>>>>> introduce
>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that
>> we
>>>>>>> now
>>>>>>>>>>>>>>>>>>>>> explicitly
>>>>>>>>>>>>>>>>>>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
>>>> format
>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>> phase
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
>>>>> behavior.”
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>>>>> 2025年4月25日
>>>>>>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
>>>> introduced
>>>>>>> by
>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown
>>>> during
>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
>>>> startup."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the
>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>>>> and
>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to
>>>>> the
>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty
>>>>>>> list
>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during
>>>>> storage
>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>>>>>> 2025年4月24日
>>>>>>>>>>>>>>>>>>>>>> 凌晨4:44
>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right
>>>> now.
>>>>>>>> With
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
>>>>>>>> accurate)
>>>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of
>>>>> agree
>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping.
>>>>>>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more
>> likely
>>>>> to
>>>>>>>> be
>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> mistake
>>>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None,
>>>>> the
>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will
>>>> let
>>>>>>> the
>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
>>>>> Chia-Ping
>>>>>>>>>>> Tsai
>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand
>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> maintaining
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
>>>>> documentation,
>>>>>>>> we
>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This
>>>> could
>>>>>>>>>>>>>> streamline
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid>
>> 於
>>>>>>>>>>>>>> 2025年4月23日
>>>>>>>>>>>>>>>>>>>>> 週三
>>>>>>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not
>>>>>>>> delete
>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>> compact
>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't
>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>> segments
>>>>>>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
>>>> log.retention.ms,
>>>>>>>> but
>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
>>>>>>>>>>> log.local.retention.bytes/
>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk
>> space
>>>>>>> when
>>>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun
>>>>>>> Rao <
>>>>>>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
>>>> values
>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>> group.coordinator.rebalance.protocols
>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM
>>>> 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
>>>>>>>> accordingly
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> revised
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy
>>>>>>> cannot
>>>>>>>> be
>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an
>> empty
>>>>>>>>>>>>>>>>>>>>>> cleanup.policy,
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig
>>>>>>> step.
>>>>>>>>>>>>>>>>>>>>>>>> Additionally, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
>> <jun@confluent.io.INVALID
>>>>> 
>>>>>>> 於
>>>>>>>>>>>>>>>>>>>>> 2025年4月22日
>>>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:52
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
>>>> comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new
>>>>>>> value
>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are
>> valid.
>>>>>>> For
>>>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be
>>>>>>> useful
>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during
>>>>>>> upgrade
>>>>>>>>>>> when
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not
>>>> clear
>>>>>>>> what
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM
>>>> 黃竣陽
>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
>>>> discussion
>>>>>>> on
>>>>>>>>>>>>>>>>>>>>> KIP-1161:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve
>>>> the
>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left
>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
>>>> proposed
>>>>>>>>>>>>>>>>>>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate
>>>>>>>> whether
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value
>>>>> for
>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply.

JR 26. early.start.listeners and interceptor.classes: It seems that both
could be empty?

JR27. The following configs of type LIST are missing.

ConsumerConfig
partition.assignment.strategy

BrokerSecurityConfigs

QuorumConfig

LogConfig
leader.replication.throttled.replicas
follower.replication.throttled.replicas

StreamsConfig
rack.aware.assignment.tags
bootstrap.servers

MirrorCheckpointConfig
groups
groups.exclude

MirrorCheckpointConfig
task.assigned.groups

MirrorSourceConfig
topics
topics.exclude
config.properties.exclude

ConnectorConfig
transforms
predicates

SourceConnectorConfig
topic.creation.groups

WorkerConfig
plugin.path

DistributedConfig
inter.worker.verification.algorithms

RestServerConfig
rest.extension.classes
admin.listeners
listeners

Thanks,

Jun

On Tue, Jul 15, 2025 at 6:22 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun Chia,
>
> Thanks for the reply,
>
> JR20: This configuration should remain unchanged. I have updated the KIP.
>
> JR21: For both Consumer and Producer, the default value is an empty list,
> and the
> validator used is NonNullValidator. I think we can change the validator to
> ValidList.anyNonDuplicateValues
> with isNullAllowed = false and isEmptyAllowed = false.
>
> JR23: I’ve addressed this change in the KIP. Although the default value of
> early.start.listeners remains
> unchanged, the validator now disallows empty lists.
>
> JR24: Since log.dirs does not change its default value, I used a dash (-)
> to indicate that the field remains unchanged.
>
> JR25: I’ve updated the table and aligned similar validators as much as
> possible for consistency.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月15日 下午2:12 寫道：
> >
> >> JR20. controller.listener.names : It's required on the broker. Is it
> > required on the controller?
> >
> > there is a small "sugar" in the controller, which we will
> > create listener.security.protocol.map based on controller.listener.names
> > if listener.security.protocol.map is not explicitly set [0]
> > Additionally, `controller.listener.names` is also used to filter out the
> > advertised controller listeners [1]
> >
> >
> >> JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
> > null, right?
> >
> > I don't think so. There is no null check after calling
> >
> `getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
> > which causes an NPE if getList returns null."
> >
> > [0]
> >
> https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
> > [1]
> >
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
> > [2]
> >
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
> >
> > Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：
> >
> >> Hi, Jiunn-Yang,
> >>
> >> Thanks for the updated KIP.
> >>
> >> JR20. controller.listener.names : It's required on the broker. Is it
> >> required on the controller?
> >>
> >> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer)
> already
> >> require it to be non-null.
> >>
> >> JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
> >> null, right?
> >>
> >> JR23. early.start.listeners: It allows default value of null, which
> means
> >> that all listeners included in controller.listener.names will also be
> early
> >> start listeners.
> >>
> >> JR24. log.dirs: It currently has a default value of null.
> >>
> >> JR25. It seems there are quite a few more configs of type list. Do they
> >> need to be changed too?
> >>
> >> AdminClientConfig: 4
> >> ConsumerConfig: 5
> >> ProducerConfig: 4
> >> SaslConfigs: 1
> >> SslConfigs: 2
> >> BrokerSecurityConfigs: 5
> >> MirrorClientConfig: 1
> >> org.apache.kafka.connect.mirror : 16
> >> org.apache.kafka.connect.runtime : 21
> >> GroupCoordinatorConfig: 3
> >> QuorumConfig: 2
> >> ServerConfigs: 1
> >> KRaftConfigs: 1
> >> ClientMetricsConfigs: 2
> >> MetricConfigs: 2
> >> LogConfig: 4
> >> StreamsConfig: 4
> >>
> >> Jun
> >>
> >> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hi Chia, Jun
> >>>
> >>> Thanks for the reply,
> >>> I have updated the table in the KIP.
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
> >>>>
> >>>> Hi, Jun
> >>>>
> >>>> Thanks for the reply.
> >>>>
> >>>> I think the following configs could use null as the default value and
> >>> treat
> >>>> an "empty list" as an illegal value.
> >>>>
> >>>> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
> >>>> `ssl.enabled.protocols`,
> >>>> and `log.dirs`
> >>>>
> >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
> >>>>
> >>>>> Hi, Chia-Ping, Jiunn-Yang,
> >>>>>
> >>>>> Thanks for the reply.
> >>>>>
> >>>>> As you said, if ssl.cipher.suites is empty, intuitively this means
> >> that
> >>> no
> >>>>> cipher is allowed. So, this should be an illegal config. It just
> >> happens
> >>>>> that our implementation treats it as if the config is not set
> (meaning
> >>> the
> >>>>> value is null). So, it seems that a more intuitive convention is: if
> >> the
> >>>>> config is not set, it defaults to all available cipher suites. If
> it's
> >>>>> explicitly set, it can't be empty.
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>
> >>>>>> Hi Jun, Chia
> >>>>>>
> >>>>>>>> Also, it would be useful to add a compatibility column in the
> >> table.
> >>>>> If
> >>>>>> a
> >>>>>>>> value is formally disallowed, it would be useful to compare the
> old
> >>>>> and
> >>>>>> the
> >>>>>>>> new behavior (e.g. a different exception is thrown, etc).
> >>>>>> I’ve updated the table to show which exceptions will be thrown for
> >> each
> >>>>>> behavior.
> >>>>>>
> >>>>>>>>> I don't understand the point of "5.0". Could you please share
> more
> >>>>>> details?
> >>>>>>>>> It seems to me changing the type or default value should be fine
> >> in
> >>>>>> minor
> >>>>>>>>> release if we don't break the existing property files.
> >>>>>> I don't think this requires any compatibility considerations, so
> I’ve
> >>>>>> removed the Kafka 5.0 changes section.
> >>>>>>
> >>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
> >> using
> >>>>>> `LIST`
> >>>>>>>>> instead of `String` does not seems to make sense.
> >>>>>> I have removed the config from KIP
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> >>>>>>>
> >>>>>>>> For ssl.cipher.suites, changing the default value from null to
> >> empty
> >>>>> is
> >>>>>>>> actually a breaking change. According to the doc, null means that
> >> all
> >>>>>> the
> >>>>>>>> available cipher suites are supported. I am thinking that we
> should
> >>>>>>>> continue to allow null as the default since it could represent a
> >>> state
> >>>>>>>> different from empty.
> >>>>>>>
> >>>>>>> "Empty list" seems to be a weird case. It appears to signify that
> no
> >>>>>> cipher suites are accepted. However, in the codebase, an empty list
> >> is
> >>>>>> handled as if all available cipher suites are supported [0]. We
> >> should
> >>>>> not
> >>>>>> support the case of "no supported cipher suite," as it doesn't make
> >>>>> sense.
> >>>>>>>
> >>>>>>> In short, changing the default value from null to empty does not
> >> break
> >>>>>> the behavior.
> >>>>>>>
> >>>>>>> [0]
> >>>>>>
> >>>>>
> >>>
> >>
> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> >>>>>>>
> >>>>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
> >>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we
> >>>>> will
> >>>>>>>> just continue to allow it in all future releases.
> >>>>>>>>
> >>>>>>>> For ssl.cipher.suites, changing the default value from null to
> >> empty
> >>>>> is
> >>>>>>>> actually a breaking change. According to the doc, null means that
> >> all
> >>>>>> the
> >>>>>>>> available cipher suites are supported. I am thinking that we
> should
> >>>>>>>> continue to allow null as the default since it could represent a
> >>> state
> >>>>>>>> different from empty.
> >>>>>>>>
> >>>>>>>> Also, it would be useful to add a compatibility column in the
> >> table.
> >>>>> If
> >>>>>> a
> >>>>>>>> value is formally disallowed, it would be useful to compare the
> old
> >>>>> and
> >>>>>> the
> >>>>>>>> new behavior (e.g. a different exception is thrown, etc).
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <chia7712@gmail.com
> >
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> hi Jiunn
> >>>>>>>>>
> >>>>>>>>> Thanks for the explanation. I have updated the KIP accordingly:
> in
> >>>>>> Kafka
> >>>>>>>>>> 4.x, cleanup.policy
> >>>>>>>>>> can be set to an empty list with a warning message emitted;
> >>> however,
> >>>>>> in
> >>>>>>>>>> Kafka 5.0, setting an
> >>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> If we agree to support "empty list", the approach of "none" could
> >> be
> >>>>>>>>> eliminated, right?
> >>>>>>>>>
> >>>>>>>>> I don't understand the point of "5.0". Could you please share
> more
> >>>>>> details?
> >>>>>>>>> It seems to me changing the type or default value should be fine
> >> in
> >>>>>> minor
> >>>>>>>>> release if we don't break the existing property files.
> >>>>>>>>>
> >>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
> >> using
> >>>>>> `LIST`
> >>>>>>>>> instead of `String` does not seems to make sense.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Chia-Ping
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> >>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>>>> Null could make sense as the default value, but it just means
> >>> that
> >>>>>> the
> >>>>>>>>>>>> config key is not set explicitly.
> >>>>>>>>>>>> Given that, it's probably simpler to just allow
> >>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>> Thanks for the explanation. I have updated the KIP accordingly:
> >> in
> >>>>>> Kafka
> >>>>>>>>>> 4.x, cleanup.policy
> >>>>>>>>>> can be set to an empty list with a warning message emitted;
> >>> however,
> >>>>>> in
> >>>>>>>>>> Kafka 5.0, setting an
> >>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
> >>>>>>>>>>
> >>>>>>>>>>> For another, there is an additional issue which could be
> handled
> >>> by
> >>>>>>>>> this
> >>>>>>>>>>> KIP. Some configs have "string" type, but they are handled as a
> >>>>> LIST.
> >>>>>>>>> For
> >>>>>>>>>>> example, `early.start.listeners`, `security.providers`. Could
> we
> >>>>>> change
> >>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
> them
> >>>>>>>>>>> non-nullable.
> >>>>>>>>>> I have also updated the KIP to include the tables detailing the
> >>>>>> changes
> >>>>>>>>>> for LIST-type configuration
> >>>>>>>>>> default values and the migration from STRING-type to LIST-type
> >>>>>>>>>> configurations.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>
> >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> >>>>>>>>>>>
> >>>>>>>>>>> hi all,
> >>>>>>>>>>>
> >>>>>>>>>>> JR11. There is an existing config interceptor.classes that
> >> allows
> >>>>> an
> >>>>>>>>>> empty
> >>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok
> to
> >>>>>>>>> support
> >>>>>>>>>>>> empty lists in general. Given that, it's probably simpler to
> >> just
> >>>>>>>>> allow
> >>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> yes, that is what I meant before. Additionally, the `LIST` type
> >>>>>> should
> >>>>>>>>>> not
> >>>>>>>>>>> accept "null". Instead, we should use an empty list as the
> >> default
> >>>>>>>>> value.
> >>>>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should never
> >> has
> >>>>>>>>> "null"
> >>>>>>>>>>> value. That could eliminate the null check.
> >>>>>>>>>>>
> >>>>>>>>>>> For another, there is an additional issue which could be
> handled
> >>> by
> >>>>>>>>> this
> >>>>>>>>>>> KIP. Some configs have "string" type, but they are handled as a
> >>>>> LIST.
> >>>>>>>>> For
> >>>>>>>>>>> example, `early.start.listeners`, `security.providers`. Could
> we
> >>>>>> change
> >>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make
> them
> >>>>>>>>>>> non-nullable.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR10. Regarding supporting the null value, I am not sure how
> >> one
> >>>>>> could
> >>>>>>>>>> set
> >>>>>>>>>>>> a null value in a config file. For example, if you set the
> >>>>> following
> >>>>>>>>> in
> >>>>>>>>>> a
> >>>>>>>>>>>> config file.
> >>>>>>>>>>>> configkey=
> >>>>>>>>>>>> The value of configkey will be an empty list, instead of null,
> >>>>>> right?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Null could make sense as the default value, but it just means
> >>> that
> >>>>>> the
> >>>>>>>>>>>> config key is not set explicitly.
> >>>>>>>>>>>>
> >>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
> >> allows
> >>>>> an
> >>>>>>>>>> empty
> >>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok
> to
> >>>>>>>>> support
> >>>>>>>>>>>> empty lists in general. Given that, it's probably simpler to
> >> just
> >>>>>>>>> allow
> >>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional,
> that
> >>> is
> >>>>>>>>>> right,
> >>>>>>>>>>>>> so I think
> >>>>>>>>>>>>> that we can allow null value, but not allow the empty
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR11: In this KIP, the goal is to address configurations that
> >>>>>> should
> >>>>>>>>>> not
> >>>>>>>>>>>>> accept empty arrays.
> >>>>>>>>>>>>> We want to prevent users from passing in an empty list when
> >> it's
> >>>>>> not
> >>>>>>>>>>>>> valid. I believe there are two
> >>>>>>>>>>>>> main scenarios to consider:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1. Nullable but not empty:
> >>>>>>>>>>>>> The parameter allows either a null value or a non-empty list.
> >> In
> >>>>>> this
> >>>>>>>>>>>>> case,
> >>>>>>>>>>>>> if the user provides an empty list, we should throw a
> >>>>>>>>> ConfigException.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
> >>>>> contain a
> >>>>>>>>>>>>> non-empty list — null is also
> >>>>>>>>>>>>> invalid. In this case, if the user provides either a null or
> >> an
> >>>>>> empty
> >>>>>>>>>>>>> list, we should throw a ConfigException.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Therefore, I want to prevent scenarios where users can pass
> in
> >>> an
> >>>>>>>>> empty
> >>>>>>>>>>>>> list.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
> >> allowNull
> >>>>>>>>>>>> property.
> >>>>>>>>>>>>> This allows us to specify
> >>>>>>>>>>>>> whether a configuration permits null values or not.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR13: I have updated the table
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sorry for the late reply. A few more comments.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says
> >> that
> >>>>>> it's
> >>>>>>>>>>>>>> optional. So an empty list seems to be valid.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support empty
> >>>>> lists
> >>>>>> in
> >>>>>>>>>>>>>> general, it's probably more consistent to just allow
> >>>>>>>>>> log.cleanup.policy
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>> be empty as Luke suggested, instead of introducing a new
> >> option
> >>>>>>>>> None.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It
> >>>>> seems
> >>>>>>>>> that
> >>>>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
> >>> shouldn't
> >>>>>> be
> >>>>>>>>>>>> null.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> JR13. The table is very helpful. Could you also describe the
> >>>>>> old/new
> >>>>>>>>>>>>>> behavior for each config with an empty list or duplicate
> >> entry?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com>
> >>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I am manually bumping this thread. Any feedback would be
> >>>>>>>>> appreciated.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
> >>>>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
> >>>>>>>>>>>>>>>> provided list is either null or a non-empty list without
> >>>>>> duplicate
> >>>>>>>>>>>>>>> entries. I’ve also listed the configurations that will
> >>>>>>>>>>>>>>>> be validated using this logic. Please feel free to share
> >> any
> >>>>>>>>>> feedback
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>>>> suggestions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29
> >> 寫道：
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It seems there are quite a few other configs of type list
> >>>>> (e.g.
> >>>>>>>>>>>>> several
> >>>>>>>>>>>>>>> SSL
> >>>>>>>>>>>>>>>>> related ones). It would be useful to understand if empty
> >>>>> lists
> >>>>>>>>> are
> >>>>>>>>>>>>> valid
> >>>>>>>>>>>>>>>>> for them or not.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
> >> jun@confluent.io>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
> >>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> introduced?),
> >>>>>> there
> >>>>>>>>>>>> could
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
> >> time.
> >>> "
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> If you do a google search on "kafka infinite retention",
> >>> you
> >>>>>>>>> will
> >>>>>>>>>>>> get
> >>>>>>>>>>>>>>>>>> links like
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> >>>>>>>>>>>>>>> ,
> >>>>>>>>>>>>>>>>>> both pointing to setting the retention time and
> retention
> >>>>> size
> >>>>>>>>> to
> >>>>>>>>>>>> -1
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
> >>> intentionally
> >>>>>>>>> wants
> >>>>>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>> retention, it's more likely they will use that approach
> >>>>>> instead
> >>>>>>>>> of
> >>>>>>>>>>>>>>> setting
> >>>>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our
> API/tools
> >>>>> make
> >>>>>>>>> it
> >>>>>>>>>>>>>>> possible
> >>>>>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving the
> >>>>>> config
> >>>>>>>>>>>> value
> >>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
> >>>>> showuon@gmail.com
> >>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> >> documented
> >>>>>> that
> >>>>>>>>> an
> >>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
> >> one
> >>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
> >>>>> since
> >>>>>>>>> it
> >>>>>>>>>>>>> will
> >>>>>>>>>>>>>>> stop
> >>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
> too.
> >>>>> So,
> >>>>>>>>>>>> leaving
> >>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> >> mistake
> >>>>>> than
> >>>>>>>>> an
> >>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
> >>>>> behind
> >>>>>>>>> the
> >>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware
> >> of
> >>>>>> this
> >>>>>>>>>>>>> likely
> >>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
> >>>>>>>>> infinite
> >>>>>>>>>>>>>>> retention
> >>>>>>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
> >> implies
> >>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>> retention, we also never documented (or failed) the
> >> empty
> >>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
> >> behavior
> >>>>>> has
> >>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>>> existed
> >>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> >> introduced?),
> >>>>>> there
> >>>>>>>>>>>>> could
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
> >> time.
> >>>>> It
> >>>>>>>>> is
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>> good
> >>>>>>>>>>>>>>>>>>> to break the backward compatibility in a minor release
> >>>>>> version
> >>>>>>>>>>>> (ex:
> >>>>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
> >>> existing
> >>>>>>>>> bug,
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>> be a surprise to users.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >> cleanup.policy
> >>>>> by
> >>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> >>>>> realize
> >>>>>>>>> this
> >>>>>>>>>>>> if
> >>>>>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x
> >> and
> >>>>> in
> >>>>>>>>> 5.0
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>> exception directly to make it fail fast
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> This suggestion looks good to me!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thank you.
> >>>>>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> >>>>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> hi all,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is
> >>>>> worth
> >>>>>>>>>>>>>>> considering
> >>>>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
> >>>>>> encountered
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
> >>>>>> configurations
> >>>>>>>>>>>>>>> instead.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >>> cleanup.policy
> >>>>>> by
> >>>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> >>>>>> realize
> >>>>>>>>>>>> this
> >>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense
> >> for
> >>>>>>>>> remote
> >>>>>>>>>>>>>>>>>>> storage.
> >>>>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered
> >>>>> storage
> >>>>>>>>> use
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>> valid
> >>>>>>>>>>>>>>>>>>>> delete policy.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> >>>>>>>>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> >> documented
> >>>>>> that
> >>>>>>>>>> an
> >>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact,
> if
> >>>>> one
> >>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
> >> storage
> >>>>>> since
> >>>>>>>>>> it
> >>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>> stop
> >>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
> >> too.
> >>>>>> So,
> >>>>>>>>>>>>> leaving
> >>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> >>> mistake
> >>>>>>>>> than
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the
> idea
> >>>>>> behind
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is
> aware
> >>> of
> >>>>>>>>> this
> >>>>>>>>>>>>>>> likely
> >>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
> >> wanting
> >>>>>>>>>> infinite
> >>>>>>>>>>>>>>>>>>>> retention
> >>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> >>> cleanup.policy
> >>>>>> by
> >>>>>>>>>>>>> fixing
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> >>>>>> realize
> >>>>>>>>>>>> this
> >>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
> >>>>>> showuon@gmail.com
> >>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Comments:
> >>>>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy
> >> is
> >>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> >>>>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
> >>> trade-off
> >>>>>>>>>>>>> justified?
> >>>>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in
> >>>>> kafka,
> >>>>>>>>> why
> >>>>>>>>>>>> do
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>> provide
> >>>>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
> >>>>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep
> all
> >>>>>>>>> history
> >>>>>>>>>>>>> data
> >>>>>>>>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>>>>>>>> a cleanup policy.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for
> >> empty
> >>>>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
> >>>>>> "process.roles"
> >>>>>>>>>>>>> configs
> >>>>>>>>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>>>>> they are invalid.
> >>>>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive
> >>>>>> reason
> >>>>>>>>>> why
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
> >>>>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to configure
> >>> Kafka
> >>>>>> to
> >>>>>>>>>>>>> retain
> >>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
> >>>>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but
> >> to
> >>>>> me,
> >>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a clear
> >> and
> >>>>>>>>> direct
> >>>>>>>>>>>> way
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>> kafka".
> >>>>>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy"
> >>> config?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> >>>>>>>>>>>> <jun@confluent.io.invalid
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
> >>> s7133700@gmail.com
> >>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> >>>>>> addressed
> >>>>>>>>> in
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
> >>>>>>>>> Additionally,
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
> >>>>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
> >>>>>>>>>> inNonEmpty
> >>>>>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日
> >>>>>> 凌晨12:46
> >>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass
> >> empty
> >>>>>>>>>> config,
> >>>>>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw
> exception
> >>>>>>>>>> directly.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日
> 週二
> >>>>>>>>>>>> 上午12:28寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references
> >> to
> >>>>>> the
> >>>>>>>>>>>>>>>>>>>> deprecated
> >>>>>>>>>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> >>>>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed
> >> in
> >>>>>> Kafka
> >>>>>>>>>>>> 5.0"
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
> >>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> >>>>>>>>> addressed
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> >> 2025年5月3日
> >>>>>>>>> 凌晨2:03
> >>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
> >> Migration
> >>>>>>>>> Plan",
> >>>>>>>>>>>>>>>>>>>> could you
> >>>>>>>>>>>>>>>>>>>>>>> add
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
> >>> "clean.policy="
> >>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>> replaced
> >>>>>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable
> >>>>>> delete
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> compact.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> 2025年5月3日
> >>> 週六
> >>>>>>>>>>>>>>>>>>> 上午1:55寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for
> >> now.
> >>>>>>>>> Could
> >>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>> update
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
> >>>>>>>>> deprecation.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> >>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
> >>>>> ValidList#in
> >>>>>>>>>>>>>>>>>>> method,
> >>>>>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
> >> useful
> >>>>> to
> >>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>>>>> methods:
> >>>>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty
> >>> lists).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> >>>>>>>>> document
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
> >>>>> fail
> >>>>>> to
> >>>>>>>>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>> 2025年5月2日
> >>>>>>>>>>>> 凌晨12:50
> >>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> >>>>>>>>> document
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
> >>>>> fail
> >>>>>> to
> >>>>>>>>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> >>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need
> to
> >>>>>>>>> maintain
> >>>>>>>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
> >> isEmptyAllowed
> >>>>>> flag
> >>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of
> isNonEmpty()
> >>> to
> >>>>>>>>>> remove
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> boolean
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>> 2025年5月1日
> >>>>>>>>>>>> 凌晨4:17
> >>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's
> >>>>> awkward
> >>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when
> it's
> >>>>> set
> >>>>>> to
> >>>>>>>>>>>>>>>>>>> true.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> >>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
> >>> introduce
> >>>>>> new
> >>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that
> we
> >>>>> now
> >>>>>>>>>>>>>>>>>>> explicitly
> >>>>>>>>>>>>>>>>>>>>>>> throw a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
> >> format
> >>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>> phase
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
> >>> behavior.”
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>>>>> 2025年4月25日
> >>>>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
> >> introduced
> >>>>> by
> >>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown
> >> during
> >>>>>> the
> >>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
> >> startup."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the
> >>>>> storage
> >>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>> fails
> >>>>>>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> >> and
> >>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> >>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> >>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to
> >>> the
> >>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty
> >>>>> list
> >>>>>>>>> will
> >>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during
> >>> storage
> >>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>>>>>> 2025年4月24日
> >>>>>>>>>>>>>>>>>>>> 凌晨4:44
> >>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> >>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> >>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right
> >> now.
> >>>>>> With
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
> >>>>>> accurate)
> >>>>>>>>>>>>>>>>>>>> exception.
> >>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of
> >>> agree
> >>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>> Chia-Ping.
> >>>>>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more
> likely
> >>> to
> >>>>>> be
> >>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> mistake
> >>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None,
> >>> the
> >>>>>>>>> user
> >>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>> never
> >>>>>>>>>>>>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will
> >> let
> >>>>> the
> >>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
> >>> Chia-Ping
> >>>>>>>>> Tsai
> >>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand
> >>>>> that
> >>>>>>>>>>>>>>>>>>>> maintaining
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
> >>> documentation,
> >>>>>> we
> >>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>> allow
> >>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
> >>>>>>>>> consider
> >>>>>>>>>>>>>>>>>>>> throwing
> >>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This
> >> could
> >>>>>>>>>>>> streamline
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid>
> 於
> >>>>>>>>>>>> 2025年4月23日
> >>>>>>>>>>>>>>>>>>> 週三
> >>>>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not
> >>>>>> delete
> >>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> compact
> >>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't
> >>>>> delete
> >>>>>>>>>>>>>>>>>>> segments
> >>>>>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
> >> log.retention.ms,
> >>>>>> but
> >>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>> continue
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> >>>>>>>>> log.local.retention.bytes/
> >>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk
> space
> >>>>> when
> >>>>>>>>>>>> remote
> >>>>>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun
> >>>>> Rao <
> >>>>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
> >> values
> >>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >> group.coordinator.rebalance.protocols
> >>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM
> >> 黃竣陽 <
> >>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
> >>>>>> accordingly
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> revised
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy
> >>>>> cannot
> >>>>>> be
> >>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an
> empty
> >>>>>>>>>>>>>>>>>>>> cleanup.policy,
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig
> >>>>> step.
> >>>>>>>>>>>>>>>>>>>>>> Additionally, a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao
> <jun@confluent.io.INVALID
> >>>
> >>>>> 於
> >>>>>>>>>>>>>>>>>>> 2025年4月22日
> >>>>>>>>>>>>>>>>>>>>>>> 凌晨4:52
> >>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
> >> comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new
> >>>>> value
> >>>>>>>>> None
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are
> valid.
> >>>>> For
> >>>>>>>>>>>>>>>>>>> example,
> >>>>>>>>>>>>>>>>>>>> None
> >>>>>>>>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be
> >>>>> useful
> >>>>>> to
> >>>>>>>>>>>>>>>>>>>> document
> >>>>>>>>>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during
> >>>>> upgrade
> >>>>>>>>> when
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not
> >> clear
> >>>>>> what
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>> check
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> does.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM
> >> 黃竣陽
> >>>>> <
> >>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> >> discussion
> >>>>> on
> >>>>>>>>>>>>>>>>>>> KIP-1161:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve
> >> the
> >>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left
> >>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
> >> proposed
> >>>>>>>>>>>>>>>>>>> improvements:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate
> >>>>>> whether
> >>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value
> >>> for
> >>>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun Chia,

Thanks for the reply,

JR20: This configuration should remain unchanged. I have updated the KIP.

JR21: For both Consumer and Producer, the default value is an empty list, and the 
validator used is NonNullValidator. I think we can change the validator to ValidList.anyNonDuplicateValues 
with isNullAllowed = false and isEmptyAllowed = false. 

JR23: I’ve addressed this change in the KIP. Although the default value of early.start.listeners remains 
unchanged, the validator now disallows empty lists. 

JR24: Since log.dirs does not change its default value, I used a dash (-) to indicate that the field remains unchanged. 

JR25: I’ve updated the table and aligned similar validators as much as possible for consistency.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月15日 下午2:12 寫道：
> 
>> JR20. controller.listener.names : It's required on the broker. Is it
> required on the controller?
> 
> there is a small "sugar" in the controller, which we will
> create listener.security.protocol.map based on controller.listener.names
> if listener.security.protocol.map is not explicitly set [0]
> Additionally, `controller.listener.names` is also used to filter out the
> advertised controller listeners [1]
> 
> 
>> JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
> null, right?
> 
> I don't think so. There is no null check after calling
> `getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
> which causes an NPE if getList returns null."
> 
> [0]
> https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
> [1]
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
> [2]
> https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373
> 
> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：
> 
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the updated KIP.
>> 
>> JR20. controller.listener.names : It's required on the broker. Is it
>> required on the controller?
>> 
>> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer) already
>> require it to be non-null.
>> 
>> JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
>> null, right?
>> 
>> JR23. early.start.listeners: It allows default value of null, which means
>> that all listeners included in controller.listener.names will also be early
>> start listeners.
>> 
>> JR24. log.dirs: It currently has a default value of null.
>> 
>> JR25. It seems there are quite a few more configs of type list. Do they
>> need to be changed too?
>> 
>> AdminClientConfig: 4
>> ConsumerConfig: 5
>> ProducerConfig: 4
>> SaslConfigs: 1
>> SslConfigs: 2
>> BrokerSecurityConfigs: 5
>> MirrorClientConfig: 1
>> org.apache.kafka.connect.mirror : 16
>> org.apache.kafka.connect.runtime : 21
>> GroupCoordinatorConfig: 3
>> QuorumConfig: 2
>> ServerConfigs: 1
>> KRaftConfigs: 1
>> ClientMetricsConfigs: 2
>> MetricConfigs: 2
>> LogConfig: 4
>> StreamsConfig: 4
>> 
>> Jun
>> 
>> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hi Chia, Jun
>>> 
>>> Thanks for the reply,
>>> I have updated the table in the KIP.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
>>>> 
>>>> Hi, Jun
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> I think the following configs could use null as the default value and
>>> treat
>>>> an "empty list" as an illegal value.
>>>> 
>>>> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
>>>> `ssl.enabled.protocols`,
>>>> and `log.dirs`
>>>> 
>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
>>>> 
>>>>> Hi, Chia-Ping, Jiunn-Yang,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> As you said, if ssl.cipher.suites is empty, intuitively this means
>> that
>>> no
>>>>> cipher is allowed. So, this should be an illegal config. It just
>> happens
>>>>> that our implementation treats it as if the config is not set (meaning
>>> the
>>>>> value is null). So, it seems that a more intuitive convention is: if
>> the
>>>>> config is not set, it defaults to all available cipher suites. If it's
>>>>> explicitly set, it can't be empty.
>>>>> 
>>>>> Jun
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hi Jun, Chia
>>>>>> 
>>>>>>>> Also, it would be useful to add a compatibility column in the
>> table.
>>>>> If
>>>>>> a
>>>>>>>> value is formally disallowed, it would be useful to compare the old
>>>>> and
>>>>>> the
>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>>> I’ve updated the table to show which exceptions will be thrown for
>> each
>>>>>> behavior.
>>>>>> 
>>>>>>>>> I don't understand the point of "5.0". Could you please share more
>>>>>> details?
>>>>>>>>> It seems to me changing the type or default value should be fine
>> in
>>>>>> minor
>>>>>>>>> release if we don't break the existing property files.
>>>>>> I don't think this requires any compatibility considerations, so I’ve
>>>>>> removed the Kafka 5.0 changes section.
>>>>>> 
>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
>> using
>>>>>> `LIST`
>>>>>>>>> instead of `String` does not seems to make sense.
>>>>>> I have removed the config from KIP
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
>>>>>>> 
>>>>>>>> For ssl.cipher.suites, changing the default value from null to
>> empty
>>>>> is
>>>>>>>> actually a breaking change. According to the doc, null means that
>> all
>>>>>> the
>>>>>>>> available cipher suites are supported. I am thinking that we should
>>>>>>>> continue to allow null as the default since it could represent a
>>> state
>>>>>>>> different from empty.
>>>>>>> 
>>>>>>> "Empty list" seems to be a weird case. It appears to signify that no
>>>>>> cipher suites are accepted. However, in the codebase, an empty list
>> is
>>>>>> handled as if all available cipher suites are supported [0]. We
>> should
>>>>> not
>>>>>> support the case of "no supported cipher suite," as it doesn't make
>>>>> sense.
>>>>>>> 
>>>>>>> In short, changing the default value from null to empty does not
>> break
>>>>>> the behavior.
>>>>>>> 
>>>>>>> [0]
>>>>>> 
>>>>> 
>>> 
>> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
>>>>>>> 
>>>>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we
>>>>> will
>>>>>>>> just continue to allow it in all future releases.
>>>>>>>> 
>>>>>>>> For ssl.cipher.suites, changing the default value from null to
>> empty
>>>>> is
>>>>>>>> actually a breaking change. According to the doc, null means that
>> all
>>>>>> the
>>>>>>>> available cipher suites are supported. I am thinking that we should
>>>>>>>> continue to allow null as the default since it could represent a
>>> state
>>>>>>>> different from empty.
>>>>>>>> 
>>>>>>>> Also, it would be useful to add a compatibility column in the
>> table.
>>>>> If
>>>>>> a
>>>>>>>> value is formally disallowed, it would be useful to compare the old
>>>>> and
>>>>>> the
>>>>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> hi Jiunn
>>>>>>>>> 
>>>>>>>>> Thanks for the explanation. I have updated the KIP accordingly: in
>>>>>> Kafka
>>>>>>>>>> 4.x, cleanup.policy
>>>>>>>>>> can be set to an empty list with a warning message emitted;
>>> however,
>>>>>> in
>>>>>>>>>> Kafka 5.0, setting an
>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> If we agree to support "empty list", the approach of "none" could
>> be
>>>>>>>>> eliminated, right?
>>>>>>>>> 
>>>>>>>>> I don't understand the point of "5.0". Could you please share more
>>>>>> details?
>>>>>>>>> It seems to me changing the type or default value should be fine
>> in
>>>>>> minor
>>>>>>>>> release if we don't break the existing property files.
>>>>>>>>> 
>>>>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
>> using
>>>>>> `LIST`
>>>>>>>>> instead of `String` does not seems to make sense.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>>>> Null could make sense as the default value, but it just means
>>> that
>>>>>> the
>>>>>>>>>>>> config key is not set explicitly.
>>>>>>>>>>>> Given that, it's probably simpler to just allow
>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>> Thanks for the explanation. I have updated the KIP accordingly:
>> in
>>>>>> Kafka
>>>>>>>>>> 4.x, cleanup.policy
>>>>>>>>>> can be set to an empty list with a warning message emitted;
>>> however,
>>>>>> in
>>>>>>>>>> Kafka 5.0, setting an
>>>>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>>>>> 
>>>>>>>>>>> For another, there is an additional issue which could be handled
>>> by
>>>>>>>>> this
>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled as a
>>>>> LIST.
>>>>>>>>> For
>>>>>>>>>>> example, `early.start.listeners`, `security.providers`. Could we
>>>>>> change
>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make them
>>>>>>>>>>> non-nullable.
>>>>>>>>>> I have also updated the KIP to include the tables detailing the
>>>>>> changes
>>>>>>>>>> for LIST-type configuration
>>>>>>>>>> default values and the migration from STRING-type to LIST-type
>>>>>>>>>> configurations.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> hi all,
>>>>>>>>>>> 
>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
>> allows
>>>>> an
>>>>>>>>>> empty
>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
>>>>>>>>> support
>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler to
>> just
>>>>>>>>> allow
>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> yes, that is what I meant before. Additionally, the `LIST` type
>>>>>> should
>>>>>>>>>> not
>>>>>>>>>>> accept "null". Instead, we should use an empty list as the
>> default
>>>>>>>>> value.
>>>>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should never
>> has
>>>>>>>>> "null"
>>>>>>>>>>> value. That could eliminate the null check.
>>>>>>>>>>> 
>>>>>>>>>>> For another, there is an additional issue which could be handled
>>> by
>>>>>>>>> this
>>>>>>>>>>> KIP. Some configs have "string" type, but they are handled as a
>>>>> LIST.
>>>>>>>>> For
>>>>>>>>>>> example, `early.start.listeners`, `security.providers`. Could we
>>>>>> change
>>>>>>>>>>> their type to LIST to benefit from this KIP? I mean to make them
>>>>>>>>>>> non-nullable.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>> 
>>>>>>>>>>>> JR10. Regarding supporting the null value, I am not sure how
>> one
>>>>>> could
>>>>>>>>>> set
>>>>>>>>>>>> a null value in a config file. For example, if you set the
>>>>> following
>>>>>>>>> in
>>>>>>>>>> a
>>>>>>>>>>>> config file.
>>>>>>>>>>>> configkey=
>>>>>>>>>>>> The value of configkey will be an empty list, instead of null,
>>>>>> right?
>>>>>>>>>>>> 
>>>>>>>>>>>> Null could make sense as the default value, but it just means
>>> that
>>>>>> the
>>>>>>>>>>>> config key is not set explicitly.
>>>>>>>>>>>> 
>>>>>>>>>>>> JR11. There is an existing config interceptor.classes that
>> allows
>>>>> an
>>>>>>>>>> empty
>>>>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
>>>>>>>>> support
>>>>>>>>>>>> empty lists in general. Given that, it's probably simpler to
>> just
>>>>>>>>> allow
>>>>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional, that
>>> is
>>>>>>>>>> right,
>>>>>>>>>>>>> so I think
>>>>>>>>>>>>> that we can allow null value, but not allow the empty
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR11: In this KIP, the goal is to address configurations that
>>>>>> should
>>>>>>>>>> not
>>>>>>>>>>>>> accept empty arrays.
>>>>>>>>>>>>> We want to prevent users from passing in an empty list when
>> it's
>>>>>> not
>>>>>>>>>>>>> valid. I believe there are two
>>>>>>>>>>>>> main scenarios to consider:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 1. Nullable but not empty:
>>>>>>>>>>>>> The parameter allows either a null value or a non-empty list.
>> In
>>>>>> this
>>>>>>>>>>>>> case,
>>>>>>>>>>>>> if the user provides an empty list, we should throw a
>>>>>>>>> ConfigException.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
>>>>> contain a
>>>>>>>>>>>>> non-empty list — null is also
>>>>>>>>>>>>> invalid. In this case, if the user provides either a null or
>> an
>>>>>> empty
>>>>>>>>>>>>> list, we should throw a ConfigException.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Therefore, I want to prevent scenarios where users can pass in
>>> an
>>>>>>>>> empty
>>>>>>>>>>>>> list.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
>> allowNull
>>>>>>>>>>>> property.
>>>>>>>>>>>>> This allows us to specify
>>>>>>>>>>>>> whether a configuration permits null values or not.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR13: I have updated the table
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Sorry for the late reply. A few more comments.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says
>> that
>>>>>> it's
>>>>>>>>>>>>>> optional. So an empty list seems to be valid.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support empty
>>>>> lists
>>>>>> in
>>>>>>>>>>>>>> general, it's probably more consistent to just allow
>>>>>>>>>> log.cleanup.policy
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> be empty as Luke suggested, instead of introducing a new
>> option
>>>>>>>>> None.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It
>>>>> seems
>>>>>>>>> that
>>>>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
>>> shouldn't
>>>>>> be
>>>>>>>>>>>> null.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> JR13. The table is very helpful. Could you also describe the
>>>>>> old/new
>>>>>>>>>>>>>> behavior for each config with an empty list or duplicate
>> entry?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I am manually bumping this thread. Any feedback would be
>>>>>>>>> appreciated.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
>>>>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
>>>>>>>>>>>>>>>> provided list is either null or a non-empty list without
>>>>>> duplicate
>>>>>>>>>>>>>>> entries. I’ve also listed the configurations that will
>>>>>>>>>>>>>>>> be validated using this logic. Please feel free to share
>> any
>>>>>>>>>> feedback
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29
>> 寫道：
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> It seems there are quite a few other configs of type list
>>>>> (e.g.
>>>>>>>>>>>>> several
>>>>>>>>>>>>>>> SSL
>>>>>>>>>>>>>>>>> related ones). It would be useful to understand if empty
>>>>> lists
>>>>>>>>> are
>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>> for them or not.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
>> jun@confluent.io>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
>>>>>> there
>>>>>>>>>>>> could
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
>> time.
>>> "
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> If you do a google search on "kafka infinite retention",
>>> you
>>>>>>>>> will
>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>> links like
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>> both pointing to setting the retention time and retention
>>>>> size
>>>>>>>>> to
>>>>>>>>>>>> -1
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
>>> intentionally
>>>>>>>>> wants
>>>>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>> retention, it's more likely they will use that approach
>>>>>> instead
>>>>>>>>> of
>>>>>>>>>>>>>>> setting
>>>>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our API/tools
>>>>> make
>>>>>>>>> it
>>>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving the
>>>>>> config
>>>>>>>>>>>> value
>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
>>>>> showuon@gmail.com
>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
>> documented
>>>>>> that
>>>>>>>>> an
>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
>> one
>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
>>>>> since
>>>>>>>>> it
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time too.
>>>>> So,
>>>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
>> mistake
>>>>>> than
>>>>>>>>> an
>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
>>>>> behind
>>>>>>>>> the
>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware
>> of
>>>>>> this
>>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
>>>>>>>>> infinite
>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
>> implies
>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>> retention, we also never documented (or failed) the
>> empty
>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
>> behavior
>>>>>> has
>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>> existed
>>>>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
>> introduced?),
>>>>>> there
>>>>>>>>>>>>> could
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
>> time.
>>>>> It
>>>>>>>>> is
>>>>>>>>>>>> not
>>>>>>>>>>>>>>> good
>>>>>>>>>>>>>>>>>>> to break the backward compatibility in a minor release
>>>>>> version
>>>>>>>>>>>> (ex:
>>>>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
>>> existing
>>>>>>>>> bug,
>>>>>>>>>>>> it
>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>> be a surprise to users.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>> cleanup.policy
>>>>> by
>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>>>>> realize
>>>>>>>>> this
>>>>>>>>>>>> if
>>>>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x
>> and
>>>>> in
>>>>>>>>> 5.0
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>> exception directly to make it fail fast
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This suggestion looks good to me!
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is
>>>>> worth
>>>>>>>>>>>>>>> considering
>>>>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
>>>>>> encountered
>>>>>>>>>>>> this
>>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
>>>>>> configurations
>>>>>>>>>>>>>>> instead.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>> cleanup.policy
>>>>>> by
>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>>>>>> realize
>>>>>>>>>>>> this
>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense
>> for
>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered
>>>>> storage
>>>>>>>>> use
>>>>>>>>>>>> a
>>>>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>>>>> delete policy.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
>> documented
>>>>>> that
>>>>>>>>>> an
>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
>>>>> one
>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
>> storage
>>>>>> since
>>>>>>>>>> it
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
>> too.
>>>>>> So,
>>>>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
>>> mistake
>>>>>>>>> than
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
>>>>>> behind
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware
>>> of
>>>>>>>>> this
>>>>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
>> wanting
>>>>>>>>>> infinite
>>>>>>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> We could also consider supporting an empty
>>> cleanup.policy
>>>>>> by
>>>>>>>>>>>>> fixing
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>>>>>> realize
>>>>>>>>>>>> this
>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
>>>>>> showuon@gmail.com
>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Comments:
>>>>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy
>> is
>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
>>>>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
>>> trade-off
>>>>>>>>>>>>> justified?
>>>>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in
>>>>> kafka,
>>>>>>>>> why
>>>>>>>>>>>> do
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
>>>>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
>>>>>>>>> history
>>>>>>>>>>>>> data
>>>>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>>>>> a cleanup policy.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for
>> empty
>>>>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
>>>>>> "process.roles"
>>>>>>>>>>>>> configs
>>>>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>>>>> they are invalid.
>>>>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive
>>>>>> reason
>>>>>>>>>> why
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
>>>>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to configure
>>> Kafka
>>>>>> to
>>>>>>>>>>>>> retain
>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
>>>>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but
>> to
>>>>> me,
>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a clear
>> and
>>>>>>>>> direct
>>>>>>>>>>>> way
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>> kafka".
>>>>>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy"
>>> config?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
>>>>>>>>>>>> <jun@confluent.io.invalid
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
>>> s7133700@gmail.com
>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
>>>>>> addressed
>>>>>>>>> in
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
>>>>>>>>> Additionally,
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
>>>>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
>>>>>>>>>> inNonEmpty
>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日
>>>>>> 凌晨12:46
>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass
>> empty
>>>>>>>>>> config,
>>>>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
>>>>>>>>>> directly.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
>>>>>>>>>>>> 上午12:28寫道：
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references
>> to
>>>>>> the
>>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed
>> in
>>>>>> Kafka
>>>>>>>>>>>> 5.0"
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
>>>>>>>>> addressed
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>> 2025年5月3日
>>>>>>>>> 凌晨2:03
>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
>> Migration
>>>>>>>>> Plan",
>>>>>>>>>>>>>>>>>>>> could you
>>>>>>>>>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
>>> "clean.policy="
>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>> replaced
>>>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable
>>>>>> delete
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> compact.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日
>>> 週六
>>>>>>>>>>>>>>>>>>> 上午1:55寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for
>> now.
>>>>>>>>> Could
>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
>>>>>>>>> deprecation.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
>>>>> ValidList#in
>>>>>>>>>>>>>>>>>>> method,
>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
>> useful
>>>>> to
>>>>>>>>> have
>>>>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>>>>> methods:
>>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty
>>> lists).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
>>>>>>>>> document
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
>>>>> fail
>>>>>> to
>>>>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>> 2025年5月2日
>>>>>>>>>>>> 凌晨12:50
>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
>>>>>>>>> document
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
>>>>> fail
>>>>>> to
>>>>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
>>>>>>>>> maintain
>>>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
>> isEmptyAllowed
>>>>>> flag
>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty()
>>> to
>>>>>>>>>> remove
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> boolean
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>> 2025年5月1日
>>>>>>>>>>>> 凌晨4:17
>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's
>>>>> awkward
>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's
>>>>> set
>>>>>> to
>>>>>>>>>>>>>>>>>>> true.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
>>> introduce
>>>>>> new
>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we
>>>>> now
>>>>>>>>>>>>>>>>>>> explicitly
>>>>>>>>>>>>>>>>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
>> format
>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>> phase
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
>>> behavior.”
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>>> 2025年4月25日
>>>>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
>> introduced
>>>>> by
>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown
>> during
>>>>>> the
>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
>> startup."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the
>>>>> storage
>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>> and
>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to
>>> the
>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty
>>>>> list
>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during
>>> storage
>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>>>> 2025年4月24日
>>>>>>>>>>>>>>>>>>>> 凌晨4:44
>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right
>> now.
>>>>>> With
>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
>>>>>> accurate)
>>>>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of
>>> agree
>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> Chia-Ping.
>>>>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely
>>> to
>>>>>> be
>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> mistake
>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None,
>>> the
>>>>>>>>> user
>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will
>> let
>>>>> the
>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
>>> Chia-Ping
>>>>>>>>> Tsai
>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand
>>>>> that
>>>>>>>>>>>>>>>>>>>> maintaining
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
>>> documentation,
>>>>>> we
>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
>>>>>>>>> consider
>>>>>>>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This
>> could
>>>>>>>>>>>> streamline
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>>>>>>>>>>>> 2025年4月23日
>>>>>>>>>>>>>>>>>>> 週三
>>>>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not
>>>>>> delete
>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> compact
>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't
>>>>> delete
>>>>>>>>>>>>>>>>>>> segments
>>>>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
>> log.retention.ms,
>>>>>> but
>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
>>>>>>>>> log.local.retention.bytes/
>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space
>>>>> when
>>>>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun
>>>>> Rao <
>>>>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
>> values
>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>> group.coordinator.rebalance.protocols
>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM
>> 黃竣陽 <
>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
>>>>>> accordingly
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> revised
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy
>>>>> cannot
>>>>>> be
>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
>>>>>>>>>>>>>>>>>>>> cleanup.policy,
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig
>>>>> step.
>>>>>>>>>>>>>>>>>>>>>> Additionally, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <jun@confluent.io.INVALID
>>> 
>>>>> 於
>>>>>>>>>>>>>>>>>>> 2025年4月22日
>>>>>>>>>>>>>>>>>>>>>>> 凌晨4:52
>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
>> comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new
>>>>> value
>>>>>>>>> None
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid.
>>>>> For
>>>>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be
>>>>> useful
>>>>>> to
>>>>>>>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during
>>>>> upgrade
>>>>>>>>> when
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not
>> clear
>>>>>> what
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM
>> 黃竣陽
>>>>> <
>>>>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
>> discussion
>>>>> on
>>>>>>>>>>>>>>>>>>> KIP-1161:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve
>> the
>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left
>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
>> proposed
>>>>>>>>>>>>>>>>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate
>>>>>> whether
>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value
>>> for
>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

> JR20. controller.listener.names : It's required on the broker. Is it
required on the controller?

there is a small "sugar" in the controller, which we will
create listener.security.protocol.map based on controller.listener.names
if listener.security.protocol.map is not explicitly set [0]
Additionally, `controller.listener.names` is also used to filter out the
advertised controller listeners [1]


> JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
null, right?

I don't think so. There is no null check after calling
`getList(GroupCoordinatorConfig.GROUP_COORDINATOR_REBALANCE_PROTOCOLS_CONFIG)`[2],
which causes an NPE if getList returns null."

[0]
https://github.com/apache/kafka/blob/trunk/server/src/main/java/org/apache/kafka/server/config/AbstractKafkaConfig.java#L133
[1]
https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L459
[2]
https://github.com/apache/kafka/blob/7ea32a0e938c22119f11908aa419aaf0ffd9b6d8/core/src/main/scala/kafka/server/KafkaConfig.scala#L373

Jun Rao <ju...@confluent.io.invalid> 於 2025年7月15日 週二 上午6:20寫道：

> Hi, Jiunn-Yang,
>
> Thanks for the updated KIP.
>
> JR20. controller.listener.names : It's required on the broker. Is it
> required on the controller?
>
> JR21. bootstrap.servers: Some of them (e.g. in producer/consumer) already
> require it to be non-null.
>
> JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
> null, right?
>
> JR23. early.start.listeners: It allows default value of null, which means
> that all listeners included in controller.listener.names will also be early
> start listeners.
>
> JR24. log.dirs: It currently has a default value of null.
>
> JR25. It seems there are quite a few more configs of type list. Do they
> need to be changed too?
>
> AdminClientConfig: 4
> ConsumerConfig: 5
> ProducerConfig: 4
> SaslConfigs: 1
> SslConfigs: 2
> BrokerSecurityConfigs: 5
> MirrorClientConfig: 1
> org.apache.kafka.connect.mirror : 16
> org.apache.kafka.connect.runtime : 21
> GroupCoordinatorConfig: 3
> QuorumConfig: 2
> ServerConfigs: 1
> KRaftConfigs: 1
> ClientMetricsConfigs: 2
> MetricConfigs: 2
> LogConfig: 4
> StreamsConfig: 4
>
> Jun
>
> On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hi Chia, Jun
> >
> > Thanks for the reply,
> > I have updated the table in the KIP.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
> > >
> > > Hi, Jun
> > >
> > > Thanks for the reply.
> > >
> > > I think the following configs could use null as the default value and
> > treat
> > > an "empty list" as an illegal value.
> > >
> > > `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
> > > `ssl.enabled.protocols`,
> > > and `log.dirs`
> > >
> > > Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
> > >
> > >> Hi, Chia-Ping, Jiunn-Yang,
> > >>
> > >> Thanks for the reply.
> > >>
> > >> As you said, if ssl.cipher.suites is empty, intuitively this means
> that
> > no
> > >> cipher is allowed. So, this should be an illegal config. It just
> happens
> > >> that our implementation treats it as if the config is not set (meaning
> > the
> > >> value is null). So, it seems that a more intuitive convention is: if
> the
> > >> config is not set, it defaults to all available cipher suites. If it's
> > >> explicitly set, it can't be empty.
> > >>
> > >> Jun
> > >>
> > >>
> > >>
> > >> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>
> > >>> Hi Jun, Chia
> > >>>
> > >>>>> Also, it would be useful to add a compatibility column in the
> table.
> > >> If
> > >>> a
> > >>>>> value is formally disallowed, it would be useful to compare the old
> > >> and
> > >>> the
> > >>>>> new behavior (e.g. a different exception is thrown, etc).
> > >>> I’ve updated the table to show which exceptions will be thrown for
> each
> > >>> behavior.
> > >>>
> > >>>>>> I don't understand the point of "5.0". Could you please share more
> > >>> details?
> > >>>>>> It seems to me changing the type or default value should be fine
> in
> > >>> minor
> > >>>>>> release if we don't break the existing property files.
> > >>> I don't think this requires any compatibility considerations, so I’ve
> > >>> removed the Kafka 5.0 changes section.
> > >>>
> > >>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
> using
> > >>> `LIST`
> > >>>>>> instead of `String` does not seems to make sense.
> > >>> I have removed the config from KIP
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> > >>>>
> > >>>>> For ssl.cipher.suites, changing the default value from null to
> empty
> > >> is
> > >>>>> actually a breaking change. According to the doc, null means that
> all
> > >>> the
> > >>>>> available cipher suites are supported. I am thinking that we should
> > >>>>> continue to allow null as the default since it could represent a
> > state
> > >>>>> different from empty.
> > >>>>
> > >>>> "Empty list" seems to be a weird case. It appears to signify that no
> > >>> cipher suites are accepted. However, in the codebase, an empty list
> is
> > >>> handled as if all available cipher suites are supported [0]. We
> should
> > >> not
> > >>> support the case of "no supported cipher suite," as it doesn't make
> > >> sense.
> > >>>>
> > >>>> In short, changing the default value from null to empty does not
> break
> > >>> the behavior.
> > >>>>
> > >>>> [0]
> > >>>
> > >>
> >
> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> > >>>>
> > >>>> On 2025/07/08 17:01:07 Jun Rao wrote:
> > >>>>> Hi, Jiunn-Yang,
> > >>>>>
> > >>>>> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we
> > >> will
> > >>>>> just continue to allow it in all future releases.
> > >>>>>
> > >>>>> For ssl.cipher.suites, changing the default value from null to
> empty
> > >> is
> > >>>>> actually a breaking change. According to the doc, null means that
> all
> > >>> the
> > >>>>> available cipher suites are supported. I am thinking that we should
> > >>>>> continue to allow null as the default since it could represent a
> > state
> > >>>>> different from empty.
> > >>>>>
> > >>>>> Also, it would be useful to add a compatibility column in the
> table.
> > >> If
> > >>> a
> > >>>>> value is formally disallowed, it would be useful to compare the old
> > >> and
> > >>> the
> > >>>>> new behavior (e.g. a different exception is thrown, etc).
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Jun
> > >>>>>
> > >>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com>
> > >>> wrote:
> > >>>>>
> > >>>>>> hi Jiunn
> > >>>>>>
> > >>>>>> Thanks for the explanation. I have updated the KIP accordingly: in
> > >>> Kafka
> > >>>>>>> 4.x, cleanup.policy
> > >>>>>>> can be set to an empty list with a warning message emitted;
> > however,
> > >>> in
> > >>>>>>> Kafka 5.0, setting an
> > >>>>>>> empty list for cleanup.policy will no longer be allowed.
> > >>>>>>
> > >>>>>>
> > >>>>>> If we agree to support "empty list", the approach of "none" could
> be
> > >>>>>> eliminated, right?
> > >>>>>>
> > >>>>>> I don't understand the point of "5.0". Could you please share more
> > >>> details?
> > >>>>>> It seems to me changing the type or default value should be fine
> in
> > >>> minor
> > >>>>>> release if we don't break the existing property files.
> > >>>>>>
> > >>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so
> using
> > >>> `LIST`
> > >>>>>> instead of `String` does not seems to make sense.
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Chia-Ping
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> > >>>>>>
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>>>> Null could make sense as the default value, but it just means
> > that
> > >>> the
> > >>>>>>>>> config key is not set explicitly.
> > >>>>>>>>> Given that, it's probably simpler to just allow
> > >>>>>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>>>>> Thanks for the explanation. I have updated the KIP accordingly:
> in
> > >>> Kafka
> > >>>>>>> 4.x, cleanup.policy
> > >>>>>>> can be set to an empty list with a warning message emitted;
> > however,
> > >>> in
> > >>>>>>> Kafka 5.0, setting an
> > >>>>>>> empty list for cleanup.policy will no longer be allowed.
> > >>>>>>>
> > >>>>>>>> For another, there is an additional issue which could be handled
> > by
> > >>>>>> this
> > >>>>>>>> KIP. Some configs have "string" type, but they are handled as a
> > >> LIST.
> > >>>>>> For
> > >>>>>>>> example, `early.start.listeners`, `security.providers`. Could we
> > >>> change
> > >>>>>>>> their type to LIST to benefit from this KIP? I mean to make them
> > >>>>>>>> non-nullable.
> > >>>>>>> I have also updated the KIP to include the tables detailing the
> > >>> changes
> > >>>>>>> for LIST-type configuration
> > >>>>>>> default values and the migration from STRING-type to LIST-type
> > >>>>>>> configurations.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>>>
> > >>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> > >>>>>>>>
> > >>>>>>>> hi all,
> > >>>>>>>>
> > >>>>>>>> JR11. There is an existing config interceptor.classes that
> allows
> > >> an
> > >>>>>>> empty
> > >>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
> > >>>>>> support
> > >>>>>>>>> empty lists in general. Given that, it's probably simpler to
> just
> > >>>>>> allow
> > >>>>>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> yes, that is what I meant before. Additionally, the `LIST` type
> > >>> should
> > >>>>>>> not
> > >>>>>>>> accept "null". Instead, we should use an empty list as the
> default
> > >>>>>> value.
> > >>>>>>>> For example, `sasl.oauthbearer.expected.audience` should never
> has
> > >>>>>> "null"
> > >>>>>>>> value. That could eliminate the null check.
> > >>>>>>>>
> > >>>>>>>> For another, there is an additional issue which could be handled
> > by
> > >>>>>> this
> > >>>>>>>> KIP. Some configs have "string" type, but they are handled as a
> > >> LIST.
> > >>>>>> For
> > >>>>>>>> example, `early.start.listeners`, `security.providers`. Could we
> > >>> change
> > >>>>>>>> their type to LIST to benefit from this KIP? I mean to make them
> > >>>>>>>> non-nullable.
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> Chia-Ping
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> > >>>>>>>>
> > >>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for the reply.
> > >>>>>>>>>
> > >>>>>>>>> JR10. Regarding supporting the null value, I am not sure how
> one
> > >>> could
> > >>>>>>> set
> > >>>>>>>>> a null value in a config file. For example, if you set the
> > >> following
> > >>>>>> in
> > >>>>>>> a
> > >>>>>>>>> config file.
> > >>>>>>>>> configkey=
> > >>>>>>>>> The value of configkey will be an empty list, instead of null,
> > >>> right?
> > >>>>>>>>>
> > >>>>>>>>> Null could make sense as the default value, but it just means
> > that
> > >>> the
> > >>>>>>>>> config key is not set explicitly.
> > >>>>>>>>>
> > >>>>>>>>> JR11. There is an existing config interceptor.classes that
> allows
> > >> an
> > >>>>>>> empty
> > >>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
> > >>>>>> support
> > >>>>>>>>> empty lists in general. Given that, it's probably simpler to
> just
> > >>>>>> allow
> > >>>>>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>>>>>>>
> > >>>>>>>>> Jun
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hello Jun,
> > >>>>>>>>>>
> > >>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional, that
> > is
> > >>>>>>> right,
> > >>>>>>>>>> so I think
> > >>>>>>>>>> that we can allow null value, but not allow the empty
> > >>>>>>>>>>
> > >>>>>>>>>> JR11: In this KIP, the goal is to address configurations that
> > >>> should
> > >>>>>>> not
> > >>>>>>>>>> accept empty arrays.
> > >>>>>>>>>> We want to prevent users from passing in an empty list when
> it's
> > >>> not
> > >>>>>>>>>> valid. I believe there are two
> > >>>>>>>>>> main scenarios to consider:
> > >>>>>>>>>>
> > >>>>>>>>>> 1. Nullable but not empty:
> > >>>>>>>>>> The parameter allows either a null value or a non-empty list.
> In
> > >>> this
> > >>>>>>>>>> case,
> > >>>>>>>>>> if the user provides an empty list, we should throw a
> > >>>>>> ConfigException.
> > >>>>>>>>>>
> > >>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
> > >> contain a
> > >>>>>>>>>> non-empty list — null is also
> > >>>>>>>>>> invalid. In this case, if the user provides either a null or
> an
> > >>> empty
> > >>>>>>>>>> list, we should throw a ConfigException.
> > >>>>>>>>>>
> > >>>>>>>>>> Therefore, I want to prevent scenarios where users can pass in
> > an
> > >>>>>> empty
> > >>>>>>>>>> list.
> > >>>>>>>>>>
> > >>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an
> allowNull
> > >>>>>>>>> property.
> > >>>>>>>>>> This allows us to specify
> > >>>>>>>>>> whether a configuration permits null values or not.
> > >>>>>>>>>>
> > >>>>>>>>>> JR13: I have updated the table
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Sorry for the late reply. A few more comments.
> > >>>>>>>>>>>
> > >>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says
> that
> > >>> it's
> > >>>>>>>>>>> optional. So an empty list seems to be valid.
> > >>>>>>>>>>>
> > >>>>>>>>>>> JR11. Thinking a bit more. If we are going to support empty
> > >> lists
> > >>> in
> > >>>>>>>>>>> general, it's probably more consistent to just allow
> > >>>>>>> log.cleanup.policy
> > >>>>>>>>>> to
> > >>>>>>>>>>> be empty as Luke suggested, instead of introducing a new
> option
> > >>>>>> None.
> > >>>>>>>>>>>
> > >>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It
> > >> seems
> > >>>>>> that
> > >>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
> > shouldn't
> > >>> be
> > >>>>>>>>> null.
> > >>>>>>>>>>>
> > >>>>>>>>>>> JR13. The table is very helpful. Could you also describe the
> > >>> old/new
> > >>>>>>>>>>> behavior for each config with an empty list or duplicate
> entry?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Jun
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I am manually bumping this thread. Any feedback would be
> > >>>>>> appreciated.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
> > >>>>>>>>>>>> NonEmptyListValidator, which ensures that the
> > >>>>>>>>>>>>> provided list is either null or a non-empty list without
> > >>> duplicate
> > >>>>>>>>>>>> entries. I’ve also listed the configurations that will
> > >>>>>>>>>>>>> be validated using this logic. Please feel free to share
> any
> > >>>>>>> feedback
> > >>>>>>>>>> or
> > >>>>>>>>>>>> suggestions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29
> 寫道：
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> It seems there are quite a few other configs of type list
> > >> (e.g.
> > >>>>>>>>>> several
> > >>>>>>>>>>>> SSL
> > >>>>>>>>>>>>>> related ones). It would be useful to understand if empty
> > >> lists
> > >>>>>> are
> > >>>>>>>>>> valid
> > >>>>>>>>>>>>>> for them or not.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <
> jun@confluent.io>
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi, Luke,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> "My thought is that this behavior has been existed
> > >>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
> > >>> there
> > >>>>>>>>> could
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
> time.
> > "
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> If you do a google search on "kafka infinite retention",
> > you
> > >>>>>> will
> > >>>>>>>>> get
> > >>>>>>>>>>>>>>> links like
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> > >>>>>>>>>>>> ,
> > >>>>>>>>>>>>>>> both pointing to setting the retention time and retention
> > >> size
> > >>>>>> to
> > >>>>>>>>> -1
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
> > intentionally
> > >>>>>> wants
> > >>>>>>>>>>>> infinite
> > >>>>>>>>>>>>>>> retention, it's more likely they will use that approach
> > >>> instead
> > >>>>>> of
> > >>>>>>>>>>>> setting
> > >>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our API/tools
> > >> make
> > >>>>>> it
> > >>>>>>>>>>>> possible
> > >>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving the
> > >>> config
> > >>>>>>>>> value
> > >>>>>>>>>>>> empty.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
> > >> showuon@gmail.com
> > >>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> documented
> > >>> that
> > >>>>>> an
> > >>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
> one
> > >>>>>> leaves
> > >>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
> > >> since
> > >>>>>> it
> > >>>>>>>>>> will
> > >>>>>>>>>>>> stop
> > >>>>>>>>>>>>>>>> the cleaning based on local retention size and time too.
> > >> So,
> > >>>>>>>>> leaving
> > >>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> mistake
> > >>> than
> > >>>>>> an
> > >>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
> > >> behind
> > >>>>>> the
> > >>>>>>>>>> KIP
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware
> of
> > >>> this
> > >>>>>>>>>> likely
> > >>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
> > >>>>>> infinite
> > >>>>>>>>>>>> retention
> > >>>>>>>>>>>>>>>> to opt in explicitly.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy
> implies
> > >>>>>>>>> infinite
> > >>>>>>>>>>>>>>>> retention, we also never documented (or failed) the
> empty
> > >>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>> an invalid configuration. My thought is that this
> behavior
> > >>> has
> > >>>>>>>>> been
> > >>>>>>>>>>>>>>>> existed
> > >>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was
> introduced?),
> > >>> there
> > >>>>>>>>>> could
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long
> time.
> > >> It
> > >>>>>> is
> > >>>>>>>>> not
> > >>>>>>>>>>>> good
> > >>>>>>>>>>>>>>>> to break the backward compatibility in a minor release
> > >>> version
> > >>>>>>>>> (ex:
> > >>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
> > existing
> > >>>>>> bug,
> > >>>>>>>>> it
> > >>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>> be a surprise to users.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> We could also consider supporting an empty
> cleanup.policy
> > >> by
> > >>>>>>>>> fixing
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> > >> realize
> > >>>>>> this
> > >>>>>>>>> if
> > >>>>>>>>>>>> it's
> > >>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Chia-Ping,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x
> and
> > >> in
> > >>>>>> 5.0
> > >>>>>>>>> we
> > >>>>>>>>>>>>>>>> throw
> > >>>>>>>>>>>>>>>> exception directly to make it fail fast
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This suggestion looks good to me!
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thank you.
> > >>>>>>>>>>>>>>>> Luke
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> > >>>>>>>>> chia7712@apache.org
> > >>>>>>>>>>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> hi all,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is
> > >> worth
> > >>>>>>>>>>>> considering
> > >>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
> > >>> encountered
> > >>>>>>>>> this
> > >>>>>>>>>>>> use
> > >>>>>>>>>>>>>>>>> case but advised customers to utilize retention
> > >>> configurations
> > >>>>>>>>>>>> instead.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> > cleanup.policy
> > >>> by
> > >>>>>>>>>> fixing
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> > >>> realize
> > >>>>>>>>> this
> > >>>>>>>>>> if
> > >>>>>>>>>>>>>>>>> it's a
> > >>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense
> for
> > >>>>>> remote
> > >>>>>>>>>>>>>>>> storage.
> > >>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered
> > >> storage
> > >>>>>> use
> > >>>>>>>>> a
> > >>>>>>>>>>>> valid
> > >>>>>>>>>>>>>>>>> delete policy.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> > >>>>>>>>>>>>>>>>>> Hi, Luke,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never
> documented
> > >>> that
> > >>>>>>> an
> > >>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
> > >> one
> > >>>>>>>>> leaves
> > >>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote
> storage
> > >>> since
> > >>>>>>> it
> > >>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>> stop
> > >>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time
> too.
> > >>> So,
> > >>>>>>>>>> leaving
> > >>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> > mistake
> > >>>>>> than
> > >>>>>>>>> an
> > >>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
> > >>> behind
> > >>>>>>>>> the
> > >>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware
> > of
> > >>>>>> this
> > >>>>>>>>>>>> likely
> > >>>>>>>>>>>>>>>>>> mistake and provide a new option None for users
> wanting
> > >>>>>>> infinite
> > >>>>>>>>>>>>>>>>> retention
> > >>>>>>>>>>>>>>>>>> to opt in explicitly.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> > cleanup.policy
> > >>> by
> > >>>>>>>>>> fixing
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> > >>> realize
> > >>>>>>>>> this
> > >>>>>>>>>> if
> > >>>>>>>>>>>>>>>>> it's a
> > >>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
> > >>> showuon@gmail.com
> > >>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Comments:
> > >>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy
> is
> > >>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> > >>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
> > trade-off
> > >>>>>>>>>> justified?
> > >>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in
> > >> kafka,
> > >>>>>> why
> > >>>>>>>>> do
> > >>>>>>>>>> we
> > >>>>>>>>>>>>>>>>> provide
> > >>>>>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
> > >>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
> > >>>>>> history
> > >>>>>>>>>> data
> > >>>>>>>>>>>>>>>>> without
> > >>>>>>>>>>>>>>>>>>> a cleanup policy.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for
> empty
> > >>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
> > >>> "process.roles"
> > >>>>>>>>>> configs
> > >>>>>>>>>>>>>>>>> since
> > >>>>>>>>>>>>>>>>>>> they are invalid.
> > >>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive
> > >>> reason
> > >>>>>>> why
> > >>>>>>>>>> we
> > >>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
> > >>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to configure
> > Kafka
> > >>> to
> > >>>>>>>>>> retain
> > >>>>>>>>>>>>>>>> all
> > >>>>>>>>>>>>>>>>> log
> > >>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
> > >>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but
> to
> > >> me,
> > >>>>>>>>>> backward
> > >>>>>>>>>>>>>>>>>>> compatibility is much more important than "a clear
> and
> > >>>>>> direct
> > >>>>>>>>> way
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> config
> > >>>>>>>>>>>>>>>>>>> kafka".
> > >>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy"
> > config?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks.
> > >>>>>>>>>>>>>>>>>>> Luke
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> > >>>>>>>>> <jun@confluent.io.invalid
> > >>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
> > s7133700@gmail.com
> > >>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> > >>> addressed
> > >>>>>> in
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
> > >>>>>> Additionally,
> > >>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>> will now be thrown if a
> > >>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
> > >>>>>>> inNonEmpty
> > >>>>>>>>>>>>>>>>> method.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日
> > >>> 凌晨12:46
> > >>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass
> empty
> > >>>>>>> config,
> > >>>>>>>>>>>>>>>> so
> > >>>>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>> be non-empty too?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
> > >>>>>>> directly.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
> > >>>>>>>>> 上午12:28寫道：
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references
> to
> > >>> the
> > >>>>>>>>>>>>>>>>> deprecated
> > >>>>>>>>>>>>>>>>>>>>> method.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> > >>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed
> in
> > >>> Kafka
> > >>>>>>>>> 5.0"
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
> > >>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> > >>>>>> addressed
> > >>>>>>>>> in
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> 2025年5月3日
> > >>>>>> 凌晨2:03
> > >>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and
> Migration
> > >>>>>> Plan",
> > >>>>>>>>>>>>>>>>> could you
> > >>>>>>>>>>>>>>>>>>>> add
> > >>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
> > "clean.policy="
> > >>>>>>> should
> > >>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>> replaced
> > >>>>>>>>>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable
> > >>> delete
> > >>>>>>> and
> > >>>>>>>>>>>>>>>>> compact.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日
> > 週六
> > >>>>>>>>>>>>>>>> 上午1:55寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for
> now.
> > >>>>>> Could
> > >>>>>>>>> you
> > >>>>>>>>>>>>>>>>> update
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
> > >>>>>> deprecation.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> > >>>>>> s7133700@gmail.com
> > >>>>>>>>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
> > >> ValidList#in
> > >>>>>>>>>>>>>>>> method,
> > >>>>>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s
> useful
> > >> to
> > >>>>>> have
> > >>>>>>>>>>>>>>>> both
> > >>>>>>>>>>>>>>>>>>>> methods:
> > >>>>>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty
> > lists).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> > >>>>>> document
> > >>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
> > >> fail
> > >>> to
> > >>>>>>>>>>>>>>>> start.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > 2025年5月2日
> > >>>>>>>>> 凌晨12:50
> > >>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> > >>>>>> document
> > >>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
> > >> fail
> > >>> to
> > >>>>>>>>>>>>>>>> start.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> > >>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
> > >>>>>> maintain
> > >>>>>>>>>>>>>>>>> backward
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the
> isEmptyAllowed
> > >>> flag
> > >>>>>> is
> > >>>>>>>>>>>>>>>>>>> necessary.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty()
> > to
> > >>>>>>> remove
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> boolean
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >> 2025年5月1日
> > >>>>>>>>> 凌晨4:17
> > >>>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's
> > >> awkward
> > >>>>>>> since
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's
> > >> set
> > >>> to
> > >>>>>>>>>>>>>>>> true.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> > >>>>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
> > introduce
> > >>> new
> > >>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we
> > >> now
> > >>>>>>>>>>>>>>>> explicitly
> > >>>>>>>>>>>>>>>>>>>> throw a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage
> format
> > >>>>>>>>> validation
> > >>>>>>>>>>>>>>>>> phase
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
> > behavior.”
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >>>>>> 2025年4月25日
> > >>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference
> introduced
> > >> by
> > >>>>>>> this
> > >>>>>>>>>>>>>>>>> change
> > >>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown
> during
> > >>> the
> > >>>>>>>>>>>>>>>> storage
> > >>>>>>>>>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server
> startup."
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the
> > >> storage
> > >>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>> fails
> > >>>>>>>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> and
> > >>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>>>>>>>> empty.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> > >>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> > >>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to
> > the
> > >>>>>>>>> storage
> > >>>>>>>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty
> > >> list
> > >>>>>> will
> > >>>>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during
> > storage
> > >>>>>>> format
> > >>>>>>>>>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >>>>>>> 2025年4月24日
> > >>>>>>>>>>>>>>>>> 凌晨4:44
> > >>>>>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> > >>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> > >>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right
> now.
> > >>> With
> > >>>>>>>>> this
> > >>>>>>>>>>>>>>>>> KIP,
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
> > >>> accurate)
> > >>>>>>>>>>>>>>>>> exception.
> > >>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of
> > agree
> > >>>>>> with
> > >>>>>>>>>>>>>>>>> Chia-Ping.
> > >>>>>>>>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely
> > to
> > >>> be
> > >>>>>> a
> > >>>>>>>>>>>>>>>>> mistake
> > >>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None,
> > the
> > >>>>>> user
> > >>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>> never
> > >>>>>>>>>>>>>>>>>>>>>>>> realize
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will
> let
> > >> the
> > >>>>>>> user
> > >>>>>>>>>>>>>>>>> realize
> > >>>>>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
> > Chia-Ping
> > >>>>>> Tsai
> > >>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand
> > >> that
> > >>>>>>>>>>>>>>>>> maintaining
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
> > documentation,
> > >>> we
> > >>>>>>>>> don't
> > >>>>>>>>>>>>>>>>> allow
> > >>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
> > >>>>>> consider
> > >>>>>>>>>>>>>>>>> throwing
> > >>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This
> could
> > >>>>>>>>> streamline
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> code
> > >>>>>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> > >>>>>>>>> 2025年4月23日
> > >>>>>>>>>>>>>>>> 週三
> > >>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not
> > >>> delete
> > >>>>>> or
> > >>>>>>>>>>>>>>>>> compact
> > >>>>>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't
> > >> delete
> > >>>>>>>>>>>>>>>> segments
> > >>>>>>>>>>>>>>>>> based
> > >>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/
> log.retention.ms,
> > >>> but
> > >>>>>> we
> > >>>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>>>>> continue
> > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> > >>>>>> log.local.retention.bytes/
> > >>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space
> > >> when
> > >>>>>>>>> remote
> > >>>>>>>>>>>>>>>>>>> storage
> > >>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun
> > >> Rao <
> > >>>>>>>>>>>>>>>>>>>> jun@confluent.io>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty
> values
> > >> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> group.coordinator.rebalance.protocols
> > >>> and
> > >>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM
> 黃竣陽 <
> > >>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
> > >>> accordingly
> > >>>>>>> and
> > >>>>>>>>>>>>>>>>> revised
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy
> > >> cannot
> > >>> be
> > >>>>>>>>> used
> > >>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> > >>>>>>>>>>>>>>>>> cleanup.policy,
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig
> > >> step.
> > >>>>>>>>>>>>>>>>>>> Additionally, a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> warning
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <jun@confluent.io.INVALID
> >
> > >> 於
> > >>>>>>>>>>>>>>>> 2025年4月22日
> > >>>>>>>>>>>>>>>>>>>> 凌晨4:52
> > >>>>>>>>>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few
> comments.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new
> > >> value
> > >>>>>> None
> > >>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid.
> > >> For
> > >>>>>>>>>>>>>>>> example,
> > >>>>>>>>>>>>>>>>> None
> > >>>>>>>>>>>>>>>>>>>>> can't
> > >>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be
> > >> useful
> > >>> to
> > >>>>>>>>>>>>>>>>> document
> > >>>>>>>>>>>>>>>>>>>> that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during
> > >> upgrade
> > >>>>>> when
> > >>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>> existing
> > >>>>>>>>>>>>>>>>>>>>>>>>>> config
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not
> clear
> > >>> what
> > >>>>>>>>> the
> > >>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>> check
> > >>>>>>>>>>>>>>>>>>>>>>>>>> does.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM
> 黃竣陽
> > >> <
> > >>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> discussion
> > >> on
> > >>>>>>>>>>>>>>>> KIP-1161:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve
> the
> > >>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left
> > >>> empty.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two
> proposed
> > >>>>>>>>>>>>>>>> improvements:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate
> > >>> whether
> > >>>>>>> an
> > >>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>> list
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value
> > for
> > >>>>>>>>>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the updated KIP.

JR20. controller.listener.names : It's required on the broker. Is it
required on the controller?

JR21. bootstrap.servers: Some of them (e.g. in producer/consumer) already
require it to be non-null.

JR22. group.coordinator.rebalance.protocols: Currently, it can be set to
null, right?

JR23. early.start.listeners: It allows default value of null, which means
that all listeners included in controller.listener.names will also be early
start listeners.

JR24. log.dirs: It currently has a default value of null.

JR25. It seems there are quite a few more configs of type list. Do they
need to be changed too?

AdminClientConfig: 4
ConsumerConfig: 5
ProducerConfig: 4
SaslConfigs: 1
SslConfigs: 2
BrokerSecurityConfigs: 5
MirrorClientConfig: 1
org.apache.kafka.connect.mirror : 16
org.apache.kafka.connect.runtime : 21
GroupCoordinatorConfig: 3
QuorumConfig: 2
ServerConfigs: 1
KRaftConfigs: 1
ClientMetricsConfigs: 2
MetricConfigs: 2
LogConfig: 4
StreamsConfig: 4

Jun

On Fri, Jul 11, 2025 at 8:13 PM 黃竣陽 <s7...@gmail.com> wrote:

> Hi Chia, Jun
>
> Thanks for the reply,
> I have updated the table in the KIP.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
> >
> > Hi, Jun
> >
> > Thanks for the reply.
> >
> > I think the following configs could use null as the default value and
> treat
> > an "empty list" as an illegal value.
> >
> > `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
> > `ssl.enabled.protocols`,
> > and `log.dirs`
> >
> > Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
> >
> >> Hi, Chia-Ping, Jiunn-Yang,
> >>
> >> Thanks for the reply.
> >>
> >> As you said, if ssl.cipher.suites is empty, intuitively this means that
> no
> >> cipher is allowed. So, this should be an illegal config. It just happens
> >> that our implementation treats it as if the config is not set (meaning
> the
> >> value is null). So, it seems that a more intuitive convention is: if the
> >> config is not set, it defaults to all available cipher suites. If it's
> >> explicitly set, it can't be empty.
> >>
> >> Jun
> >>
> >>
> >>
> >> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hi Jun, Chia
> >>>
> >>>>> Also, it would be useful to add a compatibility column in the table.
> >> If
> >>> a
> >>>>> value is formally disallowed, it would be useful to compare the old
> >> and
> >>> the
> >>>>> new behavior (e.g. a different exception is thrown, etc).
> >>> I’ve updated the table to show which exceptions will be thrown for each
> >>> behavior.
> >>>
> >>>>>> I don't understand the point of "5.0". Could you please share more
> >>> details?
> >>>>>> It seems to me changing the type or default value should be fine in
> >>> minor
> >>>>>> release if we don't break the existing property files.
> >>> I don't think this requires any compatibility considerations, so I’ve
> >>> removed the Kafka 5.0 changes section.
> >>>
> >>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so using
> >>> `LIST`
> >>>>>> instead of `String` does not seems to make sense.
> >>> I have removed the config from KIP
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> >>>>
> >>>>> For ssl.cipher.suites, changing the default value from null to empty
> >> is
> >>>>> actually a breaking change. According to the doc, null means that all
> >>> the
> >>>>> available cipher suites are supported. I am thinking that we should
> >>>>> continue to allow null as the default since it could represent a
> state
> >>>>> different from empty.
> >>>>
> >>>> "Empty list" seems to be a weird case. It appears to signify that no
> >>> cipher suites are accepted. However, in the codebase, an empty list is
> >>> handled as if all available cipher suites are supported [0]. We should
> >> not
> >>> support the case of "no supported cipher suite," as it doesn't make
> >> sense.
> >>>>
> >>>> In short, changing the default value from null to empty does not break
> >>> the behavior.
> >>>>
> >>>> [0]
> >>>
> >>
> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> >>>>
> >>>> On 2025/07/08 17:01:07 Jun Rao wrote:
> >>>>> Hi, Jiunn-Yang,
> >>>>>
> >>>>> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we
> >> will
> >>>>> just continue to allow it in all future releases.
> >>>>>
> >>>>> For ssl.cipher.suites, changing the default value from null to empty
> >> is
> >>>>> actually a breaking change. According to the doc, null means that all
> >>> the
> >>>>> available cipher suites are supported. I am thinking that we should
> >>>>> continue to allow null as the default since it could represent a
> state
> >>>>> different from empty.
> >>>>>
> >>>>> Also, it would be useful to add a compatibility column in the table.
> >> If
> >>> a
> >>>>> value is formally disallowed, it would be useful to compare the old
> >> and
> >>> the
> >>>>> new behavior (e.g. a different exception is thrown, etc).
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> hi Jiunn
> >>>>>>
> >>>>>> Thanks for the explanation. I have updated the KIP accordingly: in
> >>> Kafka
> >>>>>>> 4.x, cleanup.policy
> >>>>>>> can be set to an empty list with a warning message emitted;
> however,
> >>> in
> >>>>>>> Kafka 5.0, setting an
> >>>>>>> empty list for cleanup.policy will no longer be allowed.
> >>>>>>
> >>>>>>
> >>>>>> If we agree to support "empty list", the approach of "none" could be
> >>>>>> eliminated, right?
> >>>>>>
> >>>>>> I don't understand the point of "5.0". Could you please share more
> >>> details?
> >>>>>> It seems to me changing the type or default value should be fine in
> >>> minor
> >>>>>> release if we don't break the existing property files.
> >>>>>>
> >>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so using
> >>> `LIST`
> >>>>>> instead of `String` does not seems to make sense.
> >>>>>>
> >>>>>> Best,
> >>>>>> Chia-Ping
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>>>> Null could make sense as the default value, but it just means
> that
> >>> the
> >>>>>>>>> config key is not set explicitly.
> >>>>>>>>> Given that, it's probably simpler to just allow
> >>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>> Thanks for the explanation. I have updated the KIP accordingly: in
> >>> Kafka
> >>>>>>> 4.x, cleanup.policy
> >>>>>>> can be set to an empty list with a warning message emitted;
> however,
> >>> in
> >>>>>>> Kafka 5.0, setting an
> >>>>>>> empty list for cleanup.policy will no longer be allowed.
> >>>>>>>
> >>>>>>>> For another, there is an additional issue which could be handled
> by
> >>>>>> this
> >>>>>>>> KIP. Some configs have "string" type, but they are handled as a
> >> LIST.
> >>>>>> For
> >>>>>>>> example, `early.start.listeners`, `security.providers`. Could we
> >>> change
> >>>>>>>> their type to LIST to benefit from this KIP? I mean to make them
> >>>>>>>> non-nullable.
> >>>>>>> I have also updated the KIP to include the tables detailing the
> >>> changes
> >>>>>>> for LIST-type configuration
> >>>>>>> default values and the migration from STRING-type to LIST-type
> >>>>>>> configurations.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> >>>>>>>>
> >>>>>>>> hi all,
> >>>>>>>>
> >>>>>>>> JR11. There is an existing config interceptor.classes that allows
> >> an
> >>>>>>> empty
> >>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
> >>>>>> support
> >>>>>>>>> empty lists in general. Given that, it's probably simpler to just
> >>>>>> allow
> >>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> yes, that is what I meant before. Additionally, the `LIST` type
> >>> should
> >>>>>>> not
> >>>>>>>> accept "null". Instead, we should use an empty list as the default
> >>>>>> value.
> >>>>>>>> For example, `sasl.oauthbearer.expected.audience` should never has
> >>>>>> "null"
> >>>>>>>> value. That could eliminate the null check.
> >>>>>>>>
> >>>>>>>> For another, there is an additional issue which could be handled
> by
> >>>>>> this
> >>>>>>>> KIP. Some configs have "string" type, but they are handled as a
> >> LIST.
> >>>>>> For
> >>>>>>>> example, `early.start.listeners`, `security.providers`. Could we
> >>> change
> >>>>>>>> their type to LIST to benefit from this KIP? I mean to make them
> >>>>>>>> non-nullable.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> >>>>>>>>
> >>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>
> >>>>>>>>> Thanks for the reply.
> >>>>>>>>>
> >>>>>>>>> JR10. Regarding supporting the null value, I am not sure how one
> >>> could
> >>>>>>> set
> >>>>>>>>> a null value in a config file. For example, if you set the
> >> following
> >>>>>> in
> >>>>>>> a
> >>>>>>>>> config file.
> >>>>>>>>> configkey=
> >>>>>>>>> The value of configkey will be an empty list, instead of null,
> >>> right?
> >>>>>>>>>
> >>>>>>>>> Null could make sense as the default value, but it just means
> that
> >>> the
> >>>>>>>>> config key is not set explicitly.
> >>>>>>>>>
> >>>>>>>>> JR11. There is an existing config interceptor.classes that allows
> >> an
> >>>>>>> empty
> >>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
> >>>>>> support
> >>>>>>>>> empty lists in general. Given that, it's probably simpler to just
> >>>>>> allow
> >>>>>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>>>>
> >>>>>>>>> Jun
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hello Jun,
> >>>>>>>>>>
> >>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional, that
> is
> >>>>>>> right,
> >>>>>>>>>> so I think
> >>>>>>>>>> that we can allow null value, but not allow the empty
> >>>>>>>>>>
> >>>>>>>>>> JR11: In this KIP, the goal is to address configurations that
> >>> should
> >>>>>>> not
> >>>>>>>>>> accept empty arrays.
> >>>>>>>>>> We want to prevent users from passing in an empty list when it's
> >>> not
> >>>>>>>>>> valid. I believe there are two
> >>>>>>>>>> main scenarios to consider:
> >>>>>>>>>>
> >>>>>>>>>> 1. Nullable but not empty:
> >>>>>>>>>> The parameter allows either a null value or a non-empty list. In
> >>> this
> >>>>>>>>>> case,
> >>>>>>>>>> if the user provides an empty list, we should throw a
> >>>>>> ConfigException.
> >>>>>>>>>>
> >>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
> >> contain a
> >>>>>>>>>> non-empty list — null is also
> >>>>>>>>>> invalid. In this case, if the user provides either a null or an
> >>> empty
> >>>>>>>>>> list, we should throw a ConfigException.
> >>>>>>>>>>
> >>>>>>>>>> Therefore, I want to prevent scenarios where users can pass in
> an
> >>>>>> empty
> >>>>>>>>>> list.
> >>>>>>>>>>
> >>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
> >>>>>>>>> property.
> >>>>>>>>>> This allows us to specify
> >>>>>>>>>> whether a configuration permits null values or not.
> >>>>>>>>>>
> >>>>>>>>>> JR13: I have updated the table
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> >>>>>>>>>>>
> >>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>
> >>>>>>>>>>> Sorry for the late reply. A few more comments.
> >>>>>>>>>>>
> >>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says that
> >>> it's
> >>>>>>>>>>> optional. So an empty list seems to be valid.
> >>>>>>>>>>>
> >>>>>>>>>>> JR11. Thinking a bit more. If we are going to support empty
> >> lists
> >>> in
> >>>>>>>>>>> general, it's probably more consistent to just allow
> >>>>>>> log.cleanup.policy
> >>>>>>>>>> to
> >>>>>>>>>>> be empty as Luke suggested, instead of introducing a new option
> >>>>>> None.
> >>>>>>>>>>>
> >>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It
> >> seems
> >>>>>> that
> >>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones
> shouldn't
> >>> be
> >>>>>>>>> null.
> >>>>>>>>>>>
> >>>>>>>>>>> JR13. The table is very helpful. Could you also describe the
> >>> old/new
> >>>>>>>>>>> behavior for each config with an empty list or duplicate entry?
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am manually bumping this thread. Any feedback would be
> >>>>>> appreciated.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
> >>>>>>>>>>>> NonEmptyListValidator, which ensures that the
> >>>>>>>>>>>>> provided list is either null or a non-empty list without
> >>> duplicate
> >>>>>>>>>>>> entries. I’ve also listed the configurations that will
> >>>>>>>>>>>>> be validated using this logic. Please feel free to share any
> >>>>>>> feedback
> >>>>>>>>>> or
> >>>>>>>>>>>> suggestions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It seems there are quite a few other configs of type list
> >> (e.g.
> >>>>>>>>>> several
> >>>>>>>>>>>> SSL
> >>>>>>>>>>>>>> related ones). It would be useful to understand if empty
> >> lists
> >>>>>> are
> >>>>>>>>>> valid
> >>>>>>>>>>>>>> for them or not.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io>
> >>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> "My thought is that this behavior has been existed
> >>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
> >>> there
> >>>>>>>>> could
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long time.
> "
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If you do a google search on "kafka infinite retention",
> you
> >>>>>> will
> >>>>>>>>> get
> >>>>>>>>>>>>>>> links like
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>
> >>
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>
> >>
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> >>>>>>>>>>>> ,
> >>>>>>>>>>>>>>> both pointing to setting the retention time and retention
> >> size
> >>>>>> to
> >>>>>>>>> -1
> >>>>>>>>>> to
> >>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user
> intentionally
> >>>>>> wants
> >>>>>>>>>>>> infinite
> >>>>>>>>>>>>>>> retention, it's more likely they will use that approach
> >>> instead
> >>>>>> of
> >>>>>>>>>>>> setting
> >>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our API/tools
> >> make
> >>>>>> it
> >>>>>>>>>>>> possible
> >>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving the
> >>> config
> >>>>>>>>> value
> >>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
> >> showuon@gmail.com
> >>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never documented
> >>> that
> >>>>>> an
> >>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> >>>>>> leaves
> >>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
> >> since
> >>>>>> it
> >>>>>>>>>> will
> >>>>>>>>>>>> stop
> >>>>>>>>>>>>>>>> the cleaning based on local retention size and time too.
> >> So,
> >>>>>>>>> leaving
> >>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
> >>> than
> >>>>>> an
> >>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
> >> behind
> >>>>>> the
> >>>>>>>>>> KIP
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
> >>> this
> >>>>>>>>>> likely
> >>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
> >>>>>> infinite
> >>>>>>>>>>>> retention
> >>>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy implies
> >>>>>>>>> infinite
> >>>>>>>>>>>>>>>> retention, we also never documented (or failed) the empty
> >>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> an invalid configuration. My thought is that this behavior
> >>> has
> >>>>>>>>> been
> >>>>>>>>>>>>>>>> existed
> >>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
> >>> there
> >>>>>>>>>> could
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long time.
> >> It
> >>>>>> is
> >>>>>>>>> not
> >>>>>>>>>>>> good
> >>>>>>>>>>>>>>>> to break the backward compatibility in a minor release
> >>> version
> >>>>>>>>> (ex:
> >>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long
> existing
> >>>>>> bug,
> >>>>>>>>> it
> >>>>>>>>>>>> could
> >>>>>>>>>>>>>>>> be a surprise to users.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
> >> by
> >>>>>>>>> fixing
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> >> realize
> >>>>>> this
> >>>>>>>>> if
> >>>>>>>>>>>> it's
> >>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and
> >> in
> >>>>>> 5.0
> >>>>>>>>> we
> >>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>> exception directly to make it fail fast
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This suggestion looks good to me!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you.
> >>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> >>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> hi all,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is
> >> worth
> >>>>>>>>>>>> considering
> >>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
> >>> encountered
> >>>>>>>>> this
> >>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>> case but advised customers to utilize retention
> >>> configurations
> >>>>>>>>>>>> instead.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> cleanup.policy
> >>> by
> >>>>>>>>>> fixing
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> >>> realize
> >>>>>>>>> this
> >>>>>>>>>> if
> >>>>>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for
> >>>>>> remote
> >>>>>>>>>>>>>>>> storage.
> >>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered
> >> storage
> >>>>>> use
> >>>>>>>>> a
> >>>>>>>>>>>> valid
> >>>>>>>>>>>>>>>>> delete policy.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> >>>>>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never documented
> >>> that
> >>>>>>> an
> >>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
> >> one
> >>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
> >>> since
> >>>>>>> it
> >>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> stop
> >>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time too.
> >>> So,
> >>>>>>>>>> leaving
> >>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user
> mistake
> >>>>>> than
> >>>>>>>>> an
> >>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
> >>> behind
> >>>>>>>>> the
> >>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware
> of
> >>>>>> this
> >>>>>>>>>>>> likely
> >>>>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
> >>>>>>> infinite
> >>>>>>>>>>>>>>>>> retention
> >>>>>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We could also consider supporting an empty
> cleanup.policy
> >>> by
> >>>>>>>>>> fixing
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> >>> realize
> >>>>>>>>> this
> >>>>>>>>>> if
> >>>>>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
> >>> showuon@gmail.com
> >>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Comments:
> >>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
> >>>>>>>>>> considered
> >>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> >>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this
> trade-off
> >>>>>>>>>> justified?
> >>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in
> >> kafka,
> >>>>>> why
> >>>>>>>>> do
> >>>>>>>>>> we
> >>>>>>>>>>>>>>>>> provide
> >>>>>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
> >>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
> >>>>>> history
> >>>>>>>>>> data
> >>>>>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>>>>> a cleanup policy.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
> >>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
> >>> "process.roles"
> >>>>>>>>>> configs
> >>>>>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>>>>> they are invalid.
> >>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive
> >>> reason
> >>>>>>> why
> >>>>>>>>>> we
> >>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
> >>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to configure
> Kafka
> >>> to
> >>>>>>>>>> retain
> >>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
> >>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to
> >> me,
> >>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>> compatibility is much more important than "a clear and
> >>>>>> direct
> >>>>>>>>> way
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>> kafka".
> >>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy"
> config?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> >>>>>>>>> <jun@confluent.io.invalid
> >>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <
> s7133700@gmail.com
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> >>> addressed
> >>>>>> in
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
> >>>>>> Additionally,
> >>>>>>>>> an
> >>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>> will now be thrown if a
> >>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
> >>>>>>> inNonEmpty
> >>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日
> >>> 凌晨12:46
> >>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty
> >>>>>>> config,
> >>>>>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>> be non-empty too?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
> >>>>>>> directly.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
> >>>>>>>>> 上午12:28寫道：
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references to
> >>> the
> >>>>>>>>>>>>>>>>> deprecated
> >>>>>>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> >>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in
> >>> Kafka
> >>>>>>>>> 5.0"
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
> >>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> >>>>>> addressed
> >>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日
> >>>>>> 凌晨2:03
> >>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration
> >>>>>> Plan",
> >>>>>>>>>>>>>>>>> could you
> >>>>>>>>>>>>>>>>>>>> add
> >>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that
> "clean.policy="
> >>>>>>> should
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> replaced
> >>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable
> >>> delete
> >>>>>>> and
> >>>>>>>>>>>>>>>>> compact.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日
> 週六
> >>>>>>>>>>>>>>>> 上午1:55寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now.
> >>>>>> Could
> >>>>>>>>> you
> >>>>>>>>>>>>>>>>> update
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
> >>>>>> deprecation.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> >>>>>> s7133700@gmail.com
> >>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
> >> ValidList#in
> >>>>>>>>>>>>>>>> method,
> >>>>>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful
> >> to
> >>>>>> have
> >>>>>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>>>>> methods:
> >>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> >>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty
> lists).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> >>>>>> document
> >>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
> >> fail
> >>> to
> >>>>>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> 2025年5月2日
> >>>>>>>>> 凌晨12:50
> >>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> >>>>>> document
> >>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
> >> fail
> >>> to
> >>>>>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> >>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
> >>>>>> maintain
> >>>>>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed
> >>> flag
> >>>>>> is
> >>>>>>>>>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty()
> to
> >>>>>>> remove
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> boolean
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >> 2025年5月1日
> >>>>>>>>> 凌晨4:17
> >>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's
> >> awkward
> >>>>>>> since
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's
> >> set
> >>> to
> >>>>>>>>>>>>>>>> true.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> >>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not
> introduce
> >>> new
> >>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we
> >> now
> >>>>>>>>>>>>>>>> explicitly
> >>>>>>>>>>>>>>>>>>>> throw a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
> >>>>>>>>> validation
> >>>>>>>>>>>>>>>>> phase
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current
> behavior.”
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>> 2025年4月25日
> >>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced
> >> by
> >>>>>>> this
> >>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during
> >>> the
> >>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the
> >> storage
> >>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> fails
> >>>>>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> >>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> >>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to
> the
> >>>>>>>>> storage
> >>>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty
> >> list
> >>>>>> will
> >>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during
> storage
> >>>>>>> format
> >>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>>> 2025年4月24日
> >>>>>>>>>>>>>>>>> 凌晨4:44
> >>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> >>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now.
> >>> With
> >>>>>>>>> this
> >>>>>>>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
> >>> accurate)
> >>>>>>>>>>>>>>>>> exception.
> >>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of
> agree
> >>>>>> with
> >>>>>>>>>>>>>>>>> Chia-Ping.
> >>>>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely
> to
> >>> be
> >>>>>> a
> >>>>>>>>>>>>>>>>> mistake
> >>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None,
> the
> >>>>>> user
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>> never
> >>>>>>>>>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let
> >> the
> >>>>>>> user
> >>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM
> Chia-Ping
> >>>>>> Tsai
> >>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand
> >> that
> >>>>>>>>>>>>>>>>> maintaining
> >>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the
> documentation,
> >>> we
> >>>>>>>>> don't
> >>>>>>>>>>>>>>>>> allow
> >>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
> >>>>>> consider
> >>>>>>>>>>>>>>>>> throwing
> >>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
> >>>>>>>>> streamline
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> >>>>>>>>> 2025年4月23日
> >>>>>>>>>>>>>>>> 週三
> >>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not
> >>> delete
> >>>>>> or
> >>>>>>>>>>>>>>>>> compact
> >>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't
> >> delete
> >>>>>>>>>>>>>>>> segments
> >>>>>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms,
> >>> but
> >>>>>> we
> >>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>> continue
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> >>>>>> log.local.retention.bytes/
> >>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space
> >> when
> >>>>>>>>> remote
> >>>>>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun
> >> Rao <
> >>>>>>>>>>>>>>>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values
> >> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> >>> and
> >>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> >>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
> >>> accordingly
> >>>>>>> and
> >>>>>>>>>>>>>>>>> revised
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy
> >> cannot
> >>> be
> >>>>>>>>> used
> >>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> >>>>>>>>>>>>>>>>> cleanup.policy,
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig
> >> step.
> >>>>>>>>>>>>>>>>>>> Additionally, a
> >>>>>>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID>
> >> 於
> >>>>>>>>>>>>>>>> 2025年4月22日
> >>>>>>>>>>>>>>>>>>>> 凌晨4:52
> >>>>>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new
> >> value
> >>>>>> None
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid.
> >> For
> >>>>>>>>>>>>>>>> example,
> >>>>>>>>>>>>>>>>> None
> >>>>>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be
> >> useful
> >>> to
> >>>>>>>>>>>>>>>>> document
> >>>>>>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during
> >> upgrade
> >>>>>> when
> >>>>>>>>> an
> >>>>>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear
> >>> what
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>> check
> >>>>>>>>>>>>>>>>>>>>>>>>>> does.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽
> >> <
> >>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> >> on
> >>>>>>>>>>>>>>>> KIP-1161:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> >>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left
> >>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> >>>>>>>>>>>>>>>> improvements:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate
> >>> whether
> >>>>>>> an
> >>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value
> for
> >>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>>
> >>
>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi Chia, Jun

Thanks for the reply,
I have updated the table in the KIP.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月11日 清晨7:38 寫道：
> 
> Hi, Jun
> 
> Thanks for the reply.
> 
> I think the following configs could use null as the default value and treat
> an "empty list" as an illegal value.
> 
> `sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
> `ssl.enabled.protocols`,
> and `log.dirs`
> 
> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：
> 
>> Hi, Chia-Ping, Jiunn-Yang,
>> 
>> Thanks for the reply.
>> 
>> As you said, if ssl.cipher.suites is empty, intuitively this means that no
>> cipher is allowed. So, this should be an illegal config. It just happens
>> that our implementation treats it as if the config is not set (meaning the
>> value is null). So, it seems that a more intuitive convention is: if the
>> config is not set, it defaults to all available cipher suites. If it's
>> explicitly set, it can't be empty.
>> 
>> Jun
>> 
>> 
>> 
>> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hi Jun, Chia
>>> 
>>>>> Also, it would be useful to add a compatibility column in the table.
>> If
>>> a
>>>>> value is formally disallowed, it would be useful to compare the old
>> and
>>> the
>>>>> new behavior (e.g. a different exception is thrown, etc).
>>> I’ve updated the table to show which exceptions will be thrown for each
>>> behavior.
>>> 
>>>>>> I don't understand the point of "5.0". Could you please share more
>>> details?
>>>>>> It seems to me changing the type or default value should be fine in
>>> minor
>>>>>> release if we don't break the existing property files.
>>> I don't think this requires any compatibility considerations, so I’ve
>>> removed the Kafka 5.0 changes section.
>>> 
>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so using
>>> `LIST`
>>>>>> instead of `String` does not seems to make sense.
>>> I have removed the config from KIP
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
>>>> 
>>>>> For ssl.cipher.suites, changing the default value from null to empty
>> is
>>>>> actually a breaking change. According to the doc, null means that all
>>> the
>>>>> available cipher suites are supported. I am thinking that we should
>>>>> continue to allow null as the default since it could represent a state
>>>>> different from empty.
>>>> 
>>>> "Empty list" seems to be a weird case. It appears to signify that no
>>> cipher suites are accepted. However, in the codebase, an empty list is
>>> handled as if all available cipher suites are supported [0]. We should
>> not
>>> support the case of "no supported cipher suite," as it doesn't make
>> sense.
>>>> 
>>>> In short, changing the default value from null to empty does not break
>>> the behavior.
>>>> 
>>>> [0]
>>> 
>> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
>>>> 
>>>> On 2025/07/08 17:01:07 Jun Rao wrote:
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we
>> will
>>>>> just continue to allow it in all future releases.
>>>>> 
>>>>> For ssl.cipher.suites, changing the default value from null to empty
>> is
>>>>> actually a breaking change. According to the doc, null means that all
>>> the
>>>>> available cipher suites are supported. I am thinking that we should
>>>>> continue to allow null as the default since it could represent a state
>>>>> different from empty.
>>>>> 
>>>>> Also, it would be useful to add a compatibility column in the table.
>> If
>>> a
>>>>> value is formally disallowed, it would be useful to compare the old
>> and
>>> the
>>>>> new behavior (e.g. a different exception is thrown, etc).
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com>
>>> wrote:
>>>>> 
>>>>>> hi Jiunn
>>>>>> 
>>>>>> Thanks for the explanation. I have updated the KIP accordingly: in
>>> Kafka
>>>>>>> 4.x, cleanup.policy
>>>>>>> can be set to an empty list with a warning message emitted; however,
>>> in
>>>>>>> Kafka 5.0, setting an
>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>> 
>>>>>> 
>>>>>> If we agree to support "empty list", the approach of "none" could be
>>>>>> eliminated, right?
>>>>>> 
>>>>>> I don't understand the point of "5.0". Could you please share more
>>> details?
>>>>>> It seems to me changing the type or default value should be fine in
>>> minor
>>>>>> release if we don't break the existing property files.
>>>>>> 
>>>>>> `max.connections.per.ip.overrides` is parsed as "map" type, so using
>>> `LIST`
>>>>>> instead of `String` does not seems to make sense.
>>>>>> 
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>>>> Null could make sense as the default value, but it just means that
>>> the
>>>>>>>>> config key is not set explicitly.
>>>>>>>>> Given that, it's probably simpler to just allow
>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>> Thanks for the explanation. I have updated the KIP accordingly: in
>>> Kafka
>>>>>>> 4.x, cleanup.policy
>>>>>>> can be set to an empty list with a warning message emitted; however,
>>> in
>>>>>>> Kafka 5.0, setting an
>>>>>>> empty list for cleanup.policy will no longer be allowed.
>>>>>>> 
>>>>>>>> For another, there is an additional issue which could be handled by
>>>>>> this
>>>>>>>> KIP. Some configs have "string" type, but they are handled as a
>> LIST.
>>>>>> For
>>>>>>>> example, `early.start.listeners`, `security.providers`. Could we
>>> change
>>>>>>>> their type to LIST to benefit from this KIP? I mean to make them
>>>>>>>> non-nullable.
>>>>>>> I have also updated the KIP to include the tables detailing the
>>> changes
>>>>>>> for LIST-type configuration
>>>>>>> default values and the migration from STRING-type to LIST-type
>>>>>>> configurations.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
>>>>>>>> 
>>>>>>>> hi all,
>>>>>>>> 
>>>>>>>> JR11. There is an existing config interceptor.classes that allows
>> an
>>>>>>> empty
>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
>>>>>> support
>>>>>>>>> empty lists in general. Given that, it's probably simpler to just
>>>>>> allow
>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> yes, that is what I meant before. Additionally, the `LIST` type
>>> should
>>>>>>> not
>>>>>>>> accept "null". Instead, we should use an empty list as the default
>>>>>> value.
>>>>>>>> For example, `sasl.oauthbearer.expected.audience` should never has
>>>>>> "null"
>>>>>>>> value. That could eliminate the null check.
>>>>>>>> 
>>>>>>>> For another, there is an additional issue which could be handled by
>>>>>> this
>>>>>>>> KIP. Some configs have "string" type, but they are handled as a
>> LIST.
>>>>>> For
>>>>>>>> example, `early.start.listeners`, `security.providers`. Could we
>>> change
>>>>>>>> their type to LIST to benefit from this KIP? I mean to make them
>>>>>>>> non-nullable.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply.
>>>>>>>>> 
>>>>>>>>> JR10. Regarding supporting the null value, I am not sure how one
>>> could
>>>>>>> set
>>>>>>>>> a null value in a config file. For example, if you set the
>> following
>>>>>> in
>>>>>>> a
>>>>>>>>> config file.
>>>>>>>>> configkey=
>>>>>>>>> The value of configkey will be an empty list, instead of null,
>>> right?
>>>>>>>>> 
>>>>>>>>> Null could make sense as the default value, but it just means that
>>> the
>>>>>>>>> config key is not set explicitly.
>>>>>>>>> 
>>>>>>>>> JR11. There is an existing config interceptor.classes that allows
>> an
>>>>>>> empty
>>>>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
>>>>>> support
>>>>>>>>> empty lists in general. Given that, it's probably simpler to just
>>>>>> allow
>>>>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hello Jun,
>>>>>>>>>> 
>>>>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional, that is
>>>>>>> right,
>>>>>>>>>> so I think
>>>>>>>>>> that we can allow null value, but not allow the empty
>>>>>>>>>> 
>>>>>>>>>> JR11: In this KIP, the goal is to address configurations that
>>> should
>>>>>>> not
>>>>>>>>>> accept empty arrays.
>>>>>>>>>> We want to prevent users from passing in an empty list when it's
>>> not
>>>>>>>>>> valid. I believe there are two
>>>>>>>>>> main scenarios to consider:
>>>>>>>>>> 
>>>>>>>>>> 1. Nullable but not empty:
>>>>>>>>>> The parameter allows either a null value or a non-empty list. In
>>> this
>>>>>>>>>> case,
>>>>>>>>>> if the user provides an empty list, we should throw a
>>>>>> ConfigException.
>>>>>>>>>> 
>>>>>>>>>> 2. Non-null and non-empty only: The parameter must always
>> contain a
>>>>>>>>>> non-empty list — null is also
>>>>>>>>>> invalid. In this case, if the user provides either a null or an
>>> empty
>>>>>>>>>> list, we should throw a ConfigException.
>>>>>>>>>> 
>>>>>>>>>> Therefore, I want to prevent scenarios where users can pass in an
>>>>>> empty
>>>>>>>>>> list.
>>>>>>>>>> 
>>>>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
>>>>>>>>> property.
>>>>>>>>>> This allows us to specify
>>>>>>>>>> whether a configuration permits null values or not.
>>>>>>>>>> 
>>>>>>>>>> JR13: I have updated the table
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> Sorry for the late reply. A few more comments.
>>>>>>>>>>> 
>>>>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says that
>>> it's
>>>>>>>>>>> optional. So an empty list seems to be valid.
>>>>>>>>>>> 
>>>>>>>>>>> JR11. Thinking a bit more. If we are going to support empty
>> lists
>>> in
>>>>>>>>>>> general, it's probably more consistent to just allow
>>>>>>> log.cleanup.policy
>>>>>>>>>> to
>>>>>>>>>>> be empty as Luke suggested, instead of introducing a new option
>>>>>> None.
>>>>>>>>>>> 
>>>>>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It
>> seems
>>>>>> that
>>>>>>>>>>> only ssl.cipher.suites allows null. The remaining ones shouldn't
>>> be
>>>>>>>>> null.
>>>>>>>>>>> 
>>>>>>>>>>> JR13. The table is very helpful. Could you also describe the
>>> old/new
>>>>>>>>>>> behavior for each config with an empty list or duplicate entry?
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hello all,
>>>>>>>>>>>> 
>>>>>>>>>>>> I am manually bumping this thread. Any feedback would be
>>>>>> appreciated.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have updated the KIP and introduced a new validator,
>>>>>>>>>>>> NonEmptyListValidator, which ensures that the
>>>>>>>>>>>>> provided list is either null or a non-empty list without
>>> duplicate
>>>>>>>>>>>> entries. I’ve also listed the configurations that will
>>>>>>>>>>>>> be validated using this logic. Please feel free to share any
>>>>>>> feedback
>>>>>>>>>> or
>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> It seems there are quite a few other configs of type list
>> (e.g.
>>>>>>>>>> several
>>>>>>>>>>>> SSL
>>>>>>>>>>>>>> related ones). It would be useful to understand if empty
>> lists
>>>>>> are
>>>>>>>>>> valid
>>>>>>>>>>>>>> for them or not.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io>
>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> "My thought is that this behavior has been existed
>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
>>> there
>>>>>>>>> could
>>>>>>>>>>>> be
>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long time. "
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If you do a google search on "kafka infinite retention", you
>>>>>> will
>>>>>>>>> get
>>>>>>>>>>>>>>> links like
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
>>>>>>>>>>>> ,
>>>>>>>>>>>>>>> both pointing to setting the retention time and retention
>> size
>>>>>> to
>>>>>>>>> -1
>>>>>>>>>> to
>>>>>>>>>>>>>>> achieve this goal. So, it seems that if a user intentionally
>>>>>> wants
>>>>>>>>>>>> infinite
>>>>>>>>>>>>>>> retention, it's more likely they will use that approach
>>> instead
>>>>>> of
>>>>>>>>>>>> setting
>>>>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our API/tools
>> make
>>>>>> it
>>>>>>>>>>>> possible
>>>>>>>>>>>>>>> for users to make a mistake by inadvertently leaving the
>>> config
>>>>>>>>> value
>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
>> showuon@gmail.com
>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never documented
>>> that
>>>>>> an
>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
>>>>>> leaves
>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
>> since
>>>>>> it
>>>>>>>>>> will
>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>> the cleaning based on local retention size and time too.
>> So,
>>>>>>>>> leaving
>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
>>> than
>>>>>> an
>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
>> behind
>>>>>> the
>>>>>>>>>> KIP
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
>>> this
>>>>>>>>>> likely
>>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
>>>>>> infinite
>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> While we never documented the empty cleanup.policy implies
>>>>>>>>> infinite
>>>>>>>>>>>>>>>> retention, we also never documented (or failed) the empty
>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> an invalid configuration. My thought is that this behavior
>>> has
>>>>>>>>> been
>>>>>>>>>>>>>>>> existed
>>>>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
>>> there
>>>>>>>>>> could
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long time.
>> It
>>>>>> is
>>>>>>>>> not
>>>>>>>>>>>> good
>>>>>>>>>>>>>>>> to break the backward compatibility in a minor release
>>> version
>>>>>>>>> (ex:
>>>>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long existing
>>>>>> bug,
>>>>>>>>> it
>>>>>>>>>>>> could
>>>>>>>>>>>>>>>> be a surprise to users.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
>> by
>>>>>>>>> fixing
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>> realize
>>>>>> this
>>>>>>>>> if
>>>>>>>>>>>> it's
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Good point! I agree we need to fix it!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and
>> in
>>>>>> 5.0
>>>>>>>>> we
>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>> exception directly to make it fail fast
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This suggestion looks good to me!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
>>>>>>>>> chia7712@apache.org
>>>>>>>>>>> 
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is
>> worth
>>>>>>>>>>>> considering
>>>>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
>>> encountered
>>>>>>>>> this
>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> case but advised customers to utilize retention
>>> configurations
>>>>>>>>>>>> instead.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
>>> by
>>>>>>>>>> fixing
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>>> realize
>>>>>>>>> this
>>>>>>>>>> if
>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for
>>>>>> remote
>>>>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered
>> storage
>>>>>> use
>>>>>>>>> a
>>>>>>>>>>>> valid
>>>>>>>>>>>>>>>>> delete policy.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Regarding the motivation, currently, we never documented
>>> that
>>>>>>> an
>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
>> one
>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
>>> since
>>>>>>> it
>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>>>>> the cleaning based on local retention size and time too.
>>> So,
>>>>>>>>>> leaving
>>>>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
>>>>>> than
>>>>>>>>> an
>>>>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
>>> behind
>>>>>>>>> the
>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
>>>>>> this
>>>>>>>>>>>> likely
>>>>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
>>>>>>> infinite
>>>>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
>>> by
>>>>>>>>>> fixing
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> issue in remote storage. But then the user may never
>>> realize
>>>>>>>>> this
>>>>>>>>>> if
>>>>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
>>> showuon@gmail.com
>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Comments:
>>>>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
>>>>>>>>>> considered
>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
>>>>>>>>>>>>>>>>>>> Could you explain more about this? Why is this trade-off
>>>>>>>>>> justified?
>>>>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in
>> kafka,
>>>>>> why
>>>>>>>>> do
>>>>>>>>>> we
>>>>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
>>>>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
>>>>>> history
>>>>>>>>>> data
>>>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>>> a cleanup policy.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
>>>>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
>>> "process.roles"
>>>>>>>>>> configs
>>>>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>>>>> they are invalid.
>>>>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive
>>> reason
>>>>>>> why
>>>>>>>>>> we
>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>> break backward compatibility to change it.
>>>>>>>>>>>>>>>>>>> "This provides a clear and direct way to configure Kafka
>>> to
>>>>>>>>>> retain
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>>>>> segments without any automatic cleanup."
>>>>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to
>> me,
>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>> compatibility is much more important than "a clear and
>>>>>> direct
>>>>>>>>> way
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>> kafka".
>>>>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy" config?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
>>>>>>>>> <jun@confluent.io.invalid
>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7133700@gmail.com
>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
>>> addressed
>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
>>>>>> Additionally,
>>>>>>>>> an
>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>> will now be thrown if a
>>>>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
>>>>>>> inNonEmpty
>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日
>>> 凌晨12:46
>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty
>>>>>>> config,
>>>>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> be non-empty too?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
>>>>>>> directly.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
>>>>>>>>> 上午12:28寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references to
>>> the
>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in
>>> Kafka
>>>>>>>>> 5.0"
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
>>>>>> addressed
>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日
>>>>>> 凌晨2:03
>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration
>>>>>> Plan",
>>>>>>>>>>>>>>>>> could you
>>>>>>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>>>>>>>>> description to remind readers that "clean.policy="
>>>>>>> should
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> replaced
>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable
>>> delete
>>>>>>> and
>>>>>>>>>>>>>>>>> compact.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
>>>>>>>>>>>>>>>> 上午1:55寫道：
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now.
>>>>>> Could
>>>>>>>>> you
>>>>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
>>>>>> deprecation.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
>>>>>> s7133700@gmail.com
>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
>> ValidList#in
>>>>>>>>>>>>>>>> method,
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful
>> to
>>>>>> have
>>>>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>>>>> methods:
>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
>>>>>> document
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
>> fail
>>> to
>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
>>>>>>>>> 凌晨12:50
>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
>>>>>> document
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
>> fail
>>> to
>>>>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
>>>>>> maintain
>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed
>>> flag
>>>>>> is
>>>>>>>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to
>>>>>>> remove
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> boolean
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>> 2025年5月1日
>>>>>>>>> 凌晨4:17
>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's
>> awkward
>>>>>>> since
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's
>> set
>>> to
>>>>>>>>>>>>>>>> true.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce
>>> new
>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we
>> now
>>>>>>>>>>>>>>>> explicitly
>>>>>>>>>>>>>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
>>>>>>>>> validation
>>>>>>>>>>>>>>>>> phase
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>> 2025年4月25日
>>>>>>>>>>>>>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced
>> by
>>>>>>> this
>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during
>>> the
>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the
>> storage
>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
>>>>>>>>> storage
>>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty
>> list
>>>>>> will
>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage
>>>>>>> format
>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>> 2025年4月24日
>>>>>>>>>>>>>>>>> 凌晨4:44
>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now.
>>> With
>>>>>>>>> this
>>>>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
>>> accurate)
>>>>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree
>>>>>> with
>>>>>>>>>>>>>>>>> Chia-Ping.
>>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to
>>> be
>>>>>> a
>>>>>>>>>>>>>>>>> mistake
>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the
>>>>>> user
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let
>> the
>>>>>>> user
>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping
>>>>>> Tsai
>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand
>> that
>>>>>>>>>>>>>>>>> maintaining
>>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation,
>>> we
>>>>>>>>> don't
>>>>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
>>>>>> consider
>>>>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
>>>>>>>>> streamline
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>>>>>>>>> 2025年4月23日
>>>>>>>>>>>>>>>> 週三
>>>>>>>>>>>>>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not
>>> delete
>>>>>> or
>>>>>>>>>>>>>>>>> compact
>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't
>> delete
>>>>>>>>>>>>>>>> segments
>>>>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms,
>>> but
>>>>>> we
>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
>>>>>> log.local.retention.bytes/
>>>>>>>>>>>>>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space
>> when
>>>>>>>>> remote
>>>>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun
>> Rao <
>>>>>>>>>>>>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values
>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>>> and
>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
>>> accordingly
>>>>>>> and
>>>>>>>>>>>>>>>>> revised
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy
>> cannot
>>> be
>>>>>>>>> used
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
>>>>>>>>>>>>>>>>> cleanup.policy,
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig
>> step.
>>>>>>>>>>>>>>>>>>> Additionally, a
>>>>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID>
>> 於
>>>>>>>>>>>>>>>> 2025年4月22日
>>>>>>>>>>>>>>>>>>>> 凌晨4:52
>>>>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new
>> value
>>>>>> None
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid.
>> For
>>>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be
>> useful
>>> to
>>>>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during
>> upgrade
>>>>>> when
>>>>>>>>> an
>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear
>>> what
>>>>>>>>> the
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽
>> <
>>>>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
>> on
>>>>>>>>>>>>>>>> KIP-1161:
>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left
>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
>>>>>>>>>>>>>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate
>>> whether
>>>>>>> an
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

Hi, Jun

Thanks for the reply.

I think the following configs could use null as the default value and treat
an "empty list" as an illegal value.

`sasl.oauthbearer.expected.audience`, `ssl.cipher.suites`,
`ssl.enabled.protocols`,
and `log.dirs`

Jun Rao <ju...@confluent.io.invalid> 於 2025年7月11日 週五 上午6:59寫道：

> Hi, Chia-Ping, Jiunn-Yang,
>
> Thanks for the reply.
>
> As you said, if ssl.cipher.suites is empty, intuitively this means that no
> cipher is allowed. So, this should be an illegal config. It just happens
> that our implementation treats it as if the config is not set (meaning the
> value is null). So, it seems that a more intuitive convention is: if the
> config is not set, it defaults to all available cipher suites. If it's
> explicitly set, it can't be empty.
>
> Jun
>
>
>
> On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hi Jun, Chia
> >
> > >> Also, it would be useful to add a compatibility column in the table.
> If
> > a
> > >> value is formally disallowed, it would be useful to compare the old
> and
> > the
> > >> new behavior (e.g. a different exception is thrown, etc).
> > I’ve updated the table to show which exceptions will be thrown for each
> > behavior.
> >
> > >>> I don't understand the point of "5.0". Could you please share more
> > details?
> > >>> It seems to me changing the type or default value should be fine in
> > minor
> > >>> release if we don't break the existing property files.
> > I don't think this requires any compatibility considerations, so I’ve
> > removed the Kafka 5.0 changes section.
> >
> > >>> `max.connections.per.ip.overrides` is parsed as "map" type, so using
> > `LIST`
> > >>> instead of `String` does not seems to make sense.
> > I have removed the config from KIP
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> > >
> > >> For ssl.cipher.suites, changing the default value from null to empty
> is
> > >> actually a breaking change. According to the doc, null means that all
> > the
> > >> available cipher suites are supported. I am thinking that we should
> > >> continue to allow null as the default since it could represent a state
> > >> different from empty.
> > >
> > > "Empty list" seems to be a weird case. It appears to signify that no
> > cipher suites are accepted. However, in the codebase, an empty list is
> > handled as if all available cipher suites are supported [0]. We should
> not
> > support the case of "no supported cipher suite," as it doesn't make
> sense.
> > >
> > > In short, changing the default value from null to empty does not break
> > the behavior.
> > >
> > > [0]
> >
> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> > >
> > > On 2025/07/08 17:01:07 Jun Rao wrote:
> > >> Hi, Jiunn-Yang,
> > >>
> > >> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we
> will
> > >> just continue to allow it in all future releases.
> > >>
> > >> For ssl.cipher.suites, changing the default value from null to empty
> is
> > >> actually a breaking change. According to the doc, null means that all
> > the
> > >> available cipher suites are supported. I am thinking that we should
> > >> continue to allow null as the default since it could represent a state
> > >> different from empty.
> > >>
> > >> Also, it would be useful to add a compatibility column in the table.
> If
> > a
> > >> value is formally disallowed, it would be useful to compare the old
> and
> > the
> > >> new behavior (e.g. a different exception is thrown, etc).
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com>
> > wrote:
> > >>
> > >>> hi Jiunn
> > >>>
> > >>> Thanks for the explanation. I have updated the KIP accordingly: in
> > Kafka
> > >>>> 4.x, cleanup.policy
> > >>>> can be set to an empty list with a warning message emitted; however,
> > in
> > >>>> Kafka 5.0, setting an
> > >>>> empty list for cleanup.policy will no longer be allowed.
> > >>>
> > >>>
> > >>> If we agree to support "empty list", the approach of "none" could be
> > >>> eliminated, right?
> > >>>
> > >>> I don't understand the point of "5.0". Could you please share more
> > details?
> > >>> It seems to me changing the type or default value should be fine in
> > minor
> > >>> release if we don't break the existing property files.
> > >>>
> > >>> `max.connections.per.ip.overrides` is parsed as "map" type, so using
> > `LIST`
> > >>> instead of `String` does not seems to make sense.
> > >>>
> > >>> Best,
> > >>> Chia-Ping
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> > >>>
> > >>>> Hi all,
> > >>>>
> > >>>>>> Null could make sense as the default value, but it just means that
> > the
> > >>>>>> config key is not set explicitly.
> > >>>>>> Given that, it's probably simpler to just allow
> > >>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>> Thanks for the explanation. I have updated the KIP accordingly: in
> > Kafka
> > >>>> 4.x, cleanup.policy
> > >>>> can be set to an empty list with a warning message emitted; however,
> > in
> > >>>> Kafka 5.0, setting an
> > >>>> empty list for cleanup.policy will no longer be allowed.
> > >>>>
> > >>>>> For another, there is an additional issue which could be handled by
> > >>> this
> > >>>>> KIP. Some configs have "string" type, but they are handled as a
> LIST.
> > >>> For
> > >>>>> example, `early.start.listeners`, `security.providers`. Could we
> > change
> > >>>>> their type to LIST to benefit from this KIP? I mean to make them
> > >>>>> non-nullable.
> > >>>> I have also updated the KIP to include the tables detailing the
> > changes
> > >>>> for LIST-type configuration
> > >>>> default values and the migration from STRING-type to LIST-type
> > >>>> configurations.
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> > >>>>>
> > >>>>> hi all,
> > >>>>>
> > >>>>> JR11. There is an existing config interceptor.classes that allows
> an
> > >>>> empty
> > >>>>>> list and it makes intuitive sense. So, it seems that it's ok to
> > >>> support
> > >>>>>> empty lists in general. Given that, it's probably simpler to just
> > >>> allow
> > >>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>>>
> > >>>>>
> > >>>>> yes, that is what I meant before. Additionally, the `LIST` type
> > should
> > >>>> not
> > >>>>> accept "null". Instead, we should use an empty list as the default
> > >>> value.
> > >>>>> For example, `sasl.oauthbearer.expected.audience` should never has
> > >>> "null"
> > >>>>> value. That could eliminate the null check.
> > >>>>>
> > >>>>> For another, there is an additional issue which could be handled by
> > >>> this
> > >>>>> KIP. Some configs have "string" type, but they are handled as a
> LIST.
> > >>> For
> > >>>>> example, `early.start.listeners`, `security.providers`. Could we
> > change
> > >>>>> their type to LIST to benefit from this KIP? I mean to make them
> > >>>>> non-nullable.
> > >>>>>
> > >>>>> Best,
> > >>>>> Chia-Ping
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> > >>>>>
> > >>>>>> Hi, Jiunn-Yang,
> > >>>>>>
> > >>>>>> Thanks for the reply.
> > >>>>>>
> > >>>>>> JR10. Regarding supporting the null value, I am not sure how one
> > could
> > >>>> set
> > >>>>>> a null value in a config file. For example, if you set the
> following
> > >>> in
> > >>>> a
> > >>>>>> config file.
> > >>>>>> configkey=
> > >>>>>> The value of configkey will be an empty list, instead of null,
> > right?
> > >>>>>>
> > >>>>>> Null could make sense as the default value, but it just means that
> > the
> > >>>>>> config key is not set explicitly.
> > >>>>>>
> > >>>>>> JR11. There is an existing config interceptor.classes that allows
> an
> > >>>> empty
> > >>>>>> list and it makes intuitive sense. So, it seems that it's ok to
> > >>> support
> > >>>>>> empty lists in general. Given that, it's probably simpler to just
> > >>> allow
> > >>>>>> log.cleanup.policy to be empty and properly support it?
> > >>>>>>
> > >>>>>> Jun
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> Hello Jun,
> > >>>>>>>
> > >>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional, that is
> > >>>> right,
> > >>>>>>> so I think
> > >>>>>>> that we can allow null value, but not allow the empty
> > >>>>>>>
> > >>>>>>> JR11: In this KIP, the goal is to address configurations that
> > should
> > >>>> not
> > >>>>>>> accept empty arrays.
> > >>>>>>> We want to prevent users from passing in an empty list when it's
> > not
> > >>>>>>> valid. I believe there are two
> > >>>>>>> main scenarios to consider:
> > >>>>>>>
> > >>>>>>> 1. Nullable but not empty:
> > >>>>>>> The parameter allows either a null value or a non-empty list. In
> > this
> > >>>>>>> case,
> > >>>>>>> if the user provides an empty list, we should throw a
> > >>> ConfigException.
> > >>>>>>>
> > >>>>>>> 2. Non-null and non-empty only: The parameter must always
> contain a
> > >>>>>>> non-empty list — null is also
> > >>>>>>> invalid. In this case, if the user provides either a null or an
> > empty
> > >>>>>>> list, we should throw a ConfigException.
> > >>>>>>>
> > >>>>>>> Therefore, I want to prevent scenarios where users can pass in an
> > >>> empty
> > >>>>>>> list.
> > >>>>>>>
> > >>>>>>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
> > >>>>>> property.
> > >>>>>>> This allows us to specify
> > >>>>>>> whether a configuration permits null values or not.
> > >>>>>>>
> > >>>>>>> JR13: I have updated the table
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> > >>>>>>>>
> > >>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>
> > >>>>>>>> Sorry for the late reply. A few more comments.
> > >>>>>>>>
> > >>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says that
> > it's
> > >>>>>>>> optional. So an empty list seems to be valid.
> > >>>>>>>>
> > >>>>>>>> JR11. Thinking a bit more. If we are going to support empty
> lists
> > in
> > >>>>>>>> general, it's probably more consistent to just allow
> > >>>> log.cleanup.policy
> > >>>>>>> to
> > >>>>>>>> be empty as Luke suggested, instead of introducing a new option
> > >>> None.
> > >>>>>>>>
> > >>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It
> seems
> > >>> that
> > >>>>>>>> only ssl.cipher.suites allows null. The remaining ones shouldn't
> > be
> > >>>>>> null.
> > >>>>>>>>
> > >>>>>>>> JR13. The table is very helpful. Could you also describe the
> > old/new
> > >>>>>>>> behavior for each config with an empty list or duplicate entry?
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hello all,
> > >>>>>>>>>
> > >>>>>>>>> I am manually bumping this thread. Any feedback would be
> > >>> appreciated.
> > >>>>>>>>>
> > >>>>>>>>> Best regards,
> > >>>>>>>>> Jiunn-Yang
> > >>>>>>>>>
> > >>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> > >>>>>>>>>>
> > >>>>>>>>>> Hello Jun,
> > >>>>>>>>>>
> > >>>>>>>>>> I have updated the KIP and introduced a new validator,
> > >>>>>>>>> NonEmptyListValidator, which ensures that the
> > >>>>>>>>>> provided list is either null or a non-empty list without
> > duplicate
> > >>>>>>>>> entries. I’ve also listed the configurations that will
> > >>>>>>>>>> be validated using this logic. Please feel free to share any
> > >>>> feedback
> > >>>>>>> or
> > >>>>>>>>> suggestions.
> > >>>>>>>>>>
> > >>>>>>>>>> Best Regards,
> > >>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>
> > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>
> > >>>>>>>>>>> It seems there are quite a few other configs of type list
> (e.g.
> > >>>>>>> several
> > >>>>>>>>> SSL
> > >>>>>>>>>>> related ones). It would be useful to understand if empty
> lists
> > >>> are
> > >>>>>>> valid
> > >>>>>>>>>>> for them or not.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Jun
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io>
> > >>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Luke,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> "My thought is that this behavior has been existed
> > >>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
> > there
> > >>>>>> could
> > >>>>>>>>> be
> > >>>>>>>>>>>> users relying on the empty cleanup.policy for a long time. "
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If you do a google search on "kafka infinite retention", you
> > >>> will
> > >>>>>> get
> > >>>>>>>>>>>> links like
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> > >>>>>>>>>>>> and
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> >
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> > >>>>>>>>> ,
> > >>>>>>>>>>>> both pointing to setting the retention time and retention
> size
> > >>> to
> > >>>>>> -1
> > >>>>>>> to
> > >>>>>>>>>>>> achieve this goal. So, it seems that if a user intentionally
> > >>> wants
> > >>>>>>>>> infinite
> > >>>>>>>>>>>> retention, it's more likely they will use that approach
> > instead
> > >>> of
> > >>>>>>>>> setting
> > >>>>>>>>>>>> cleanup.policy to empty. On the other hand, our API/tools
> make
> > >>> it
> > >>>>>>>>> possible
> > >>>>>>>>>>>> for users to make a mistake by inadvertently leaving the
> > config
> > >>>>>> value
> > >>>>>>>>> empty.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <
> showuon@gmail.com
> > >
> > >>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Regarding the motivation, currently, we never documented
> > that
> > >>> an
> > >>>>>>>>> empty
> > >>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> > >>> leaves
> > >>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
> since
> > >>> it
> > >>>>>>> will
> > >>>>>>>>> stop
> > >>>>>>>>>>>>> the cleaning based on local retention size and time too.
> So,
> > >>>>>> leaving
> > >>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
> > than
> > >>> an
> > >>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
> behind
> > >>> the
> > >>>>>>> KIP
> > >>>>>>>>> is
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
> > this
> > >>>>>>> likely
> > >>>>>>>>>>>>> mistake and provide a new option None for users wanting
> > >>> infinite
> > >>>>>>>>> retention
> > >>>>>>>>>>>>> to opt in explicitly.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> While we never documented the empty cleanup.policy implies
> > >>>>>> infinite
> > >>>>>>>>>>>>> retention, we also never documented (or failed) the empty
> > >>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>> is
> > >>>>>>>>>>>>> an invalid configuration. My thought is that this behavior
> > has
> > >>>>>> been
> > >>>>>>>>>>>>> existed
> > >>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
> > there
> > >>>>>>> could
> > >>>>>>>>> be
> > >>>>>>>>>>>>> users relying on the empty cleanup.policy for a long time.
> It
> > >>> is
> > >>>>>> not
> > >>>>>>>>> good
> > >>>>>>>>>>>>> to break the backward compatibility in a minor release
> > version
> > >>>>>> (ex:
> > >>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long existing
> > >>> bug,
> > >>>>>> it
> > >>>>>>>>> could
> > >>>>>>>>>>>>> be a surprise to users.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
> by
> > >>>>>> fixing
> > >>>>>>>>> the
> > >>>>>>>>>>>>> issue in remote storage. But then the user may never
> realize
> > >>> this
> > >>>>>> if
> > >>>>>>>>> it's
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Good point! I agree we need to fix it!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi Chia-Ping,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and
> in
> > >>> 5.0
> > >>>>>> we
> > >>>>>>>>>>>>> throw
> > >>>>>>>>>>>>> exception directly to make it fail fast
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> This suggestion looks good to me!
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thank you.
> > >>>>>>>>>>>>> Luke
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> > >>>>>> chia7712@apache.org
> > >>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> hi all,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is
> worth
> > >>>>>>>>> considering
> > >>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
> > encountered
> > >>>>>> this
> > >>>>>>>>> use
> > >>>>>>>>>>>>>> case but advised customers to utilize retention
> > configurations
> > >>>>>>>>> instead.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
> > by
> > >>>>>>> fixing
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> > realize
> > >>>>>> this
> > >>>>>>> if
> > >>>>>>>>>>>>>> it's a
> > >>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for
> > >>> remote
> > >>>>>>>>>>>>> storage.
> > >>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered
> storage
> > >>> use
> > >>>>>> a
> > >>>>>>>>> valid
> > >>>>>>>>>>>>>> delete policy.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> > >>>>>>>>>>>>>>> Hi, Luke,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Regarding the motivation, currently, we never documented
> > that
> > >>>> an
> > >>>>>>>>> empty
> > >>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if
> one
> > >>>>>> leaves
> > >>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
> > since
> > >>>> it
> > >>>>>>>>> will
> > >>>>>>>>>>>>>> stop
> > >>>>>>>>>>>>>>> the cleaning based on local retention size and time too.
> > So,
> > >>>>>>> leaving
> > >>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
> > >>> than
> > >>>>>> an
> > >>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
> > behind
> > >>>>>> the
> > >>>>>>>>> KIP
> > >>>>>>>>>>>>> is
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
> > >>> this
> > >>>>>>>>> likely
> > >>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
> > >>>> infinite
> > >>>>>>>>>>>>>> retention
> > >>>>>>>>>>>>>>> to opt in explicitly.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
> > by
> > >>>>>>> fixing
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> > realize
> > >>>>>> this
> > >>>>>>> if
> > >>>>>>>>>>>>>> it's a
> > >>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
> > showuon@gmail.com
> > >>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Comments:
> > >>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
> > >>>>>>> considered
> > >>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> > >>>>>>>>>>>>>>>> Could you explain more about this? Why is this trade-off
> > >>>>>>> justified?
> > >>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in
> kafka,
> > >>> why
> > >>>>>> do
> > >>>>>>> we
> > >>>>>>>>>>>>>> provide
> > >>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
> > >>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
> > >>> history
> > >>>>>>> data
> > >>>>>>>>>>>>>> without
> > >>>>>>>>>>>>>>>> a cleanup policy.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
> > >>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
> > "process.roles"
> > >>>>>>> configs
> > >>>>>>>>>>>>>> since
> > >>>>>>>>>>>>>>>> they are invalid.
> > >>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive
> > reason
> > >>>> why
> > >>>>>>> we
> > >>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>> break backward compatibility to change it.
> > >>>>>>>>>>>>>>>> "This provides a clear and direct way to configure Kafka
> > to
> > >>>>>>> retain
> > >>>>>>>>>>>>> all
> > >>>>>>>>>>>>>> log
> > >>>>>>>>>>>>>>>> segments without any automatic cleanup."
> > >>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to
> me,
> > >>>>>>> backward
> > >>>>>>>>>>>>>>>> compatibility is much more important than "a clear and
> > >>> direct
> > >>>>>> way
> > >>>>>>>>> to
> > >>>>>>>>>>>>>> config
> > >>>>>>>>>>>>>>>> kafka".
> > >>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy" config?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks.
> > >>>>>>>>>>>>>>>> Luke
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> > >>>>>> <jun@confluent.io.invalid
> > >>>>>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7133700@gmail.com
> >
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi Jun, chia
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> > addressed
> > >>> in
> > >>>>>>> the
> > >>>>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
> > >>> Additionally,
> > >>>>>> an
> > >>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>> will now be thrown if a
> > >>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
> > >>>> inNonEmpty
> > >>>>>>>>>>>>>> method.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日
> > 凌晨12:46
> > >>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty
> > >>>> config,
> > >>>>>>>>>>>>> so
> > >>>>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>> be non-empty too?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
> > >>>> directly.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
> > >>>>>> 上午12:28寫道：
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references to
> > the
> > >>>>>>>>>>>>>> deprecated
> > >>>>>>>>>>>>>>>>>> method.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> > >>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in
> > Kafka
> > >>>>>> 5.0"
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
> > s7133700@gmail.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> > >>> addressed
> > >>>>>> in
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日
> > >>> 凌晨2:03
> > >>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration
> > >>> Plan",
> > >>>>>>>>>>>>>> could you
> > >>>>>>>>>>>>>>>>> add
> > >>>>>>>>>>>>>>>>>>>>>> description to remind readers that "clean.policy="
> > >>>> should
> > >>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> replaced
> > >>>>>>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable
> > delete
> > >>>> and
> > >>>>>>>>>>>>>> compact.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
> > >>>>>>>>>>>>> 上午1:55寫道：
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now.
> > >>> Could
> > >>>>>> you
> > >>>>>>>>>>>>>> update
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
> > >>> deprecation.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> > >>> s7133700@gmail.com
> > >>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the
> ValidList#in
> > >>>>>>>>>>>>> method,
> > >>>>>>>>>>>>>> as
> > >>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
> > >>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful
> to
> > >>> have
> > >>>>>>>>>>>>> both
> > >>>>>>>>>>>>>>>>> methods:
> > >>>>>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> > >>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> > >>> document
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
> fail
> > to
> > >>>>>>>>>>>>> start.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
> > >>>>>> 凌晨12:50
> > >>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> > >>> document
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will
> fail
> > to
> > >>>>>>>>>>>>> start.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> > >>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
> > >>> maintain
> > >>>>>>>>>>>>>> backward
> > >>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed
> > flag
> > >>> is
> > >>>>>>>>>>>>>>>> necessary.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to
> > >>>> remove
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> boolean
> > >>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> 2025年5月1日
> > >>>>>> 凌晨4:17
> > >>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's
> awkward
> > >>>> since
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's
> set
> > to
> > >>>>>>>>>>>>> true.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> > >>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce
> > new
> > >>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>> configurations
> > >>>>>>>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we
> now
> > >>>>>>>>>>>>> explicitly
> > >>>>>>>>>>>>>>>>> throw a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
> > >>>>>> validation
> > >>>>>>>>>>>>>> phase
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >>> 2025年4月25日
> > >>>>>>>>>>>>>> 凌晨1:17 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced
> by
> > >>>> this
> > >>>>>>>>>>>>>> change
> > >>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during
> > the
> > >>>>>>>>>>>>> storage
> > >>>>>>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>>>>>>>> validation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the
> storage
> > >>>>>>>>>>>>> already
> > >>>>>>>>>>>>>> fails
> > >>>>>>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>>>>> empty.
> > >>>>>>>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> > >>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> > >>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
> > >>>>>> storage
> > >>>>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty
> list
> > >>> will
> > >>>>>>>>>>>>> now
> > >>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage
> > >>>> format
> > >>>>>>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >>>> 2025年4月24日
> > >>>>>>>>>>>>>> 凌晨4:44
> > >>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> > >>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> > >>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now.
> > With
> > >>>>>> this
> > >>>>>>>>>>>>>> KIP,
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
> > accurate)
> > >>>>>>>>>>>>>> exception.
> > >>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree
> > >>> with
> > >>>>>>>>>>>>>> Chia-Ping.
> > >>>>>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to
> > be
> > >>> a
> > >>>>>>>>>>>>>> mistake
> > >>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the
> > >>> user
> > >>>>>>>>>>>>> will
> > >>>>>>>>>>>>>> never
> > >>>>>>>>>>>>>>>>>>>>> realize
> > >>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let
> the
> > >>>> user
> > >>>>>>>>>>>>>> realize
> > >>>>>>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping
> > >>> Tsai
> > >>>> <
> > >>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand
> that
> > >>>>>>>>>>>>>> maintaining
> > >>>>>>>>>>>>>>>>>>>>>>>> compatibility
> > >>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation,
> > we
> > >>>>>> don't
> > >>>>>>>>>>>>>> allow
> > >>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>> value
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
> > >>> consider
> > >>>>>>>>>>>>>> throwing
> > >>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
> > >>>>>> streamline
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> code
> > >>>>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>> avoid
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> > >>>>>> 2025年4月23日
> > >>>>>>>>>>>>> 週三
> > >>>>>>>>>>>>>>>>>> 上午6:33寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not
> > delete
> > >>> or
> > >>>>>>>>>>>>>> compact
> > >>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>>>>>>> segments",
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't
> delete
> > >>>>>>>>>>>>> segments
> > >>>>>>>>>>>>>> based
> > >>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms,
> > but
> > >>> we
> > >>>>>>>>>>>>>> should
> > >>>>>>>>>>>>>>>>>>>> continue
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> > >>> log.local.retention.bytes/
> > >>>>>>>>>>>>>>>>>>>> log.retention.ms.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space
> when
> > >>>>>> remote
> > >>>>>>>>>>>>>>>> storage
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun
> Rao <
> > >>>>>>>>>>>>>>>>> jun@confluent.io>
> > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values
> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> > and
> > >>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> > >>>>>>>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
> > accordingly
> > >>>> and
> > >>>>>>>>>>>>>> revised
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy
> cannot
> > be
> > >>>>>> used
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> > >>>>>>>>>>>>>> cleanup.policy,
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig
> step.
> > >>>>>>>>>>>>>>>> Additionally, a
> > >>>>>>>>>>>>>>>>>>>>>>> warning
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID>
> 於
> > >>>>>>>>>>>>> 2025年4月22日
> > >>>>>>>>>>>>>>>>> 凌晨4:52
> > >>>>>>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new
> value
> > >>> None
> > >>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> But
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid.
> For
> > >>>>>>>>>>>>> example,
> > >>>>>>>>>>>>>> None
> > >>>>>>>>>>>>>>>>>> can't
> > >>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be
> useful
> > to
> > >>>>>>>>>>>>>> document
> > >>>>>>>>>>>>>>>>> that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during
> upgrade
> > >>> when
> > >>>>>> an
> > >>>>>>>>>>>>>>>> existing
> > >>>>>>>>>>>>>>>>>>>>>>> config
> > >>>>>>>>>>>>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear
> > what
> > >>>>>> the
> > >>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>> check
> > >>>>>>>>>>>>>>>>>>>>>>> does.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> How
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽
> <
> > >>>>>>>>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> on
> > >>>>>>>>>>>>> KIP-1161:
> > >>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> > >>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>> configuration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left
> > empty.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> > >>>>>>>>>>>>> improvements:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate
> > whether
> > >>>> an
> > >>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>> list
> > >>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> > >>>>>>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Chia-Ping, Jiunn-Yang,

Thanks for the reply.

As you said, if ssl.cipher.suites is empty, intuitively this means that no
cipher is allowed. So, this should be an illegal config. It just happens
that our implementation treats it as if the config is not set (meaning the
value is null). So, it seems that a more intuitive convention is: if the
config is not set, it defaults to all available cipher suites. If it's
explicitly set, it can't be empty.

Jun



On Wed, Jul 9, 2025 at 8:27 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hi Jun, Chia
>
> >> Also, it would be useful to add a compatibility column in the table. If
> a
> >> value is formally disallowed, it would be useful to compare the old and
> the
> >> new behavior (e.g. a different exception is thrown, etc).
> I’ve updated the table to show which exceptions will be thrown for each
> behavior.
>
> >>> I don't understand the point of "5.0". Could you please share more
> details?
> >>> It seems to me changing the type or default value should be fine in
> minor
> >>> release if we don't break the existing property files.
> I don't think this requires any compatibility considerations, so I’ve
> removed the Kafka 5.0 changes section.
>
> >>> `max.connections.per.ip.overrides` is parsed as "map" type, so using
> `LIST`
> >>> instead of `String` does not seems to make sense.
> I have removed the config from KIP
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> >
> >> For ssl.cipher.suites, changing the default value from null to empty is
> >> actually a breaking change. According to the doc, null means that all
> the
> >> available cipher suites are supported. I am thinking that we should
> >> continue to allow null as the default since it could represent a state
> >> different from empty.
> >
> > "Empty list" seems to be a weird case. It appears to signify that no
> cipher suites are accepted. However, in the codebase, an empty list is
> handled as if all available cipher suites are supported [0]. We should not
> support the case of "no supported cipher suite," as it doesn't make sense.
> >
> > In short, changing the default value from null to empty does not break
> the behavior.
> >
> > [0]
> https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> >
> > On 2025/07/08 17:01:07 Jun Rao wrote:
> >> Hi, Jiunn-Yang,
> >>
> >> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we will
> >> just continue to allow it in all future releases.
> >>
> >> For ssl.cipher.suites, changing the default value from null to empty is
> >> actually a breaking change. According to the doc, null means that all
> the
> >> available cipher suites are supported. I am thinking that we should
> >> continue to allow null as the default since it could represent a state
> >> different from empty.
> >>
> >> Also, it would be useful to add a compatibility column in the table. If
> a
> >> value is formally disallowed, it would be useful to compare the old and
> the
> >> new behavior (e.g. a different exception is thrown, etc).
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com>
> wrote:
> >>
> >>> hi Jiunn
> >>>
> >>> Thanks for the explanation. I have updated the KIP accordingly: in
> Kafka
> >>>> 4.x, cleanup.policy
> >>>> can be set to an empty list with a warning message emitted; however,
> in
> >>>> Kafka 5.0, setting an
> >>>> empty list for cleanup.policy will no longer be allowed.
> >>>
> >>>
> >>> If we agree to support "empty list", the approach of "none" could be
> >>> eliminated, right?
> >>>
> >>> I don't understand the point of "5.0". Could you please share more
> details?
> >>> It seems to me changing the type or default value should be fine in
> minor
> >>> release if we don't break the existing property files.
> >>>
> >>> `max.connections.per.ip.overrides` is parsed as "map" type, so using
> `LIST`
> >>> instead of `String` does not seems to make sense.
> >>>
> >>> Best,
> >>> Chia-Ping
> >>>
> >>>
> >>>
> >>>
> >>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> >>>
> >>>> Hi all,
> >>>>
> >>>>>> Null could make sense as the default value, but it just means that
> the
> >>>>>> config key is not set explicitly.
> >>>>>> Given that, it's probably simpler to just allow
> >>>>>> log.cleanup.policy to be empty and properly support it?
> >>>> Thanks for the explanation. I have updated the KIP accordingly: in
> Kafka
> >>>> 4.x, cleanup.policy
> >>>> can be set to an empty list with a warning message emitted; however,
> in
> >>>> Kafka 5.0, setting an
> >>>> empty list for cleanup.policy will no longer be allowed.
> >>>>
> >>>>> For another, there is an additional issue which could be handled by
> >>> this
> >>>>> KIP. Some configs have "string" type, but they are handled as a LIST.
> >>> For
> >>>>> example, `early.start.listeners`, `security.providers`. Could we
> change
> >>>>> their type to LIST to benefit from this KIP? I mean to make them
> >>>>> non-nullable.
> >>>> I have also updated the KIP to include the tables detailing the
> changes
> >>>> for LIST-type configuration
> >>>> default values and the migration from STRING-type to LIST-type
> >>>> configurations.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> >>>>>
> >>>>> hi all,
> >>>>>
> >>>>> JR11. There is an existing config interceptor.classes that allows an
> >>>> empty
> >>>>>> list and it makes intuitive sense. So, it seems that it's ok to
> >>> support
> >>>>>> empty lists in general. Given that, it's probably simpler to just
> >>> allow
> >>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>
> >>>>>
> >>>>> yes, that is what I meant before. Additionally, the `LIST` type
> should
> >>>> not
> >>>>> accept "null". Instead, we should use an empty list as the default
> >>> value.
> >>>>> For example, `sasl.oauthbearer.expected.audience` should never has
> >>> "null"
> >>>>> value. That could eliminate the null check.
> >>>>>
> >>>>> For another, there is an additional issue which could be handled by
> >>> this
> >>>>> KIP. Some configs have "string" type, but they are handled as a LIST.
> >>> For
> >>>>> example, `early.start.listeners`, `security.providers`. Could we
> change
> >>>>> their type to LIST to benefit from this KIP? I mean to make them
> >>>>> non-nullable.
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>>
> >>>>>
> >>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> >>>>>
> >>>>>> Hi, Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> JR10. Regarding supporting the null value, I am not sure how one
> could
> >>>> set
> >>>>>> a null value in a config file. For example, if you set the following
> >>> in
> >>>> a
> >>>>>> config file.
> >>>>>> configkey=
> >>>>>> The value of configkey will be an empty list, instead of null,
> right?
> >>>>>>
> >>>>>> Null could make sense as the default value, but it just means that
> the
> >>>>>> config key is not set explicitly.
> >>>>>>
> >>>>>> JR11. There is an existing config interceptor.classes that allows an
> >>>> empty
> >>>>>> list and it makes intuitive sense. So, it seems that it's ok to
> >>> support
> >>>>>> empty lists in general. Given that, it's probably simpler to just
> >>> allow
> >>>>>> log.cleanup.policy to be empty and properly support it?
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hello Jun,
> >>>>>>>
> >>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional, that is
> >>>> right,
> >>>>>>> so I think
> >>>>>>> that we can allow null value, but not allow the empty
> >>>>>>>
> >>>>>>> JR11: In this KIP, the goal is to address configurations that
> should
> >>>> not
> >>>>>>> accept empty arrays.
> >>>>>>> We want to prevent users from passing in an empty list when it's
> not
> >>>>>>> valid. I believe there are two
> >>>>>>> main scenarios to consider:
> >>>>>>>
> >>>>>>> 1. Nullable but not empty:
> >>>>>>> The parameter allows either a null value or a non-empty list. In
> this
> >>>>>>> case,
> >>>>>>> if the user provides an empty list, we should throw a
> >>> ConfigException.
> >>>>>>>
> >>>>>>> 2. Non-null and non-empty only: The parameter must always contain a
> >>>>>>> non-empty list — null is also
> >>>>>>> invalid. In this case, if the user provides either a null or an
> empty
> >>>>>>> list, we should throw a ConfigException.
> >>>>>>>
> >>>>>>> Therefore, I want to prevent scenarios where users can pass in an
> >>> empty
> >>>>>>> list.
> >>>>>>>
> >>>>>>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
> >>>>>> property.
> >>>>>>> This allows us to specify
> >>>>>>> whether a configuration permits null values or not.
> >>>>>>>
> >>>>>>> JR13: I have updated the table
> >>>>>>>
> >>>>>>>
> >>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> >>>>>>>>
> >>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> Sorry for the late reply. A few more comments.
> >>>>>>>>
> >>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says that
> it's
> >>>>>>>> optional. So an empty list seems to be valid.
> >>>>>>>>
> >>>>>>>> JR11. Thinking a bit more. If we are going to support empty lists
> in
> >>>>>>>> general, it's probably more consistent to just allow
> >>>> log.cleanup.policy
> >>>>>>> to
> >>>>>>>> be empty as Luke suggested, instead of introducing a new option
> >>> None.
> >>>>>>>>
> >>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It seems
> >>> that
> >>>>>>>> only ssl.cipher.suites allows null. The remaining ones shouldn't
> be
> >>>>>> null.
> >>>>>>>>
> >>>>>>>> JR13. The table is very helpful. Could you also describe the
> old/new
> >>>>>>>> behavior for each config with an empty list or duplicate entry?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hello all,
> >>>>>>>>>
> >>>>>>>>> I am manually bumping this thread. Any feedback would be
> >>> appreciated.
> >>>>>>>>>
> >>>>>>>>> Best regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> >>>>>>>>>>
> >>>>>>>>>> Hello Jun,
> >>>>>>>>>>
> >>>>>>>>>> I have updated the KIP and introduced a new validator,
> >>>>>>>>> NonEmptyListValidator, which ensures that the
> >>>>>>>>>> provided list is either null or a non-empty list without
> duplicate
> >>>>>>>>> entries. I’ve also listed the configurations that will
> >>>>>>>>>> be validated using this logic. Please feel free to share any
> >>>> feedback
> >>>>>>> or
> >>>>>>>>> suggestions.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>
> >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> >>>>>>>>>>>
> >>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>
> >>>>>>>>>>> It seems there are quite a few other configs of type list (e.g.
> >>>>>>> several
> >>>>>>>>> SSL
> >>>>>>>>>>> related ones). It would be useful to understand if empty lists
> >>> are
> >>>>>>> valid
> >>>>>>>>>>> for them or not.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io>
> >>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>
> >>>>>>>>>>>> "My thought is that this behavior has been existed
> >>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
> there
> >>>>>> could
> >>>>>>>>> be
> >>>>>>>>>>>> users relying on the empty cleanup.policy for a long time. "
> >>>>>>>>>>>>
> >>>>>>>>>>>> If you do a google search on "kafka infinite retention", you
> >>> will
> >>>>>> get
> >>>>>>>>>>>> links like
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> >>>>>>>>>>>> and
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> >>>>>>>>> ,
> >>>>>>>>>>>> both pointing to setting the retention time and retention size
> >>> to
> >>>>>> -1
> >>>>>>> to
> >>>>>>>>>>>> achieve this goal. So, it seems that if a user intentionally
> >>> wants
> >>>>>>>>> infinite
> >>>>>>>>>>>> retention, it's more likely they will use that approach
> instead
> >>> of
> >>>>>>>>> setting
> >>>>>>>>>>>> cleanup.policy to empty. On the other hand, our API/tools make
> >>> it
> >>>>>>>>> possible
> >>>>>>>>>>>> for users to make a mistake by inadvertently leaving the
> config
> >>>>>> value
> >>>>>>>>> empty.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <showuon@gmail.com
> >
> >>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regarding the motivation, currently, we never documented
> that
> >>> an
> >>>>>>>>> empty
> >>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> >>> leaves
> >>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage since
> >>> it
> >>>>>>> will
> >>>>>>>>> stop
> >>>>>>>>>>>>> the cleaning based on local retention size and time too. So,
> >>>>>> leaving
> >>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
> than
> >>> an
> >>>>>>>>>>>>> intentional choice. Based on this assumption, the idea behind
> >>> the
> >>>>>>> KIP
> >>>>>>>>> is
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
> this
> >>>>>>> likely
> >>>>>>>>>>>>> mistake and provide a new option None for users wanting
> >>> infinite
> >>>>>>>>> retention
> >>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> While we never documented the empty cleanup.policy implies
> >>>>>> infinite
> >>>>>>>>>>>>> retention, we also never documented (or failed) the empty
> >>>>>>>>> cleanup.policy
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>> an invalid configuration. My thought is that this behavior
> has
> >>>>>> been
> >>>>>>>>>>>>> existed
> >>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?),
> there
> >>>>>>> could
> >>>>>>>>> be
> >>>>>>>>>>>>> users relying on the empty cleanup.policy for a long time. It
> >>> is
> >>>>>> not
> >>>>>>>>> good
> >>>>>>>>>>>>> to break the backward compatibility in a minor release
> version
> >>>>>> (ex:
> >>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long existing
> >>> bug,
> >>>>>> it
> >>>>>>>>> could
> >>>>>>>>>>>>> be a surprise to users.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> >>>>>> fixing
> >>>>>>>>> the
> >>>>>>>>>>>>> issue in remote storage. But then the user may never realize
> >>> this
> >>>>>> if
> >>>>>>>>> it's
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Good point! I agree we need to fix it!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and in
> >>> 5.0
> >>>>>> we
> >>>>>>>>>>>>> throw
> >>>>>>>>>>>>> exception directly to make it fail fast
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This suggestion looks good to me!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you.
> >>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> >>>>>> chia7712@apache.org
> >>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> hi all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is worth
> >>>>>>>>> considering
> >>>>>>>>>>>>>> the compatibility issue. In fact, I had previously
> encountered
> >>>>>> this
> >>>>>>>>> use
> >>>>>>>>>>>>>> case but advised customers to utilize retention
> configurations
> >>>>>>>>> instead.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
> by
> >>>>>>> fixing
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> realize
> >>>>>> this
> >>>>>>> if
> >>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for
> >>> remote
> >>>>>>>>>>>>> storage.
> >>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage
> >>> use
> >>>>>> a
> >>>>>>>>> valid
> >>>>>>>>>>>>>> delete policy.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> >>>>>>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regarding the motivation, currently, we never documented
> that
> >>>> an
> >>>>>>>>> empty
> >>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> >>>>>> leaves
> >>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage
> since
> >>>> it
> >>>>>>>>> will
> >>>>>>>>>>>>>> stop
> >>>>>>>>>>>>>>> the cleaning based on local retention size and time too.
> So,
> >>>>>>> leaving
> >>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
> >>> than
> >>>>>> an
> >>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea
> behind
> >>>>>> the
> >>>>>>>>> KIP
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
> >>> this
> >>>>>>>>> likely
> >>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
> >>>> infinite
> >>>>>>>>>>>>>> retention
> >>>>>>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy
> by
> >>>>>>> fixing
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> issue in remote storage. But then the user may never
> realize
> >>>>>> this
> >>>>>>> if
> >>>>>>>>>>>>>> it's a
> >>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <
> showuon@gmail.com
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Comments:
> >>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
> >>>>>>> considered
> >>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> >>>>>>>>>>>>>>>> Could you explain more about this? Why is this trade-off
> >>>>>>> justified?
> >>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in kafka,
> >>> why
> >>>>>> do
> >>>>>>> we
> >>>>>>>>>>>>>> provide
> >>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
> >>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
> >>> history
> >>>>>>> data
> >>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>> a cleanup policy.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
> >>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and
> "process.roles"
> >>>>>>> configs
> >>>>>>>>>>>>>> since
> >>>>>>>>>>>>>>>> they are invalid.
> >>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive
> reason
> >>>> why
> >>>>>>> we
> >>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>> break backward compatibility to change it.
> >>>>>>>>>>>>>>>> "This provides a clear and direct way to configure Kafka
> to
> >>>>>>> retain
> >>>>>>>>>>>>> all
> >>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>> segments without any automatic cleanup."
> >>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to me,
> >>>>>>> backward
> >>>>>>>>>>>>>>>> compatibility is much more important than "a clear and
> >>> direct
> >>>>>> way
> >>>>>>>>> to
> >>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>> kafka".
> >>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy" config?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>>> Luke
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> >>>>>> <jun@confluent.io.invalid
> >>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Jun, chia
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> addressed
> >>> in
> >>>>>>> the
> >>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
> >>> Additionally,
> >>>>>> an
> >>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>> will now be thrown if a
> >>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
> >>>> inNonEmpty
> >>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日
> 凌晨12:46
> >>>> 寫道：
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty
> >>>> config,
> >>>>>>>>>>>>> so
> >>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> be non-empty too?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
> >>>> directly.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
> >>>>>> 上午12:28寫道：
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references to
> the
> >>>>>>>>>>>>>> deprecated
> >>>>>>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> >>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in
> Kafka
> >>>>>> 5.0"
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <
> s7133700@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> >>> addressed
> >>>>>> in
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日
> >>> 凌晨2:03
> >>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration
> >>> Plan",
> >>>>>>>>>>>>>> could you
> >>>>>>>>>>>>>>>>> add
> >>>>>>>>>>>>>>>>>>>>>> description to remind readers that "clean.policy="
> >>>> should
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> replaced
> >>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable
> delete
> >>>> and
> >>>>>>>>>>>>>> compact.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
> >>>>>>>>>>>>> 上午1:55寫道：
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now.
> >>> Could
> >>>>>> you
> >>>>>>>>>>>>>> update
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
> >>> deprecation.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> >>> s7133700@gmail.com
> >>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
> >>>>>>>>>>>>> method,
> >>>>>>>>>>>>>> as
> >>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
> >>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to
> >>> have
> >>>>>>>>>>>>> both
> >>>>>>>>>>>>>>>>> methods:
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> >>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> >>> document
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail
> to
> >>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
> >>>>>> 凌晨12:50
> >>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> >>> document
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail
> to
> >>>>>>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> >>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
> >>> maintain
> >>>>>>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed
> flag
> >>> is
> >>>>>>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to
> >>>> remove
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> boolean
> >>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日
> >>>>>> 凌晨4:17
> >>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward
> >>>> since
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> method
> >>>>>>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set
> to
> >>>>>>>>>>>>> true.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> >>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce
> new
> >>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
> >>>>>>>>>>>>> explicitly
> >>>>>>>>>>>>>>>>> throw a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
> >>>>>> validation
> >>>>>>>>>>>>>> phase
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>> 2025年4月25日
> >>>>>>>>>>>>>> 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by
> >>>> this
> >>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during
> the
> >>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
> >>>>>>>>>>>>> already
> >>>>>>>>>>>>>> fails
> >>>>>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> >>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> >>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
> >>>>>> storage
> >>>>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list
> >>> will
> >>>>>>>>>>>>> now
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage
> >>>> format
> >>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>> 2025年4月24日
> >>>>>>>>>>>>>> 凌晨4:44
> >>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> >>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now.
> With
> >>>>>> this
> >>>>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more
> accurate)
> >>>>>>>>>>>>>> exception.
> >>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree
> >>> with
> >>>>>>>>>>>>>> Chia-Ping.
> >>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to
> be
> >>> a
> >>>>>>>>>>>>>> mistake
> >>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the
> >>> user
> >>>>>>>>>>>>> will
> >>>>>>>>>>>>>> never
> >>>>>>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the
> >>>> user
> >>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping
> >>> Tsai
> >>>> <
> >>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> >>>>>>>>>>>>>> maintaining
> >>>>>>>>>>>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation,
> we
> >>>>>> don't
> >>>>>>>>>>>>>> allow
> >>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
> >>> consider
> >>>>>>>>>>>>>> throwing
> >>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
> >>>>>> streamline
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> >>>>>> 2025年4月23日
> >>>>>>>>>>>>> 週三
> >>>>>>>>>>>>>>>>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not
> delete
> >>> or
> >>>>>>>>>>>>>> compact
> >>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>>>>>> segments",
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
> >>>>>>>>>>>>> segments
> >>>>>>>>>>>>>> based
> >>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms,
> but
> >>> we
> >>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>> continue
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> >>> log.local.retention.bytes/
> >>>>>>>>>>>>>>>>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when
> >>>>>> remote
> >>>>>>>>>>>>>>>> storage
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> >>>>>>>>>>>>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
> and
> >>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> >>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name
> accordingly
> >>>> and
> >>>>>>>>>>>>>> revised
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot
> be
> >>>>>> used
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> >>>>>>>>>>>>>> cleanup.policy,
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> >>>>>>>>>>>>>>>> Additionally, a
> >>>>>>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>>>>>>>>> 2025年4月22日
> >>>>>>>>>>>>>>>>> 凌晨4:52
> >>>>>>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value
> >>> None
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
> >>>>>>>>>>>>> example,
> >>>>>>>>>>>>>> None
> >>>>>>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful
> to
> >>>>>>>>>>>>>> document
> >>>>>>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade
> >>> when
> >>>>>> an
> >>>>>>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear
> what
> >>>>>> the
> >>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>> check
> >>>>>>>>>>>>>>>>>>>>>>> does.
> >>>>>>>>>>>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> >>>>>>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> >>>>>>>>>>>>> KIP-1161:
> >>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> >>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left
> empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> >>>>>>>>>>>>> improvements:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate
> whether
> >>>> an
> >>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>> list
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> >>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi Jun, Chia

>> Also, it would be useful to add a compatibility column in the table. If a
>> value is formally disallowed, it would be useful to compare the old and the
>> new behavior (e.g. a different exception is thrown, etc).
I’ve updated the table to show which exceptions will be thrown for each behavior.

>>> I don't understand the point of "5.0". Could you please share more details?
>>> It seems to me changing the type or default value should be fine in minor
>>> release if we don't break the existing property files.
I don't think this requires any compatibility considerations, so I’ve removed the Kafka 5.0 changes section.

>>> `max.connections.per.ip.overrides` is parsed as "map" type, so using `LIST`
>>> instead of `String` does not seems to make sense.
I have removed the config from KIP

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@apache.org> 於 2025年7月9日 晚上10:26 寫道：
> 
>> For ssl.cipher.suites, changing the default value from null to empty is
>> actually a breaking change. According to the doc, null means that all the
>> available cipher suites are supported. I am thinking that we should
>> continue to allow null as the default since it could represent a state
>> different from empty.
> 
> "Empty list" seems to be a weird case. It appears to signify that no cipher suites are accepted. However, in the codebase, an empty list is handled as if all available cipher suites are supported [0]. We should not support the case of "no supported cipher suite," as it doesn't make sense.
> 
> In short, changing the default value from null to empty does not break the behavior.
> 
> [0] https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139
> 
> On 2025/07/08 17:01:07 Jun Rao wrote:
>> Hi, Jiunn-Yang,
>> 
>> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we will
>> just continue to allow it in all future releases.
>> 
>> For ssl.cipher.suites, changing the default value from null to empty is
>> actually a breaking change. According to the doc, null means that all the
>> available cipher suites are supported. I am thinking that we should
>> continue to allow null as the default since it could represent a state
>> different from empty.
>> 
>> Also, it would be useful to add a compatibility column in the table. If a
>> value is formally disallowed, it would be useful to compare the old and the
>> new behavior (e.g. a different exception is thrown, etc).
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>> 
>>> hi Jiunn
>>> 
>>> Thanks for the explanation. I have updated the KIP accordingly: in Kafka
>>>> 4.x, cleanup.policy
>>>> can be set to an empty list with a warning message emitted; however, in
>>>> Kafka 5.0, setting an
>>>> empty list for cleanup.policy will no longer be allowed.
>>> 
>>> 
>>> If we agree to support "empty list", the approach of "none" could be
>>> eliminated, right?
>>> 
>>> I don't understand the point of "5.0". Could you please share more details?
>>> It seems to me changing the type or default value should be fine in minor
>>> release if we don't break the existing property files.
>>> 
>>> `max.connections.per.ip.overrides` is parsed as "map" type, so using `LIST`
>>> instead of `String` does not seems to make sense.
>>> 
>>> Best,
>>> Chia-Ping
>>> 
>>> 
>>> 
>>> 
>>> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
>>> 
>>>> Hi all,
>>>> 
>>>>>> Null could make sense as the default value, but it just means that the
>>>>>> config key is not set explicitly.
>>>>>> Given that, it's probably simpler to just allow
>>>>>> log.cleanup.policy to be empty and properly support it?
>>>> Thanks for the explanation. I have updated the KIP accordingly: in Kafka
>>>> 4.x, cleanup.policy
>>>> can be set to an empty list with a warning message emitted; however, in
>>>> Kafka 5.0, setting an
>>>> empty list for cleanup.policy will no longer be allowed.
>>>> 
>>>>> For another, there is an additional issue which could be handled by
>>> this
>>>>> KIP. Some configs have "string" type, but they are handled as a LIST.
>>> For
>>>>> example, `early.start.listeners`, `security.providers`. Could we change
>>>>> their type to LIST to benefit from this KIP? I mean to make them
>>>>> non-nullable.
>>>> I have also updated the KIP to include the tables detailing the changes
>>>> for LIST-type configuration
>>>> default values and the migration from STRING-type to LIST-type
>>>> configurations.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
>>>>> 
>>>>> hi all,
>>>>> 
>>>>> JR11. There is an existing config interceptor.classes that allows an
>>>> empty
>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
>>> support
>>>>>> empty lists in general. Given that, it's probably simpler to just
>>> allow
>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>> 
>>>>> 
>>>>> yes, that is what I meant before. Additionally, the `LIST` type should
>>>> not
>>>>> accept "null". Instead, we should use an empty list as the default
>>> value.
>>>>> For example, `sasl.oauthbearer.expected.audience` should never has
>>> "null"
>>>>> value. That could eliminate the null check.
>>>>> 
>>>>> For another, there is an additional issue which could be handled by
>>> this
>>>>> KIP. Some configs have "string" type, but they are handled as a LIST.
>>> For
>>>>> example, `early.start.listeners`, `security.providers`. Could we change
>>>>> their type to LIST to benefit from this KIP? I mean to make them
>>>>> non-nullable.
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> 
>>>>> 
>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply.
>>>>>> 
>>>>>> JR10. Regarding supporting the null value, I am not sure how one could
>>>> set
>>>>>> a null value in a config file. For example, if you set the following
>>> in
>>>> a
>>>>>> config file.
>>>>>> configkey=
>>>>>> The value of configkey will be an empty list, instead of null, right?
>>>>>> 
>>>>>> Null could make sense as the default value, but it just means that the
>>>>>> config key is not set explicitly.
>>>>>> 
>>>>>> JR11. There is an existing config interceptor.classes that allows an
>>>> empty
>>>>>> list and it makes intuitive sense. So, it seems that it's ok to
>>> support
>>>>>> empty lists in general. Given that, it's probably simpler to just
>>> allow
>>>>>> log.cleanup.policy to be empty and properly support it?
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hello Jun,
>>>>>>> 
>>>>>>> JR10: The sasl.oauthbearer.expected.audience is optional, that is
>>>> right,
>>>>>>> so I think
>>>>>>> that we can allow null value, but not allow the empty
>>>>>>> 
>>>>>>> JR11: In this KIP, the goal is to address configurations that should
>>>> not
>>>>>>> accept empty arrays.
>>>>>>> We want to prevent users from passing in an empty list when it's not
>>>>>>> valid. I believe there are two
>>>>>>> main scenarios to consider:
>>>>>>> 
>>>>>>> 1. Nullable but not empty:
>>>>>>> The parameter allows either a null value or a non-empty list. In this
>>>>>>> case,
>>>>>>> if the user provides an empty list, we should throw a
>>> ConfigException.
>>>>>>> 
>>>>>>> 2. Non-null and non-empty only: The parameter must always contain a
>>>>>>> non-empty list — null is also
>>>>>>> invalid. In this case, if the user provides either a null or an empty
>>>>>>> list, we should throw a ConfigException.
>>>>>>> 
>>>>>>> Therefore, I want to prevent scenarios where users can pass in an
>>> empty
>>>>>>> list.
>>>>>>> 
>>>>>>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
>>>>>> property.
>>>>>>> This allows us to specify
>>>>>>> whether a configuration permits null values or not.
>>>>>>> 
>>>>>>> JR13: I have updated the table
>>>>>>> 
>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
>>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Sorry for the late reply. A few more comments.
>>>>>>>> 
>>>>>>>> JR10. The doc for sasl.oauthbearer.expected.audience says that it's
>>>>>>>> optional. So an empty list seems to be valid.
>>>>>>>> 
>>>>>>>> JR11. Thinking a bit more. If we are going to support empty lists in
>>>>>>>> general, it's probably more consistent to just allow
>>>> log.cleanup.policy
>>>>>>> to
>>>>>>>> be empty as Luke suggested, instead of introducing a new option
>>> None.
>>>>>>>> 
>>>>>>>> JR12. NonEmptyListValidator allows the input to be null. It seems
>>> that
>>>>>>>> only ssl.cipher.suites allows null. The remaining ones shouldn't be
>>>>>> null.
>>>>>>>> 
>>>>>>>> JR13. The table is very helpful. Could you also describe the old/new
>>>>>>>> behavior for each config with an empty list or duplicate entry?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hello all,
>>>>>>>>> 
>>>>>>>>> I am manually bumping this thread. Any feedback would be
>>> appreciated.
>>>>>>>>> 
>>>>>>>>> Best regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
>>>>>>>>>> 
>>>>>>>>>> Hello Jun,
>>>>>>>>>> 
>>>>>>>>>> I have updated the KIP and introduced a new validator,
>>>>>>>>> NonEmptyListValidator, which ensures that the
>>>>>>>>>> provided list is either null or a non-empty list without duplicate
>>>>>>>>> entries. I’ve also listed the configurations that will
>>>>>>>>>> be validated using this logic. Please feel free to share any
>>>> feedback
>>>>>>> or
>>>>>>>>> suggestions.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> It seems there are quite a few other configs of type list (e.g.
>>>>>>> several
>>>>>>>>> SSL
>>>>>>>>>>> related ones). It would be useful to understand if empty lists
>>> are
>>>>>>> valid
>>>>>>>>>>> for them or not.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io>
>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>> 
>>>>>>>>>>>> "My thought is that this behavior has been existed
>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
>>>>>> could
>>>>>>>>> be
>>>>>>>>>>>> users relying on the empty cleanup.policy for a long time. "
>>>>>>>>>>>> 
>>>>>>>>>>>> If you do a google search on "kafka infinite retention", you
>>> will
>>>>>> get
>>>>>>>>>>>> links like
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>>>>>>>>>>> and
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
>>>>>>>>> ,
>>>>>>>>>>>> both pointing to setting the retention time and retention size
>>> to
>>>>>> -1
>>>>>>> to
>>>>>>>>>>>> achieve this goal. So, it seems that if a user intentionally
>>> wants
>>>>>>>>> infinite
>>>>>>>>>>>> retention, it's more likely they will use that approach instead
>>> of
>>>>>>>>> setting
>>>>>>>>>>>> cleanup.policy to empty. On the other hand, our API/tools make
>>> it
>>>>>>>>> possible
>>>>>>>>>>>> for users to make a mistake by inadvertently leaving the config
>>>>>> value
>>>>>>>>> empty.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regarding the motivation, currently, we never documented that
>>> an
>>>>>>>>> empty
>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
>>> leaves
>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage since
>>> it
>>>>>>> will
>>>>>>>>> stop
>>>>>>>>>>>>> the cleaning based on local retention size and time too. So,
>>>>>> leaving
>>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake than
>>> an
>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea behind
>>> the
>>>>>>> KIP
>>>>>>>>> is
>>>>>>>>>>>>> to
>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of this
>>>>>>> likely
>>>>>>>>>>>>> mistake and provide a new option None for users wanting
>>> infinite
>>>>>>>>> retention
>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> While we never documented the empty cleanup.policy implies
>>>>>> infinite
>>>>>>>>>>>>> retention, we also never documented (or failed) the empty
>>>>>>>>> cleanup.policy
>>>>>>>>>>>>> is
>>>>>>>>>>>>> an invalid configuration. My thought is that this behavior has
>>>>>> been
>>>>>>>>>>>>> existed
>>>>>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
>>>>>>> could
>>>>>>>>> be
>>>>>>>>>>>>> users relying on the empty cleanup.policy for a long time. It
>>> is
>>>>>> not
>>>>>>>>> good
>>>>>>>>>>>>> to break the backward compatibility in a minor release version
>>>>>> (ex:
>>>>>>>>>>>>> v4.1.0). Even though we think we are fixing a long existing
>>> bug,
>>>>>> it
>>>>>>>>> could
>>>>>>>>>>>>> be a surprise to users.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
>>>>>> fixing
>>>>>>>>> the
>>>>>>>>>>>>> issue in remote storage. But then the user may never realize
>>> this
>>>>>> if
>>>>>>>>> it's
>>>>>>>>>>>>> a
>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Good point! I agree we need to fix it!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Chia-Ping,
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and in
>>> 5.0
>>>>>> we
>>>>>>>>>>>>> throw
>>>>>>>>>>>>> exception directly to make it fail fast
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This suggestion looks good to me!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>> Luke
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
>>>>>> chia7712@apache.org
>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> hi all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Given Luke mentioned the existence of use cases, it is worth
>>>>>>>>> considering
>>>>>>>>>>>>>> the compatibility issue. In fact, I had previously encountered
>>>>>> this
>>>>>>>>> use
>>>>>>>>>>>>>> case but advised customers to utilize retention configurations
>>>>>>>>> instead.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
>>>>>>> fixing
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> issue in remote storage. But then the user may never realize
>>>>>> this
>>>>>>> if
>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for
>>> remote
>>>>>>>>>>>>> storage.
>>>>>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage
>>> use
>>>>>> a
>>>>>>>>> valid
>>>>>>>>>>>>>> delete policy.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regarding the motivation, currently, we never documented that
>>>> an
>>>>>>>>> empty
>>>>>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
>>>>>> leaves
>>>>>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage since
>>>> it
>>>>>>>>> will
>>>>>>>>>>>>>> stop
>>>>>>>>>>>>>>> the cleaning based on local retention size and time too. So,
>>>>>>> leaving
>>>>>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
>>> than
>>>>>> an
>>>>>>>>>>>>>>> intentional choice. Based on this assumption, the idea behind
>>>>>> the
>>>>>>>>> KIP
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
>>> this
>>>>>>>>> likely
>>>>>>>>>>>>>>> mistake and provide a new option None for users wanting
>>>> infinite
>>>>>>>>>>>>>> retention
>>>>>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
>>>>>>> fixing
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> issue in remote storage. But then the user may never realize
>>>>>> this
>>>>>>> if
>>>>>>>>>>>>>> it's a
>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <showuon@gmail.com
>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Comments:
>>>>>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
>>>>>>> considered
>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
>>>>>>>>>>>>>>>> Could you explain more about this? Why is this trade-off
>>>>>>> justified?
>>>>>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in kafka,
>>> why
>>>>>> do
>>>>>>> we
>>>>>>>>>>>>>> provide
>>>>>>>>>>>>>>>> an additional "none" option to users in this KIP?
>>>>>>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
>>> history
>>>>>>> data
>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>> a cleanup policy.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
>>>>>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles"
>>>>>>> configs
>>>>>>>>>>>>>> since
>>>>>>>>>>>>>>>> they are invalid.
>>>>>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive reason
>>>> why
>>>>>>> we
>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> break backward compatibility to change it.
>>>>>>>>>>>>>>>> "This provides a clear and direct way to configure Kafka to
>>>>>>> retain
>>>>>>>>>>>>> all
>>>>>>>>>>>>>> log
>>>>>>>>>>>>>>>> segments without any automatic cleanup."
>>>>>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to me,
>>>>>>> backward
>>>>>>>>>>>>>>>> compatibility is much more important than "a clear and
>>> direct
>>>>>> way
>>>>>>>>> to
>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>> kafka".
>>>>>>>>>>>>>>>> Do we really have to change the "cleanup.policy" config?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>> Luke
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
>>>>>> <jun@confluent.io.invalid
>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Jun, chia
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed
>>> in
>>>>>>> the
>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I've removed all deprecated-related sections.
>>> Additionally,
>>>>>> an
>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>> will now be thrown if a
>>>>>>>>>>>>>>>>>> developer passes an empty validStrings list to the
>>>> inNonEmpty
>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46
>>>> 寫道：
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty
>>>> config,
>>>>>>>>>>>>> so
>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> be non-empty too?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
>>>> directly.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
>>>>>> 上午12:28寫道：
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the reply. There are still references to the
>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka
>>>>>> 5.0"
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
>>> addressed
>>>>>> in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日
>>> 凌晨2:03
>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration
>>> Plan",
>>>>>>>>>>>>>> could you
>>>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>>>>>> description to remind readers that "clean.policy="
>>>> should
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> replaced
>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete
>>>> and
>>>>>>>>>>>>>> compact.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
>>>>>>>>>>>>> 上午1:55寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now.
>>> Could
>>>>>> you
>>>>>>>>>>>>>> update
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>> completely? There are still references to
>>> deprecation.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
>>> s7133700@gmail.com
>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
>>>>>>>>>>>>> method,
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to
>>> have
>>>>>>>>>>>>> both
>>>>>>>>>>>>>>>>> methods:
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
>>> document
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
>>>>>> 凌晨12:50
>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
>>> document
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>>>>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
>>> maintain
>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag
>>> is
>>>>>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to
>>>> remove
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> boolean
>>>>>>>>>>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日
>>>>>> 凌晨4:17
>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward
>>>> since
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> method
>>>>>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
>>>>>>>>>>>>> true.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
>>>>>>>>>>>>> explicitly
>>>>>>>>>>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
>>>>>> validation
>>>>>>>>>>>>>> phase
>>>>>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>> 2025年4月25日
>>>>>>>>>>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by
>>>> this
>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
>>>>>>>>>>>>> already
>>>>>>>>>>>>>> fails
>>>>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
>>>>>> storage
>>>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list
>>> will
>>>>>>>>>>>>> now
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage
>>>> format
>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>> 2025年4月24日
>>>>>>>>>>>>>> 凌晨4:44
>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>>>>>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With
>>>>>> this
>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
>>>>>>>>>>>>>> exception.
>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree
>>> with
>>>>>>>>>>>>>> Chia-Ping.
>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be
>>> a
>>>>>>>>>>>>>> mistake
>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the
>>> user
>>>>>>>>>>>>> will
>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the
>>>> user
>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping
>>> Tsai
>>>> <
>>>>>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
>>>>>>>>>>>>>> maintaining
>>>>>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we
>>>>>> don't
>>>>>>>>>>>>>> allow
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
>>> consider
>>>>>>>>>>>>>> throwing
>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
>>>>>> streamline
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>>>>>> 2025年4月23日
>>>>>>>>>>>>> 週三
>>>>>>>>>>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete
>>> or
>>>>>>>>>>>>>> compact
>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
>>>>>>>>>>>>> segments
>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but
>>> we
>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
>>> log.local.retention.bytes/
>>>>>>>>>>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when
>>>>>> remote
>>>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>>>>>>>>>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly
>>>> and
>>>>>>>>>>>>>> revised
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be
>>>>>> used
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
>>>>>>>>>>>>>> cleanup.policy,
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
>>>>>>>>>>>>>>>> Additionally, a
>>>>>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>>>>>>> 2025年4月22日
>>>>>>>>>>>>>>>>> 凌晨4:52
>>>>>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value
>>> None
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
>>>>>>>>>>>>> example,
>>>>>>>>>>>>>> None
>>>>>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
>>>>>>>>>>>>>> document
>>>>>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade
>>> when
>>>>>> an
>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what
>>>>>> the
>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>>>>>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>>>>>>>>>>>>> KIP-1161:
>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
>>>>>>>>>>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether
>>>> an
>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@apache.org>]

> For ssl.cipher.suites, changing the default value from null to empty is
> actually a breaking change. According to the doc, null means that all the
> available cipher suites are supported. I am thinking that we should
> continue to allow null as the default since it could represent a state
> different from empty.

"Empty list" seems to be a weird case. It appears to signify that no cipher suites are accepted. However, in the codebase, an empty list is handled as if all available cipher suites are supported [0]. We should not support the case of "no supported cipher suite," as it doesn't make sense.

In short, changing the default value from null to empty does not break the behavior.

[0] https://github.com/apache/kafka/blob/dabde76ebf105aaa945db60b7753331c83a8c989/clients/src/main/java/org/apache/kafka/common/security/ssl/DefaultSslEngineFactory.java#L139

On 2025/07/08 17:01:07 Jun Rao wrote:
> Hi, Jiunn-Yang,
> 
> I agree with Chia-Ping. If we allow cleanup.policy to be empty, we will
> just continue to allow it in all future releases.
> 
> For ssl.cipher.suites, changing the default value from null to empty is
> actually a breaking change. According to the doc, null means that all the
> available cipher suites are supported. I am thinking that we should
> continue to allow null as the default since it could represent a state
> different from empty.
> 
> Also, it would be useful to add a compatibility column in the table. If a
> value is formally disallowed, it would be useful to compare the old and the
> new behavior (e.g. a different exception is thrown, etc).
> 
> Thanks,
> 
> Jun
> 
> On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
> 
> > hi Jiunn
> >
> > Thanks for the explanation. I have updated the KIP accordingly: in Kafka
> > > 4.x, cleanup.policy
> > > can be set to an empty list with a warning message emitted; however, in
> > > Kafka 5.0, setting an
> > > empty list for cleanup.policy will no longer be allowed.
> >
> >
> > If we agree to support "empty list", the approach of "none" could be
> > eliminated, right?
> >
> > I don't understand the point of "5.0". Could you please share more details?
> > It seems to me changing the type or default value should be fine in minor
> > release if we don't break the existing property files.
> >
> > `max.connections.per.ip.overrides` is parsed as "map" type, so using `LIST`
> > instead of `String` does not seems to make sense.
> >
> > Best,
> > Chia-Ping
> >
> >
> >
> >
> > 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
> >
> > > Hi all,
> > >
> > > >> Null could make sense as the default value, but it just means that the
> > > >> config key is not set explicitly.
> > > >> Given that, it's probably simpler to just allow
> > > >> log.cleanup.policy to be empty and properly support it?
> > > Thanks for the explanation. I have updated the KIP accordingly: in Kafka
> > > 4.x, cleanup.policy
> > > can be set to an empty list with a warning message emitted; however, in
> > > Kafka 5.0, setting an
> > > empty list for cleanup.policy will no longer be allowed.
> > >
> > > > For another, there is an additional issue which could be handled by
> > this
> > > > KIP. Some configs have "string" type, but they are handled as a LIST.
> > For
> > > > example, `early.start.listeners`, `security.providers`. Could we change
> > > > their type to LIST to benefit from this KIP? I mean to make them
> > > > non-nullable.
> > > I have also updated the KIP to include the tables detailing the changes
> > > for LIST-type configuration
> > > default values and the migration from STRING-type to LIST-type
> > > configurations.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> > > >
> > > > hi all,
> > > >
> > > > JR11. There is an existing config interceptor.classes that allows an
> > > empty
> > > >> list and it makes intuitive sense. So, it seems that it's ok to
> > support
> > > >> empty lists in general. Given that, it's probably simpler to just
> > allow
> > > >> log.cleanup.policy to be empty and properly support it?
> > > >
> > > >
> > > > yes, that is what I meant before. Additionally, the `LIST` type should
> > > not
> > > > accept "null". Instead, we should use an empty list as the default
> > value.
> > > > For example, `sasl.oauthbearer.expected.audience` should never has
> > "null"
> > > > value. That could eliminate the null check.
> > > >
> > > > For another, there is an additional issue which could be handled by
> > this
> > > > KIP. Some configs have "string" type, but they are handled as a LIST.
> > For
> > > > example, `early.start.listeners`, `security.providers`. Could we change
> > > > their type to LIST to benefit from this KIP? I mean to make them
> > > > non-nullable.
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > >
> > > >
> > > > Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> > > >
> > > >> Hi, Jiunn-Yang,
> > > >>
> > > >> Thanks for the reply.
> > > >>
> > > >> JR10. Regarding supporting the null value, I am not sure how one could
> > > set
> > > >> a null value in a config file. For example, if you set the following
> > in
> > > a
> > > >> config file.
> > > >>  configkey=
> > > >> The value of configkey will be an empty list, instead of null, right?
> > > >>
> > > >> Null could make sense as the default value, but it just means that the
> > > >> config key is not set explicitly.
> > > >>
> > > >> JR11. There is an existing config interceptor.classes that allows an
> > > empty
> > > >> list and it makes intuitive sense. So, it seems that it's ok to
> > support
> > > >> empty lists in general. Given that, it's probably simpler to just
> > allow
> > > >> log.cleanup.policy to be empty and properly support it?
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>
> > > >>> Hello Jun,
> > > >>>
> > > >>> JR10: The sasl.oauthbearer.expected.audience is optional, that is
> > > right,
> > > >>> so I think
> > > >>> that we can allow null value, but not allow the empty
> > > >>>
> > > >>> JR11: In this KIP, the goal is to address configurations that should
> > > not
> > > >>> accept empty arrays.
> > > >>> We want to prevent users from passing in an empty list when it's not
> > > >>> valid. I believe there are two
> > > >>> main scenarios to consider:
> > > >>>
> > > >>> 1. Nullable but not empty:
> > > >>> The parameter allows either a null value or a non-empty list. In this
> > > >>> case,
> > > >>> if the user provides an empty list, we should throw a
> > ConfigException.
> > > >>>
> > > >>> 2. Non-null and non-empty only: The parameter must always contain a
> > > >>> non-empty list — null is also
> > > >>> invalid. In this case, if the user provides either a null or an empty
> > > >>> list, we should throw a ConfigException.
> > > >>>
> > > >>> Therefore, I want to prevent scenarios where users can pass in an
> > empty
> > > >>> list.
> > > >>>
> > > >>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
> > > >> property.
> > > >>> This allows us to specify
> > > >>> whether a configuration permits null values or not.
> > > >>>
> > > >>> JR13: I have updated the table
> > > >>>
> > > >>>
> > > >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> > > >>>>
> > > >>>> Hi, Jiunn-Yang,
> > > >>>>
> > > >>>> Sorry for the late reply. A few more comments.
> > > >>>>
> > > >>>> JR10. The doc for sasl.oauthbearer.expected.audience says that it's
> > > >>>> optional. So an empty list seems to be valid.
> > > >>>>
> > > >>>> JR11. Thinking a bit more. If we are going to support empty lists in
> > > >>>> general, it's probably more consistent to just allow
> > > log.cleanup.policy
> > > >>> to
> > > >>>> be empty as Luke suggested, instead of introducing a new option
> > None.
> > > >>>>
> > > >>>> JR12. NonEmptyListValidator allows the input to be null. It seems
> > that
> > > >>>> only ssl.cipher.suites allows null. The remaining ones shouldn't be
> > > >> null.
> > > >>>>
> > > >>>> JR13. The table is very helpful. Could you also describe the old/new
> > > >>>> behavior for each config with an empty list or duplicate entry?
> > > >>>>
> > > >>>> Thanks,
> > > >>>>
> > > >>>> Jun
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>>>
> > > >>>>> Hello all,
> > > >>>>>
> > > >>>>> I am manually bumping this thread. Any feedback would be
> > appreciated.
> > > >>>>>
> > > >>>>> Best regards,
> > > >>>>> Jiunn-Yang
> > > >>>>>
> > > >>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> > > >>>>>>
> > > >>>>>> Hello Jun,
> > > >>>>>>
> > > >>>>>> I have updated the KIP and introduced a new validator,
> > > >>>>> NonEmptyListValidator, which ensures that the
> > > >>>>>> provided list is either null or a non-empty list without duplicate
> > > >>>>> entries. I’ve also listed the configurations that will
> > > >>>>>> be validated using this logic. Please feel free to share any
> > > feedback
> > > >>> or
> > > >>>>> suggestions.
> > > >>>>>>
> > > >>>>>> Best Regards,
> > > >>>>>> Jiunn-Yang
> > > >>>>>>
> > > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> > > >>>>>>>
> > > >>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>
> > > >>>>>>> It seems there are quite a few other configs of type list (e.g.
> > > >>> several
> > > >>>>> SSL
> > > >>>>>>> related ones). It would be useful to understand if empty lists
> > are
> > > >>> valid
> > > >>>>>>> for them or not.
> > > >>>>>>>
> > > >>>>>>> Thanks,
> > > >>>>>>>
> > > >>>>>>> Jun
> > > >>>>>>>
> > > >>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io>
> > wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi, Luke,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the reply.
> > > >>>>>>>>
> > > >>>>>>>> "My thought is that this behavior has been existed
> > > >>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
> > > >> could
> > > >>>>> be
> > > >>>>>>>> users relying on the empty cleanup.policy for a long time. "
> > > >>>>>>>>
> > > >>>>>>>> If you do a google search on "kafka infinite retention", you
> > will
> > > >> get
> > > >>>>>>>> links like
> > > >>>>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> > https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> > > >>>>>>>> and
> > > >>>>>>>>
> > > >>>>>
> > > >>>
> > > >>
> > >
> > https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> > > >>>>> ,
> > > >>>>>>>> both pointing to setting the retention time and retention size
> > to
> > > >> -1
> > > >>> to
> > > >>>>>>>> achieve this goal. So, it seems that if a user intentionally
> > wants
> > > >>>>> infinite
> > > >>>>>>>> retention, it's more likely they will use that approach instead
> > of
> > > >>>>> setting
> > > >>>>>>>> cleanup.policy to empty. On the other hand, our API/tools make
> > it
> > > >>>>> possible
> > > >>>>>>>> for users to make a mistake by inadvertently leaving the config
> > > >> value
> > > >>>>> empty.
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>>
> > > >>>>>>>> Jun
> > > >>>>>>>>
> > > >>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com>
> > > >>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi Jun,
> > > >>>>>>>>>
> > > >>>>>>>>>> Regarding the motivation, currently, we never documented that
> > an
> > > >>>>> empty
> > > >>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> > leaves
> > > >>>>>>>>> cleanup.policy empty, it actually breaks remote storage since
> > it
> > > >>> will
> > > >>>>> stop
> > > >>>>>>>>> the cleaning based on local retention size and time too. So,
> > > >> leaving
> > > >>>>>>>>> cleanup.policy empty is probably more like a user mistake than
> > an
> > > >>>>>>>>> intentional choice. Based on this assumption, the idea behind
> > the
> > > >>> KIP
> > > >>>>> is
> > > >>>>>>>>> to
> > > >>>>>>>>> fail an empty cleanup.policy so that the user is aware of this
> > > >>> likely
> > > >>>>>>>>> mistake and provide a new option None for users wanting
> > infinite
> > > >>>>> retention
> > > >>>>>>>>> to opt in explicitly.
> > > >>>>>>>>>
> > > >>>>>>>>> While we never documented the empty cleanup.policy implies
> > > >> infinite
> > > >>>>>>>>> retention, we also never documented (or failed) the empty
> > > >>>>> cleanup.policy
> > > >>>>>>>>> is
> > > >>>>>>>>> an invalid configuration. My thought is that this behavior has
> > > >> been
> > > >>>>>>>>> existed
> > > >>>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
> > > >>> could
> > > >>>>> be
> > > >>>>>>>>> users relying on the empty cleanup.policy for a long time. It
> > is
> > > >> not
> > > >>>>> good
> > > >>>>>>>>> to break the backward compatibility in a minor release version
> > > >> (ex:
> > > >>>>>>>>> v4.1.0). Even though we think we are fixing a long existing
> > bug,
> > > >> it
> > > >>>>> could
> > > >>>>>>>>> be a surprise to users.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> > > >> fixing
> > > >>>>> the
> > > >>>>>>>>> issue in remote storage. But then the user may never realize
> > this
> > > >> if
> > > >>>>> it's
> > > >>>>>>>>> a
> > > >>>>>>>>> mistake.
> > > >>>>>>>>>
> > > >>>>>>>>> Good point! I agree we need to fix it!
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Hi Chia-Ping,
> > > >>>>>>>>>
> > > >>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and in
> > 5.0
> > > >> we
> > > >>>>>>>>> throw
> > > >>>>>>>>> exception directly to make it fail fast
> > > >>>>>>>>>
> > > >>>>>>>>> This suggestion looks good to me!
> > > >>>>>>>>>
> > > >>>>>>>>> Thank you.
> > > >>>>>>>>> Luke
> > > >>>>>>>>>
> > > >>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> > > >> chia7712@apache.org
> > > >>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> hi all,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Given Luke mentioned the existence of use cases, it is worth
> > > >>>>> considering
> > > >>>>>>>>>> the compatibility issue. In fact, I had previously encountered
> > > >> this
> > > >>>>> use
> > > >>>>>>>>>> case but advised customers to utilize retention configurations
> > > >>>>> instead.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> > > >>> fixing
> > > >>>>>>>>> the
> > > >>>>>>>>>>> issue in remote storage. But then the user may never realize
> > > >> this
> > > >>> if
> > > >>>>>>>>>> it's a
> > > >>>>>>>>>>> mistake.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for
> > remote
> > > >>>>>>>>> storage.
> > > >>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage
> > use
> > > >> a
> > > >>>>> valid
> > > >>>>>>>>>> delete policy.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best,
> > > >>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> > > >>>>>>>>>>> Hi, Luke,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Regarding the motivation, currently, we never documented that
> > > an
> > > >>>>> empty
> > > >>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> > > >> leaves
> > > >>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage since
> > > it
> > > >>>>> will
> > > >>>>>>>>>> stop
> > > >>>>>>>>>>> the cleaning based on local retention size and time too. So,
> > > >>> leaving
> > > >>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
> > than
> > > >> an
> > > >>>>>>>>>>> intentional choice. Based on this assumption, the idea behind
> > > >> the
> > > >>>>> KIP
> > > >>>>>>>>> is
> > > >>>>>>>>>> to
> > > >>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
> > this
> > > >>>>> likely
> > > >>>>>>>>>>> mistake and provide a new option None for users wanting
> > > infinite
> > > >>>>>>>>>> retention
> > > >>>>>>>>>>> to opt in explicitly.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> > > >>> fixing
> > > >>>>>>>>> the
> > > >>>>>>>>>>> issue in remote storage. But then the user may never realize
> > > >> this
> > > >>> if
> > > >>>>>>>>>> it's a
> > > >>>>>>>>>>> mistake.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Jun
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <showuon@gmail.com
> > >
> > > >>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi Jiunn-Yang,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for the KIP.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Comments:
> > > >>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
> > > >>> considered
> > > >>>>>>>>>> invalid
> > > >>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> > > >>>>>>>>>>>> Could you explain more about this? Why is this trade-off
> > > >>> justified?
> > > >>>>>>>>>>>> If we think the empty cleanup.policy is invalid in kafka,
> > why
> > > >> do
> > > >>> we
> > > >>>>>>>>>> provide
> > > >>>>>>>>>>>> an additional "none" option to users in this KIP?
> > > >>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
> > history
> > > >>> data
> > > >>>>>>>>>> without
> > > >>>>>>>>>>>> a cleanup policy.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
> > > >>>>>>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles"
> > > >>> configs
> > > >>>>>>>>>> since
> > > >>>>>>>>>>>> they are invalid.
> > > >>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive reason
> > > why
> > > >>> we
> > > >>>>>>>>>> should
> > > >>>>>>>>>>>> break backward compatibility to change it.
> > > >>>>>>>>>>>> "This provides a clear and direct way to configure Kafka to
> > > >>> retain
> > > >>>>>>>>> all
> > > >>>>>>>>>> log
> > > >>>>>>>>>>>> segments without any automatic cleanup."
> > > >>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to me,
> > > >>> backward
> > > >>>>>>>>>>>> compatibility is much more important than "a clear and
> > direct
> > > >> way
> > > >>>>> to
> > > >>>>>>>>>> config
> > > >>>>>>>>>>>> kafka".
> > > >>>>>>>>>>>> Do we really have to change the "cleanup.policy" config?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks.
> > > >>>>>>>>>>>> Luke
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> > > >> <jun@confluent.io.invalid
> > > >>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com>
> > > >> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi Jun, chia
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed
> > in
> > > >>> the
> > > >>>>>>>>>>>> updated
> > > >>>>>>>>>>>>>> KIP.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> I've removed all deprecated-related sections.
> > Additionally,
> > > >> an
> > > >>>>>>>>>>>> exception
> > > >>>>>>>>>>>>>> will now be thrown if a
> > > >>>>>>>>>>>>>> developer passes an empty validStrings list to the
> > > inNonEmpty
> > > >>>>>>>>>> method.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46
> > > 寫道：
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> hi Jiunn-Yang
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty
> > > config,
> > > >>>>>>>>> so
> > > >>>>>>>>>>>> should
> > > >>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>> be non-empty too?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
> > > directly.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
> > > >> 上午12:28寫道：
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks for the reply. There are still references to the
> > > >>>>>>>>>> deprecated
> > > >>>>>>>>>>>>>> method.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> > > >>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka
> > > >> 5.0"
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi Jun, chia,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> > addressed
> > > >> in
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> updated
> > > >>>>>>>>>>>>>>>>> KIP.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日
> > 凌晨2:03
> > > >> 寫道：
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration
> > Plan",
> > > >>>>>>>>>> could you
> > > >>>>>>>>>>>>> add
> > > >>>>>>>>>>>>>>>>>> description to remind readers that "clean.policy="
> > > should
> > > >>>>>>>>> be
> > > >>>>>>>>>>>>> replaced
> > > >>>>>>>>>>>>>>>> by
> > > >>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete
> > > and
> > > >>>>>>>>>> compact.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
> > > >>>>>>>>> 上午1:55寫道：
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now.
> > Could
> > > >> you
> > > >>>>>>>>>> update
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> KIP
> > > >>>>>>>>>>>>>>>>>>> completely? There are still references to
> > deprecation.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> > s7133700@gmail.com
> > > >
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hi Jun,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
> > > >>>>>>>>> method,
> > > >>>>>>>>>> as
> > > >>>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>>> may
> > > >>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>> future scenarios where we want
> > > >>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to
> > have
> > > >>>>>>>>> both
> > > >>>>>>>>>>>>> methods:
> > > >>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> > > >>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> > document
> > > >>>>>>>>> that
> > > >>>>>>>>>>>> during
> > > >>>>>>>>>>>>>>>>>>> upgrade,
> > > >>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> > > >>>>>>>>> start.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
> > > >> 凌晨12:50
> > > >>>>>>>>> 寫道：
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> > document
> > > >>>>>>>>> that
> > > >>>>>>>>>>>> during
> > > >>>>>>>>>>>>>>>>>>> upgrade,
> > > >>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> > > >>>>>>>>> start.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> > > >> s7133700@gmail.com>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
> > maintain
> > > >>>>>>>>>> backward
> > > >>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag
> > is
> > > >>>>>>>>>>>> necessary.
> > > >>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to
> > > remove
> > > >>>>>>>>> the
> > > >>>>>>>>>>>>> boolean
> > > >>>>>>>>>>>>>>>>>>>>>> parameter.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日
> > > >> 凌晨4:17
> > > >>>>>>>>>> 寫道：
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward
> > > since
> > > >>>>>>>>> the
> > > >>>>>>>>>>>> method
> > > >>>>>>>>>>>>>>>> name
> > > >>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
> > > >>>>>>>>> true.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> > > >>>>>>>>> s7133700@gmail.com>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
> > > >>>>>>>>> invalid
> > > >>>>>>>>>>>>>>>>>>>> configurations
> > > >>>>>>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> > > >>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
> > > >>>>>>>>> explicitly
> > > >>>>>>>>>>>>> throw a
> > > >>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
> > > >> validation
> > > >>>>>>>>>> phase
> > > >>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > 2025年4月25日
> > > >>>>>>>>>> 凌晨1:17 寫道：
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by
> > > this
> > > >>>>>>>>>> change
> > > >>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
> > > >>>>>>>>> storage
> > > >>>>>>>>>>>> format
> > > >>>>>>>>>>>>>>>>>>>>>> validation
> > > >>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
> > > >>>>>>>>> already
> > > >>>>>>>>>> fails
> > > >>>>>>>>>>>>> if
> > > >>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > > >>>>>>>>> process.roles
> > > >>>>>>>>>>>> are
> > > >>>>>>>>>>>>>>>>> empty.
> > > >>>>>>>>>>>>>>>>>>>> It
> > > >>>>>>>>>>>>>>>>>>>>>>>> just
> > > >>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> > > >>>>>>>>>> s7133700@gmail.com>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> > > >>>>>>>>>> group.coordinator.rebalance.protocols and
> > > >>>>>>>>>>>>>>>>>>>>>> process.roles
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> will be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
> > > >> storage
> > > >>>>>>>>>> format
> > > >>>>>>>>>>>>>>>> phase.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list
> > will
> > > >>>>>>>>> now
> > > >>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> considered
> > > >>>>>>>>>>>>>>>>>>>>>>>> invalid
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> and will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage
> > > format
> > > >>>>>>>>>> phase.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > > 2025年4月24日
> > > >>>>>>>>>> 凌晨4:44
> > > >>>>>>>>>>>> 寫道：
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> > > >>>>>>>>>> group.coordinator.rebalance.protocols
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With
> > > >> this
> > > >>>>>>>>>> KIP,
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> user
> > > >>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
> > > >>>>>>>>>> exception.
> > > >>>>>>>>>>>> It
> > > >>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>> useful
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree
> > with
> > > >>>>>>>>>> Chia-Ping.
> > > >>>>>>>>>>>>> If
> > > >>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>> user
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be
> > a
> > > >>>>>>>>>> mistake
> > > >>>>>>>>>>>> than
> > > >>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>> intention.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the
> > user
> > > >>>>>>>>> will
> > > >>>>>>>>>> never
> > > >>>>>>>>>>>>>>>>> realize
> > > >>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the
> > > user
> > > >>>>>>>>>> realize
> > > >>>>>>>>>>>>>> there
> > > >>>>>>>>>>>>>>>>>>> is a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping
> > Tsai
> > > <
> > > >>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> > > >>>>>>>>>> maintaining
> > > >>>>>>>>>>>>>>>>>>>> compatibility
> > > >>>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we
> > > >> don't
> > > >>>>>>>>>> allow
> > > >>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>> empty
> > > >>>>>>>>>>>>>>>>>>>>>> value
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
> > consider
> > > >>>>>>>>>> throwing
> > > >>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>> exception
> > > >>>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
> > > >> streamline
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>> code
> > > >>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>> avoid
> > > >>>>>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> > > >> 2025年4月23日
> > > >>>>>>>>> 週三
> > > >>>>>>>>>>>>>> 上午6:33寫道：
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete
> > or
> > > >>>>>>>>>> compact
> > > >>>>>>>>>>>> any
> > > >>>>>>>>>>>>>>>>>>>>>> segments",
> > > >>>>>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
> > > >>>>>>>>> segments
> > > >>>>>>>>>> based
> > > >>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but
> > we
> > > >>>>>>>>>> should
> > > >>>>>>>>>>>>>>>> continue
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>> delete
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> > log.local.retention.bytes/
> > > >>>>>>>>>>>>>>>> log.retention.ms.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when
> > > >> remote
> > > >>>>>>>>>>>> storage
> > > >>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>> enabled.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> > > >>>>>>>>>>>>> jun@confluent.io>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > > >>>>>>>>>>>> process.roles
> > > >>>>>>>>>>>>>>>>> during
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> > > >>>>>>>>>>>> s7133700@gmail.com
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly
> > > and
> > > >>>>>>>>>> revised
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be
> > > >> used
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>> any
> > > >>>>>>>>>>>>>>>>> other
> > > >>>>>>>>>>>>>>>>>>>>>>>> policy.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> > > >>>>>>>>>> cleanup.policy,
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> approach
> > > >>>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> > > >>>>>>>>>>>> Additionally, a
> > > >>>>>>>>>>>>>>>>>>> warning
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > > >>>>>>>>> 2025年4月22日
> > > >>>>>>>>>>>>> 凌晨4:52
> > > >>>>>>>>>>>>>>>> 寫道：
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value
> > None
> > > >>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>> cleanup.policy.
> > > >>>>>>>>>>>>>>>>>>>>>> But
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
> > > >>>>>>>>> example,
> > > >>>>>>>>>> None
> > > >>>>>>>>>>>>>> can't
> > > >>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> > > >>>>>>>>>> document
> > > >>>>>>>>>>>>> that.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade
> > when
> > > >> an
> > > >>>>>>>>>>>> existing
> > > >>>>>>>>>>>>>>>>>>> config
> > > >>>>>>>>>>>>>>>>>>>>>> has
> > > >>>>>>>>>>>>>>>>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what
> > > >> the
> > > >>>>>>>>>> empty
> > > >>>>>>>>>>>>> check
> > > >>>>>>>>>>>>>>>>>>> does.
> > > >>>>>>>>>>>>>>>>>>>>>> How
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> > > >>>>>>>>>>>>> s7133700@gmail.com
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> > > >>>>>>>>> KIP-1161:
> > > >>>>>>>>>>>>>>>>>>>> cleanup.policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> > > >>>>>>>>>> cleanup.policy
> > > >>>>>>>>>>>>>>>>>>>> configuration.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> > > >>>>>>>>> improvements:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether
> > > an
> > > >>>>>>>>>> empty
> > > >>>>>>>>>>>> list
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>> allowed.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> > > >>>>>>>>>> cleanup.policy.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> > >
> >
> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

I agree with Chia-Ping. If we allow cleanup.policy to be empty, we will
just continue to allow it in all future releases.

For ssl.cipher.suites, changing the default value from null to empty is
actually a breaking change. According to the doc, null means that all the
available cipher suites are supported. I am thinking that we should
continue to allow null as the default since it could represent a state
different from empty.

Also, it would be useful to add a compatibility column in the table. If a
value is formally disallowed, it would be useful to compare the old and the
new behavior (e.g. a different exception is thrown, etc).

Thanks,

Jun

On Tue, Jul 8, 2025 at 8:36 AM Chia-Ping Tsai <ch...@gmail.com> wrote:

> hi Jiunn
>
> Thanks for the explanation. I have updated the KIP accordingly: in Kafka
> > 4.x, cleanup.policy
> > can be set to an empty list with a warning message emitted; however, in
> > Kafka 5.0, setting an
> > empty list for cleanup.policy will no longer be allowed.
>
>
> If we agree to support "empty list", the approach of "none" could be
> eliminated, right?
>
> I don't understand the point of "5.0". Could you please share more details?
> It seems to me changing the type or default value should be fine in minor
> release if we don't break the existing property files.
>
> `max.connections.per.ip.overrides` is parsed as "map" type, so using `LIST`
> instead of `String` does not seems to make sense.
>
> Best,
> Chia-Ping
>
>
>
>
> 黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：
>
> > Hi all,
> >
> > >> Null could make sense as the default value, but it just means that the
> > >> config key is not set explicitly.
> > >> Given that, it's probably simpler to just allow
> > >> log.cleanup.policy to be empty and properly support it?
> > Thanks for the explanation. I have updated the KIP accordingly: in Kafka
> > 4.x, cleanup.policy
> > can be set to an empty list with a warning message emitted; however, in
> > Kafka 5.0, setting an
> > empty list for cleanup.policy will no longer be allowed.
> >
> > > For another, there is an additional issue which could be handled by
> this
> > > KIP. Some configs have "string" type, but they are handled as a LIST.
> For
> > > example, `early.start.listeners`, `security.providers`. Could we change
> > > their type to LIST to benefit from this KIP? I mean to make them
> > > non-nullable.
> > I have also updated the KIP to include the tables detailing the changes
> > for LIST-type configuration
> > default values and the migration from STRING-type to LIST-type
> > configurations.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> > >
> > > hi all,
> > >
> > > JR11. There is an existing config interceptor.classes that allows an
> > empty
> > >> list and it makes intuitive sense. So, it seems that it's ok to
> support
> > >> empty lists in general. Given that, it's probably simpler to just
> allow
> > >> log.cleanup.policy to be empty and properly support it?
> > >
> > >
> > > yes, that is what I meant before. Additionally, the `LIST` type should
> > not
> > > accept "null". Instead, we should use an empty list as the default
> value.
> > > For example, `sasl.oauthbearer.expected.audience` should never has
> "null"
> > > value. That could eliminate the null check.
> > >
> > > For another, there is an additional issue which could be handled by
> this
> > > KIP. Some configs have "string" type, but they are handled as a LIST.
> For
> > > example, `early.start.listeners`, `security.providers`. Could we change
> > > their type to LIST to benefit from this KIP? I mean to make them
> > > non-nullable.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > >
> > >
> > > Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> > >
> > >> Hi, Jiunn-Yang,
> > >>
> > >> Thanks for the reply.
> > >>
> > >> JR10. Regarding supporting the null value, I am not sure how one could
> > set
> > >> a null value in a config file. For example, if you set the following
> in
> > a
> > >> config file.
> > >>  configkey=
> > >> The value of configkey will be an empty list, instead of null, right?
> > >>
> > >> Null could make sense as the default value, but it just means that the
> > >> config key is not set explicitly.
> > >>
> > >> JR11. There is an existing config interceptor.classes that allows an
> > empty
> > >> list and it makes intuitive sense. So, it seems that it's ok to
> support
> > >> empty lists in general. Given that, it's probably simpler to just
> allow
> > >> log.cleanup.policy to be empty and properly support it?
> > >>
> > >> Jun
> > >>
> > >>
> > >>
> > >> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>
> > >>> Hello Jun,
> > >>>
> > >>> JR10: The sasl.oauthbearer.expected.audience is optional, that is
> > right,
> > >>> so I think
> > >>> that we can allow null value, but not allow the empty
> > >>>
> > >>> JR11: In this KIP, the goal is to address configurations that should
> > not
> > >>> accept empty arrays.
> > >>> We want to prevent users from passing in an empty list when it's not
> > >>> valid. I believe there are two
> > >>> main scenarios to consider:
> > >>>
> > >>> 1. Nullable but not empty:
> > >>> The parameter allows either a null value or a non-empty list. In this
> > >>> case,
> > >>> if the user provides an empty list, we should throw a
> ConfigException.
> > >>>
> > >>> 2. Non-null and non-empty only: The parameter must always contain a
> > >>> non-empty list — null is also
> > >>> invalid. In this case, if the user provides either a null or an empty
> > >>> list, we should throw a ConfigException.
> > >>>
> > >>> Therefore, I want to prevent scenarios where users can pass in an
> empty
> > >>> list.
> > >>>
> > >>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
> > >> property.
> > >>> This allows us to specify
> > >>> whether a configuration permits null values or not.
> > >>>
> > >>> JR13: I have updated the table
> > >>>
> > >>>
> > >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> > >>>>
> > >>>> Hi, Jiunn-Yang,
> > >>>>
> > >>>> Sorry for the late reply. A few more comments.
> > >>>>
> > >>>> JR10. The doc for sasl.oauthbearer.expected.audience says that it's
> > >>>> optional. So an empty list seems to be valid.
> > >>>>
> > >>>> JR11. Thinking a bit more. If we are going to support empty lists in
> > >>>> general, it's probably more consistent to just allow
> > log.cleanup.policy
> > >>> to
> > >>>> be empty as Luke suggested, instead of introducing a new option
> None.
> > >>>>
> > >>>> JR12. NonEmptyListValidator allows the input to be null. It seems
> that
> > >>>> only ssl.cipher.suites allows null. The remaining ones shouldn't be
> > >> null.
> > >>>>
> > >>>> JR13. The table is very helpful. Could you also describe the old/new
> > >>>> behavior for each config with an empty list or duplicate entry?
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>
> > >>>>> Hello all,
> > >>>>>
> > >>>>> I am manually bumping this thread. Any feedback would be
> appreciated.
> > >>>>>
> > >>>>> Best regards,
> > >>>>> Jiunn-Yang
> > >>>>>
> > >>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> > >>>>>>
> > >>>>>> Hello Jun,
> > >>>>>>
> > >>>>>> I have updated the KIP and introduced a new validator,
> > >>>>> NonEmptyListValidator, which ensures that the
> > >>>>>> provided list is either null or a non-empty list without duplicate
> > >>>>> entries. I’ve also listed the configurations that will
> > >>>>>> be validated using this logic. Please feel free to share any
> > feedback
> > >>> or
> > >>>>> suggestions.
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>> Jiunn-Yang
> > >>>>>>
> > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> > >>>>>>>
> > >>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>
> > >>>>>>> It seems there are quite a few other configs of type list (e.g.
> > >>> several
> > >>>>> SSL
> > >>>>>>> related ones). It would be useful to understand if empty lists
> are
> > >>> valid
> > >>>>>>> for them or not.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>>
> > >>>>>>> Jun
> > >>>>>>>
> > >>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io>
> wrote:
> > >>>>>>>
> > >>>>>>>> Hi, Luke,
> > >>>>>>>>
> > >>>>>>>> Thanks for the reply.
> > >>>>>>>>
> > >>>>>>>> "My thought is that this behavior has been existed
> > >>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
> > >> could
> > >>>>> be
> > >>>>>>>> users relying on the empty cleanup.policy for a long time. "
> > >>>>>>>>
> > >>>>>>>> If you do a google search on "kafka infinite retention", you
> will
> > >> get
> > >>>>>>>> links like
> > >>>>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> > >>>>>>>> and
> > >>>>>>>>
> > >>>>>
> > >>>
> > >>
> >
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> > >>>>> ,
> > >>>>>>>> both pointing to setting the retention time and retention size
> to
> > >> -1
> > >>> to
> > >>>>>>>> achieve this goal. So, it seems that if a user intentionally
> wants
> > >>>>> infinite
> > >>>>>>>> retention, it's more likely they will use that approach instead
> of
> > >>>>> setting
> > >>>>>>>> cleanup.policy to empty. On the other hand, our API/tools make
> it
> > >>>>> possible
> > >>>>>>>> for users to make a mistake by inadvertently leaving the config
> > >> value
> > >>>>> empty.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com>
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Jun,
> > >>>>>>>>>
> > >>>>>>>>>> Regarding the motivation, currently, we never documented that
> an
> > >>>>> empty
> > >>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> leaves
> > >>>>>>>>> cleanup.policy empty, it actually breaks remote storage since
> it
> > >>> will
> > >>>>> stop
> > >>>>>>>>> the cleaning based on local retention size and time too. So,
> > >> leaving
> > >>>>>>>>> cleanup.policy empty is probably more like a user mistake than
> an
> > >>>>>>>>> intentional choice. Based on this assumption, the idea behind
> the
> > >>> KIP
> > >>>>> is
> > >>>>>>>>> to
> > >>>>>>>>> fail an empty cleanup.policy so that the user is aware of this
> > >>> likely
> > >>>>>>>>> mistake and provide a new option None for users wanting
> infinite
> > >>>>> retention
> > >>>>>>>>> to opt in explicitly.
> > >>>>>>>>>
> > >>>>>>>>> While we never documented the empty cleanup.policy implies
> > >> infinite
> > >>>>>>>>> retention, we also never documented (or failed) the empty
> > >>>>> cleanup.policy
> > >>>>>>>>> is
> > >>>>>>>>> an invalid configuration. My thought is that this behavior has
> > >> been
> > >>>>>>>>> existed
> > >>>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
> > >>> could
> > >>>>> be
> > >>>>>>>>> users relying on the empty cleanup.policy for a long time. It
> is
> > >> not
> > >>>>> good
> > >>>>>>>>> to break the backward compatibility in a minor release version
> > >> (ex:
> > >>>>>>>>> v4.1.0). Even though we think we are fixing a long existing
> bug,
> > >> it
> > >>>>> could
> > >>>>>>>>> be a surprise to users.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> > >> fixing
> > >>>>> the
> > >>>>>>>>> issue in remote storage. But then the user may never realize
> this
> > >> if
> > >>>>> it's
> > >>>>>>>>> a
> > >>>>>>>>> mistake.
> > >>>>>>>>>
> > >>>>>>>>> Good point! I agree we need to fix it!
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Hi Chia-Ping,
> > >>>>>>>>>
> > >>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and in
> 5.0
> > >> we
> > >>>>>>>>> throw
> > >>>>>>>>> exception directly to make it fail fast
> > >>>>>>>>>
> > >>>>>>>>> This suggestion looks good to me!
> > >>>>>>>>>
> > >>>>>>>>> Thank you.
> > >>>>>>>>> Luke
> > >>>>>>>>>
> > >>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> > >> chia7712@apache.org
> > >>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> hi all,
> > >>>>>>>>>>
> > >>>>>>>>>> Given Luke mentioned the existence of use cases, it is worth
> > >>>>> considering
> > >>>>>>>>>> the compatibility issue. In fact, I had previously encountered
> > >> this
> > >>>>> use
> > >>>>>>>>>> case but advised customers to utilize retention configurations
> > >>>>> instead.
> > >>>>>>>>>>
> > >>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> > >>> fixing
> > >>>>>>>>> the
> > >>>>>>>>>>> issue in remote storage. But then the user may never realize
> > >> this
> > >>> if
> > >>>>>>>>>> it's a
> > >>>>>>>>>>> mistake.
> > >>>>>>>>>>
> > >>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for
> remote
> > >>>>>>>>> storage.
> > >>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage
> use
> > >> a
> > >>>>> valid
> > >>>>>>>>>> delete policy.
> > >>>>>>>>>>
> > >>>>>>>>>> Best,
> > >>>>>>>>>> Chia-Ping
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> > >>>>>>>>>>> Hi, Luke,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Regarding the motivation, currently, we never documented that
> > an
> > >>>>> empty
> > >>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> > >> leaves
> > >>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage since
> > it
> > >>>>> will
> > >>>>>>>>>> stop
> > >>>>>>>>>>> the cleaning based on local retention size and time too. So,
> > >>> leaving
> > >>>>>>>>>>> cleanup.policy empty is probably more like a user mistake
> than
> > >> an
> > >>>>>>>>>>> intentional choice. Based on this assumption, the idea behind
> > >> the
> > >>>>> KIP
> > >>>>>>>>> is
> > >>>>>>>>>> to
> > >>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of
> this
> > >>>>> likely
> > >>>>>>>>>>> mistake and provide a new option None for users wanting
> > infinite
> > >>>>>>>>>> retention
> > >>>>>>>>>>> to opt in explicitly.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> > >>> fixing
> > >>>>>>>>> the
> > >>>>>>>>>>> issue in remote storage. But then the user may never realize
> > >> this
> > >>> if
> > >>>>>>>>>> it's a
> > >>>>>>>>>>> mistake.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Jun
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <showuon@gmail.com
> >
> > >>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Comments:
> > >>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
> > >>> considered
> > >>>>>>>>>> invalid
> > >>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> > >>>>>>>>>>>> Could you explain more about this? Why is this trade-off
> > >>> justified?
> > >>>>>>>>>>>> If we think the empty cleanup.policy is invalid in kafka,
> why
> > >> do
> > >>> we
> > >>>>>>>>>> provide
> > >>>>>>>>>>>> an additional "none" option to users in this KIP?
> > >>>>>>>>>>>> FYI, there are indeed use cases that need to keep all
> history
> > >>> data
> > >>>>>>>>>> without
> > >>>>>>>>>>>> a cleanup policy.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
> > >>>>>>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles"
> > >>> configs
> > >>>>>>>>>> since
> > >>>>>>>>>>>> they are invalid.
> > >>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive reason
> > why
> > >>> we
> > >>>>>>>>>> should
> > >>>>>>>>>>>> break backward compatibility to change it.
> > >>>>>>>>>>>> "This provides a clear and direct way to configure Kafka to
> > >>> retain
> > >>>>>>>>> all
> > >>>>>>>>>> log
> > >>>>>>>>>>>> segments without any automatic cleanup."
> > >>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to me,
> > >>> backward
> > >>>>>>>>>>>> compatibility is much more important than "a clear and
> direct
> > >> way
> > >>>>> to
> > >>>>>>>>>> config
> > >>>>>>>>>>>> kafka".
> > >>>>>>>>>>>> Do we really have to change the "cleanup.policy" config?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks.
> > >>>>>>>>>>>> Luke
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> > >> <jun@confluent.io.invalid
> > >>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com>
> > >> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi Jun, chia
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed
> in
> > >>> the
> > >>>>>>>>>>>> updated
> > >>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I've removed all deprecated-related sections.
> Additionally,
> > >> an
> > >>>>>>>>>>>> exception
> > >>>>>>>>>>>>>> will now be thrown if a
> > >>>>>>>>>>>>>> developer passes an empty validStrings list to the
> > inNonEmpty
> > >>>>>>>>>> method.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46
> > 寫道：
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty
> > config,
> > >>>>>>>>> so
> > >>>>>>>>>>>> should
> > >>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>> be non-empty too?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
> > directly.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
> > >> 上午12:28寫道：
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the reply. There are still references to the
> > >>>>>>>>>> deprecated
> > >>>>>>>>>>>>>> method.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> > >>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka
> > >> 5.0"
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been
> addressed
> > >> in
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> updated
> > >>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日
> 凌晨2:03
> > >> 寫道：
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration
> Plan",
> > >>>>>>>>>> could you
> > >>>>>>>>>>>>> add
> > >>>>>>>>>>>>>>>>>> description to remind readers that "clean.policy="
> > should
> > >>>>>>>>> be
> > >>>>>>>>>>>>> replaced
> > >>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete
> > and
> > >>>>>>>>>> compact.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
> > >>>>>>>>> 上午1:55寫道：
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now.
> Could
> > >> you
> > >>>>>>>>>> update
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>> completely? There are still references to
> deprecation.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <
> s7133700@gmail.com
> > >
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
> > >>>>>>>>> method,
> > >>>>>>>>>> as
> > >>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>> future scenarios where we want
> > >>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to
> have
> > >>>>>>>>> both
> > >>>>>>>>>>>>> methods:
> > >>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> > >>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> document
> > >>>>>>>>> that
> > >>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> > >>>>>>>>> start.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
> > >> 凌晨12:50
> > >>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to
> document
> > >>>>>>>>> that
> > >>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> > >>>>>>>>> start.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> > >> s7133700@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to
> maintain
> > >>>>>>>>>> backward
> > >>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag
> is
> > >>>>>>>>>>>> necessary.
> > >>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to
> > remove
> > >>>>>>>>> the
> > >>>>>>>>>>>>> boolean
> > >>>>>>>>>>>>>>>>>>>>>> parameter.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日
> > >> 凌晨4:17
> > >>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward
> > since
> > >>>>>>>>> the
> > >>>>>>>>>>>> method
> > >>>>>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
> > >>>>>>>>> true.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> > >>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
> > >>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>> configurations
> > >>>>>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> > >>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
> > >>>>>>>>> explicitly
> > >>>>>>>>>>>>> throw a
> > >>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
> > >> validation
> > >>>>>>>>>> phase
> > >>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> 2025年4月25日
> > >>>>>>>>>> 凌晨1:17 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by
> > this
> > >>>>>>>>>> change
> > >>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
> > >>>>>>>>> storage
> > >>>>>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>>>>> validation
> > >>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
> > >>>>>>>>> already
> > >>>>>>>>>> fails
> > >>>>>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>> process.roles
> > >>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>> empty.
> > >>>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> > >>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> > >>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>> will be
> > >>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
> > >> storage
> > >>>>>>>>>> format
> > >>>>>>>>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list
> will
> > >>>>>>>>> now
> > >>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>>>>> and will
> > >>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage
> > format
> > >>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > 2025年4月24日
> > >>>>>>>>>> 凌晨4:44
> > >>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> > >>>>>>>>>> group.coordinator.rebalance.protocols
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With
> > >> this
> > >>>>>>>>>> KIP,
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
> > >>>>>>>>>> exception.
> > >>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>> useful
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree
> with
> > >>>>>>>>>> Chia-Ping.
> > >>>>>>>>>>>>> If
> > >>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be
> a
> > >>>>>>>>>> mistake
> > >>>>>>>>>>>> than
> > >>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>> intention.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the
> user
> > >>>>>>>>> will
> > >>>>>>>>>> never
> > >>>>>>>>>>>>>>>>> realize
> > >>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the
> > user
> > >>>>>>>>>> realize
> > >>>>>>>>>>>>>> there
> > >>>>>>>>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping
> Tsai
> > <
> > >>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> > >>>>>>>>>> maintaining
> > >>>>>>>>>>>>>>>>>>>> compatibility
> > >>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we
> > >> don't
> > >>>>>>>>>> allow
> > >>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>> value
> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we
> consider
> > >>>>>>>>>> throwing
> > >>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
> > >> streamline
> > >>>>>>>>>> the
> > >>>>>>>>>>>> code
> > >>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>> avoid
> > >>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> > >> 2025年4月23日
> > >>>>>>>>> 週三
> > >>>>>>>>>>>>>> 上午6:33寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete
> or
> > >>>>>>>>>> compact
> > >>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>>>>> segments",
> > >>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
> > >>>>>>>>> segments
> > >>>>>>>>>> based
> > >>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but
> we
> > >>>>>>>>>> should
> > >>>>>>>>>>>>>>>> continue
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>> delete
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on
> log.local.retention.bytes/
> > >>>>>>>>>>>>>>>> log.retention.ms.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when
> > >> remote
> > >>>>>>>>>>>> storage
> > >>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>> enabled.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> > >>>>>>>>>>>>> jun@confluent.io>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> > >>>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly
> > and
> > >>>>>>>>>> revised
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be
> > >> used
> > >>>>>>>>>> with
> > >>>>>>>>>>>> any
> > >>>>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>>>>> policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> > >>>>>>>>>> cleanup.policy,
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> > >>>>>>>>>>>> Additionally, a
> > >>>>>>>>>>>>>>>>>>> warning
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >>>>>>>>> 2025年4月22日
> > >>>>>>>>>>>>> 凌晨4:52
> > >>>>>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value
> None
> > >>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>> But
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
> > >>>>>>>>> example,
> > >>>>>>>>>> None
> > >>>>>>>>>>>>>> can't
> > >>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> > >>>>>>>>>> document
> > >>>>>>>>>>>>> that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade
> when
> > >> an
> > >>>>>>>>>>>> existing
> > >>>>>>>>>>>>>>>>>>> config
> > >>>>>>>>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what
> > >> the
> > >>>>>>>>>> empty
> > >>>>>>>>>>>>> check
> > >>>>>>>>>>>>>>>>>>> does.
> > >>>>>>>>>>>>>>>>>>>>>> How
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> > >>>>>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> > >>>>>>>>> KIP-1161:
> > >>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> > >>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>> configuration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> > >>>>>>>>> improvements:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether
> > an
> > >>>>>>>>>> empty
> > >>>>>>>>>>>> list
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>> allowed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> > >>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

hi Jiunn

Thanks for the explanation. I have updated the KIP accordingly: in Kafka
> 4.x, cleanup.policy
> can be set to an empty list with a warning message emitted; however, in
> Kafka 5.0, setting an
> empty list for cleanup.policy will no longer be allowed.


If we agree to support "empty list", the approach of "none" could be
eliminated, right?

I don't understand the point of "5.0". Could you please share more details?
It seems to me changing the type or default value should be fine in minor
release if we don't break the existing property files.

`max.connections.per.ip.overrides` is parsed as "map" type, so using `LIST`
instead of `String` does not seems to make sense.

Best,
Chia-Ping




黃竣陽 <s7...@gmail.com> 於 2025年7月8日 週二 下午11:25寫道：

> Hi all,
>
> >> Null could make sense as the default value, but it just means that the
> >> config key is not set explicitly.
> >> Given that, it's probably simpler to just allow
> >> log.cleanup.policy to be empty and properly support it?
> Thanks for the explanation. I have updated the KIP accordingly: in Kafka
> 4.x, cleanup.policy
> can be set to an empty list with a warning message emitted; however, in
> Kafka 5.0, setting an
> empty list for cleanup.policy will no longer be allowed.
>
> > For another, there is an additional issue which could be handled by this
> > KIP. Some configs have "string" type, but they are handled as a LIST. For
> > example, `early.start.listeners`, `security.providers`. Could we change
> > their type to LIST to benefit from this KIP? I mean to make them
> > non-nullable.
> I have also updated the KIP to include the tables detailing the changes
> for LIST-type configuration
> default values and the migration from STRING-type to LIST-type
> configurations.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> >
> > hi all,
> >
> > JR11. There is an existing config interceptor.classes that allows an
> empty
> >> list and it makes intuitive sense. So, it seems that it's ok to support
> >> empty lists in general. Given that, it's probably simpler to just allow
> >> log.cleanup.policy to be empty and properly support it?
> >
> >
> > yes, that is what I meant before. Additionally, the `LIST` type should
> not
> > accept "null". Instead, we should use an empty list as the default value.
> > For example, `sasl.oauthbearer.expected.audience` should never has "null"
> > value. That could eliminate the null check.
> >
> > For another, there is an additional issue which could be handled by this
> > KIP. Some configs have "string" type, but they are handled as a LIST. For
> > example, `early.start.listeners`, `security.providers`. Could we change
> > their type to LIST to benefit from this KIP? I mean to make them
> > non-nullable.
> >
> > Best,
> > Chia-Ping
> >
> >
> >
> > Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> >
> >> Hi, Jiunn-Yang,
> >>
> >> Thanks for the reply.
> >>
> >> JR10. Regarding supporting the null value, I am not sure how one could
> set
> >> a null value in a config file. For example, if you set the following in
> a
> >> config file.
> >>  configkey=
> >> The value of configkey will be an empty list, instead of null, right?
> >>
> >> Null could make sense as the default value, but it just means that the
> >> config key is not set explicitly.
> >>
> >> JR11. There is an existing config interceptor.classes that allows an
> empty
> >> list and it makes intuitive sense. So, it seems that it's ok to support
> >> empty lists in general. Given that, it's probably simpler to just allow
> >> log.cleanup.policy to be empty and properly support it?
> >>
> >> Jun
> >>
> >>
> >>
> >> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hello Jun,
> >>>
> >>> JR10: The sasl.oauthbearer.expected.audience is optional, that is
> right,
> >>> so I think
> >>> that we can allow null value, but not allow the empty
> >>>
> >>> JR11: In this KIP, the goal is to address configurations that should
> not
> >>> accept empty arrays.
> >>> We want to prevent users from passing in an empty list when it's not
> >>> valid. I believe there are two
> >>> main scenarios to consider:
> >>>
> >>> 1. Nullable but not empty:
> >>> The parameter allows either a null value or a non-empty list. In this
> >>> case,
> >>> if the user provides an empty list, we should throw a ConfigException.
> >>>
> >>> 2. Non-null and non-empty only: The parameter must always contain a
> >>> non-empty list — null is also
> >>> invalid. In this case, if the user provides either a null or an empty
> >>> list, we should throw a ConfigException.
> >>>
> >>> Therefore, I want to prevent scenarios where users can pass in an empty
> >>> list.
> >>>
> >>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
> >> property.
> >>> This allows us to specify
> >>> whether a configuration permits null values or not.
> >>>
> >>> JR13: I have updated the table
> >>>
> >>>
> >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> >>>>
> >>>> Hi, Jiunn-Yang,
> >>>>
> >>>> Sorry for the late reply. A few more comments.
> >>>>
> >>>> JR10. The doc for sasl.oauthbearer.expected.audience says that it's
> >>>> optional. So an empty list seems to be valid.
> >>>>
> >>>> JR11. Thinking a bit more. If we are going to support empty lists in
> >>>> general, it's probably more consistent to just allow
> log.cleanup.policy
> >>> to
> >>>> be empty as Luke suggested, instead of introducing a new option None.
> >>>>
> >>>> JR12. NonEmptyListValidator allows the input to be null. It seems that
> >>>> only ssl.cipher.suites allows null. The remaining ones shouldn't be
> >> null.
> >>>>
> >>>> JR13. The table is very helpful. Could you also describe the old/new
> >>>> behavior for each config with an empty list or duplicate entry?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>
> >>>>> Hello all,
> >>>>>
> >>>>> I am manually bumping this thread. Any feedback would be appreciated.
> >>>>>
> >>>>> Best regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> >>>>>>
> >>>>>> Hello Jun,
> >>>>>>
> >>>>>> I have updated the KIP and introduced a new validator,
> >>>>> NonEmptyListValidator, which ensures that the
> >>>>>> provided list is either null or a non-empty list without duplicate
> >>>>> entries. I’ve also listed the configurations that will
> >>>>>> be validated using this logic. Please feel free to share any
> feedback
> >>> or
> >>>>> suggestions.
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> >>>>>>>
> >>>>>>> Hi, Jiunn-Yang,
> >>>>>>>
> >>>>>>> It seems there are quite a few other configs of type list (e.g.
> >>> several
> >>>>> SSL
> >>>>>>> related ones). It would be useful to understand if empty lists are
> >>> valid
> >>>>>>> for them or not.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Jun
> >>>>>>>
> >>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:
> >>>>>>>
> >>>>>>>> Hi, Luke,
> >>>>>>>>
> >>>>>>>> Thanks for the reply.
> >>>>>>>>
> >>>>>>>> "My thought is that this behavior has been existed
> >>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
> >> could
> >>>>> be
> >>>>>>>> users relying on the empty cleanup.policy for a long time. "
> >>>>>>>>
> >>>>>>>> If you do a google search on "kafka infinite retention", you will
> >> get
> >>>>>>>> links like
> >>>>>>>>
> >>>>>
> >>>
> >>
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> >>>>>>>> and
> >>>>>>>>
> >>>>>
> >>>
> >>
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> >>>>> ,
> >>>>>>>> both pointing to setting the retention time and retention size to
> >> -1
> >>> to
> >>>>>>>> achieve this goal. So, it seems that if a user intentionally wants
> >>>>> infinite
> >>>>>>>> retention, it's more likely they will use that approach instead of
> >>>>> setting
> >>>>>>>> cleanup.policy to empty. On the other hand, our API/tools make it
> >>>>> possible
> >>>>>>>> for users to make a mistake by inadvertently leaving the config
> >> value
> >>>>> empty.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com>
> >>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Jun,
> >>>>>>>>>
> >>>>>>>>>> Regarding the motivation, currently, we never documented that an
> >>>>> empty
> >>>>>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
> >>>>>>>>> cleanup.policy empty, it actually breaks remote storage since it
> >>> will
> >>>>> stop
> >>>>>>>>> the cleaning based on local retention size and time too. So,
> >> leaving
> >>>>>>>>> cleanup.policy empty is probably more like a user mistake than an
> >>>>>>>>> intentional choice. Based on this assumption, the idea behind the
> >>> KIP
> >>>>> is
> >>>>>>>>> to
> >>>>>>>>> fail an empty cleanup.policy so that the user is aware of this
> >>> likely
> >>>>>>>>> mistake and provide a new option None for users wanting infinite
> >>>>> retention
> >>>>>>>>> to opt in explicitly.
> >>>>>>>>>
> >>>>>>>>> While we never documented the empty cleanup.policy implies
> >> infinite
> >>>>>>>>> retention, we also never documented (or failed) the empty
> >>>>> cleanup.policy
> >>>>>>>>> is
> >>>>>>>>> an invalid configuration. My thought is that this behavior has
> >> been
> >>>>>>>>> existed
> >>>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
> >>> could
> >>>>> be
> >>>>>>>>> users relying on the empty cleanup.policy for a long time. It is
> >> not
> >>>>> good
> >>>>>>>>> to break the backward compatibility in a minor release version
> >> (ex:
> >>>>>>>>> v4.1.0). Even though we think we are fixing a long existing bug,
> >> it
> >>>>> could
> >>>>>>>>> be a surprise to users.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> >> fixing
> >>>>> the
> >>>>>>>>> issue in remote storage. But then the user may never realize this
> >> if
> >>>>> it's
> >>>>>>>>> a
> >>>>>>>>> mistake.
> >>>>>>>>>
> >>>>>>>>> Good point! I agree we need to fix it!
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hi Chia-Ping,
> >>>>>>>>>
> >>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0
> >> we
> >>>>>>>>> throw
> >>>>>>>>> exception directly to make it fail fast
> >>>>>>>>>
> >>>>>>>>> This suggestion looks good to me!
> >>>>>>>>>
> >>>>>>>>> Thank you.
> >>>>>>>>> Luke
> >>>>>>>>>
> >>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> >> chia7712@apache.org
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> hi all,
> >>>>>>>>>>
> >>>>>>>>>> Given Luke mentioned the existence of use cases, it is worth
> >>>>> considering
> >>>>>>>>>> the compatibility issue. In fact, I had previously encountered
> >> this
> >>>>> use
> >>>>>>>>>> case but advised customers to utilize retention configurations
> >>>>> instead.
> >>>>>>>>>>
> >>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> >>> fixing
> >>>>>>>>> the
> >>>>>>>>>>> issue in remote storage. But then the user may never realize
> >> this
> >>> if
> >>>>>>>>>> it's a
> >>>>>>>>>>> mistake.
> >>>>>>>>>>
> >>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for remote
> >>>>>>>>> storage.
> >>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage use
> >> a
> >>>>> valid
> >>>>>>>>>> delete policy.
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Chia-Ping
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> >>>>>>>>>>> Hi, Luke,
> >>>>>>>>>>>
> >>>>>>>>>>> Regarding the motivation, currently, we never documented that
> an
> >>>>> empty
> >>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> >> leaves
> >>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage since
> it
> >>>>> will
> >>>>>>>>>> stop
> >>>>>>>>>>> the cleaning based on local retention size and time too. So,
> >>> leaving
> >>>>>>>>>>> cleanup.policy empty is probably more like a user mistake than
> >> an
> >>>>>>>>>>> intentional choice. Based on this assumption, the idea behind
> >> the
> >>>>> KIP
> >>>>>>>>> is
> >>>>>>>>>> to
> >>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of this
> >>>>> likely
> >>>>>>>>>>> mistake and provide a new option None for users wanting
> infinite
> >>>>>>>>>> retention
> >>>>>>>>>>> to opt in explicitly.
> >>>>>>>>>>>
> >>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
> >>> fixing
> >>>>>>>>> the
> >>>>>>>>>>> issue in remote storage. But then the user may never realize
> >> this
> >>> if
> >>>>>>>>>> it's a
> >>>>>>>>>>> mistake.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Comments:
> >>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
> >>> considered
> >>>>>>>>>> invalid
> >>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> >>>>>>>>>>>> Could you explain more about this? Why is this trade-off
> >>> justified?
> >>>>>>>>>>>> If we think the empty cleanup.policy is invalid in kafka, why
> >> do
> >>> we
> >>>>>>>>>> provide
> >>>>>>>>>>>> an additional "none" option to users in this KIP?
> >>>>>>>>>>>> FYI, there are indeed use cases that need to keep all history
> >>> data
> >>>>>>>>>> without
> >>>>>>>>>>>> a cleanup policy.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
> >>>>>>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles"
> >>> configs
> >>>>>>>>>> since
> >>>>>>>>>>>> they are invalid.
> >>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive reason
> why
> >>> we
> >>>>>>>>>> should
> >>>>>>>>>>>> break backward compatibility to change it.
> >>>>>>>>>>>> "This provides a clear and direct way to configure Kafka to
> >>> retain
> >>>>>>>>> all
> >>>>>>>>>> log
> >>>>>>>>>>>> segments without any automatic cleanup."
> >>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to me,
> >>> backward
> >>>>>>>>>>>> compatibility is much more important than "a clear and direct
> >> way
> >>>>> to
> >>>>>>>>>> config
> >>>>>>>>>>>> kafka".
> >>>>>>>>>>>> Do we really have to change the "cleanup.policy" config?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks.
> >>>>>>>>>>>> Luke
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> >> <jun@confluent.io.invalid
> >>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com>
> >> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Jun, chia
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed in
> >>> the
> >>>>>>>>>>>> updated
> >>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I've removed all deprecated-related sections. Additionally,
> >> an
> >>>>>>>>>>>> exception
> >>>>>>>>>>>>>> will now be thrown if a
> >>>>>>>>>>>>>> developer passes an empty validStrings list to the
> inNonEmpty
> >>>>>>>>>> method.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46
> 寫道：
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty
> config,
> >>>>>>>>> so
> >>>>>>>>>>>> should
> >>>>>>>>>>>>> it
> >>>>>>>>>>>>>>> be non-empty too?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception
> directly.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
> >> 上午12:28寫道：
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the reply. There are still references to the
> >>>>>>>>>> deprecated
> >>>>>>>>>>>>>> method.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
> >>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka
> >> 5.0"
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed
> >> in
> >>>>>>>>>> the
> >>>>>>>>>>>>>> updated
> >>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03
> >> 寫道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan",
> >>>>>>>>>> could you
> >>>>>>>>>>>>> add
> >>>>>>>>>>>>>>>>>> description to remind readers that "clean.policy="
> should
> >>>>>>>>> be
> >>>>>>>>>>>>> replaced
> >>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete
> and
> >>>>>>>>>> compact.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
> >>>>>>>>> 上午1:55寫道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could
> >> you
> >>>>>>>>>> update
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>> completely? There are still references to deprecation.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7133700@gmail.com
> >
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
> >>>>>>>>> method,
> >>>>>>>>>> as
> >>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> future scenarios where we want
> >>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have
> >>>>>>>>> both
> >>>>>>>>>>>>> methods:
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> (which allows empty lists)
> >>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
> >>>>>>>>> that
> >>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> >>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
> >> 凌晨12:50
> >>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
> >>>>>>>>> that
> >>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> >>>>>>>>> start.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> >> s7133700@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain
> >>>>>>>>>> backward
> >>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> >>>>>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to
> remove
> >>>>>>>>> the
> >>>>>>>>>>>>> boolean
> >>>>>>>>>>>>>>>>>>>>>> parameter.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日
> >> 凌晨4:17
> >>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward
> since
> >>>>>>>>> the
> >>>>>>>>>>>> method
> >>>>>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
> >>>>>>>>> true.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> >>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
> >>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
> >>>>>>>>> explicitly
> >>>>>>>>>>>>> throw a
> >>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
> >> validation
> >>>>>>>>>> phase
> >>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
> >>>>>>>>>> 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by
> this
> >>>>>>>>>> change
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
> >>>>>>>>> storage
> >>>>>>>>>>>> format
> >>>>>>>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
> >>>>>>>>> already
> >>>>>>>>>> fails
> >>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>> process.roles
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> >>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
> >>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
> >> storage
> >>>>>>>>>> format
> >>>>>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will
> >>>>>>>>> now
> >>>>>>>>>> be
> >>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage
> format
> >>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> 2025年4月24日
> >>>>>>>>>> 凌晨4:44
> >>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> >>>>>>>>>> group.coordinator.rebalance.protocols
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With
> >> this
> >>>>>>>>>> KIP,
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
> >>>>>>>>>> exception.
> >>>>>>>>>>>> It
> >>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
> >>>>>>>>>> Chia-Ping.
> >>>>>>>>>>>>> If
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
> >>>>>>>>>> mistake
> >>>>>>>>>>>> than
> >>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user
> >>>>>>>>> will
> >>>>>>>>>> never
> >>>>>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the
> user
> >>>>>>>>>> realize
> >>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai
> <
> >>>>>>>>>>>>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> >>>>>>>>>> maintaining
> >>>>>>>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we
> >> don't
> >>>>>>>>>> allow
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
> >>>>>>>>>> throwing
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
> >> streamline
> >>>>>>>>>> the
> >>>>>>>>>>>> code
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> >> 2025年4月23日
> >>>>>>>>> 週三
> >>>>>>>>>>>>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or
> >>>>>>>>>> compact
> >>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>>>>>>> segments",
> >>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
> >>>>>>>>> segments
> >>>>>>>>>> based
> >>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
> >>>>>>>>>> should
> >>>>>>>>>>>>>>>> continue
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> >>>>>>>>>>>>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when
> >> remote
> >>>>>>>>>>>> storage
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> >>>>>>>>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> >>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly
> and
> >>>>>>>>>> revised
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be
> >> used
> >>>>>>>>>> with
> >>>>>>>>>>>> any
> >>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> >>>>>>>>>> cleanup.policy,
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> >>>>>>>>>>>> Additionally, a
> >>>>>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>>>>> 2025年4月22日
> >>>>>>>>>>>>> 凌晨4:52
> >>>>>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
> >>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
> >>>>>>>>> example,
> >>>>>>>>>> None
> >>>>>>>>>>>>>> can't
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> >>>>>>>>>> document
> >>>>>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when
> >> an
> >>>>>>>>>>>> existing
> >>>>>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what
> >> the
> >>>>>>>>>> empty
> >>>>>>>>>>>>> check
> >>>>>>>>>>>>>>>>>>> does.
> >>>>>>>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> >>>>>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> >>>>>>>>> KIP-1161:
> >>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> >>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> >>>>>>>>> improvements:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether
> an
> >>>>>>>>>> empty
> >>>>>>>>>>>> list
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> >>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi all, 

>> Null could make sense as the default value, but it just means that the
>> config key is not set explicitly.
>> Given that, it's probably simpler to just allow
>> log.cleanup.policy to be empty and properly support it?
Thanks for the explanation. I have updated the KIP accordingly: in Kafka 4.x, cleanup.policy 
can be set to an empty list with a warning message emitted; however, in Kafka 5.0, setting an 
empty list for cleanup.policy will no longer be allowed.

> For another, there is an additional issue which could be handled by this
> KIP. Some configs have "string" type, but they are handled as a LIST. For
> example, `early.start.listeners`, `security.providers`. Could we change
> their type to LIST to benefit from this KIP? I mean to make them
> non-nullable.
I have also updated the KIP to include the tables detailing the changes for LIST-type configuration 
default values and the migration from STRING-type to LIST-type configurations.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年7月8日 清晨7:06 寫道：
> 
> hi all,
> 
> JR11. There is an existing config interceptor.classes that allows an empty
>> list and it makes intuitive sense. So, it seems that it's ok to support
>> empty lists in general. Given that, it's probably simpler to just allow
>> log.cleanup.policy to be empty and properly support it?
> 
> 
> yes, that is what I meant before. Additionally, the `LIST` type should not
> accept "null". Instead, we should use an empty list as the default value.
> For example, `sasl.oauthbearer.expected.audience` should never has "null"
> value. That could eliminate the null check.
> 
> For another, there is an additional issue which could be handled by this
> KIP. Some configs have "string" type, but they are handled as a LIST. For
> example, `early.start.listeners`, `security.providers`. Could we change
> their type to LIST to benefit from this KIP? I mean to make them
> non-nullable.
> 
> Best,
> Chia-Ping
> 
> 
> 
> Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：
> 
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the reply.
>> 
>> JR10. Regarding supporting the null value, I am not sure how one could set
>> a null value in a config file. For example, if you set the following in a
>> config file.
>>  configkey=
>> The value of configkey will be an empty list, instead of null, right?
>> 
>> Null could make sense as the default value, but it just means that the
>> config key is not set explicitly.
>> 
>> JR11. There is an existing config interceptor.classes that allows an empty
>> list and it makes intuitive sense. So, it seems that it's ok to support
>> empty lists in general. Given that, it's probably simpler to just allow
>> log.cleanup.policy to be empty and properly support it?
>> 
>> Jun
>> 
>> 
>> 
>> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hello Jun,
>>> 
>>> JR10: The sasl.oauthbearer.expected.audience is optional, that is right,
>>> so I think
>>> that we can allow null value, but not allow the empty
>>> 
>>> JR11: In this KIP, the goal is to address configurations that should not
>>> accept empty arrays.
>>> We want to prevent users from passing in an empty list when it's not
>>> valid. I believe there are two
>>> main scenarios to consider:
>>> 
>>> 1. Nullable but not empty:
>>> The parameter allows either a null value or a non-empty list. In this
>>> case,
>>> if the user provides an empty list, we should throw a ConfigException.
>>> 
>>> 2. Non-null and non-empty only: The parameter must always contain a
>>> non-empty list — null is also
>>> invalid. In this case, if the user provides either a null or an empty
>>> list, we should throw a ConfigException.
>>> 
>>> Therefore, I want to prevent scenarios where users can pass in an empty
>>> list.
>>> 
>>> JR12: I’ve updated NonEmptyListValidator to include an allowNull
>> property.
>>> This allows us to specify
>>> whether a configuration permits null values or not.
>>> 
>>> JR13: I have updated the table
>>> 
>>> 
>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
>>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Sorry for the late reply. A few more comments.
>>>> 
>>>> JR10. The doc for sasl.oauthbearer.expected.audience says that it's
>>>> optional. So an empty list seems to be valid.
>>>> 
>>>> JR11. Thinking a bit more. If we are going to support empty lists in
>>>> general, it's probably more consistent to just allow log.cleanup.policy
>>> to
>>>> be empty as Luke suggested, instead of introducing a new option None.
>>>> 
>>>> JR12. NonEmptyListValidator allows the input to be null. It seems that
>>>> only ssl.cipher.suites allows null. The remaining ones shouldn't be
>> null.
>>>> 
>>>> JR13. The table is very helpful. Could you also describe the old/new
>>>> behavior for each config with an empty list or duplicate entry?
>>>> 
>>>> Thanks,
>>>> 
>>>> Jun
>>>> 
>>>> 
>>>> 
>>>> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hello all,
>>>>> 
>>>>> I am manually bumping this thread. Any feedback would be appreciated.
>>>>> 
>>>>> Best regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
>>>>>> 
>>>>>> Hello Jun,
>>>>>> 
>>>>>> I have updated the KIP and introduced a new validator,
>>>>> NonEmptyListValidator, which ensures that the
>>>>>> provided list is either null or a non-empty list without duplicate
>>>>> entries. I’ve also listed the configurations that will
>>>>>> be validated using this logic. Please feel free to share any feedback
>>> or
>>>>> suggestions.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
>>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> It seems there are quite a few other configs of type list (e.g.
>>> several
>>>>> SSL
>>>>>>> related ones). It would be useful to understand if empty lists are
>>> valid
>>>>>>> for them or not.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:
>>>>>>> 
>>>>>>>> Hi, Luke,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> "My thought is that this behavior has been existed
>>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
>> could
>>>>> be
>>>>>>>> users relying on the empty cleanup.policy for a long time. "
>>>>>>>> 
>>>>>>>> If you do a google search on "kafka infinite retention", you will
>> get
>>>>>>>> links like
>>>>>>>> 
>>>>> 
>>> 
>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>>>>>>> and
>>>>>>>> 
>>>>> 
>>> 
>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
>>>>> ,
>>>>>>>> both pointing to setting the retention time and retention size to
>> -1
>>> to
>>>>>>>> achieve this goal. So, it seems that if a user intentionally wants
>>>>> infinite
>>>>>>>> retention, it's more likely they will use that approach instead of
>>>>> setting
>>>>>>>> cleanup.policy to empty. On the other hand, our API/tools make it
>>>>> possible
>>>>>>>> for users to make a mistake by inadvertently leaving the config
>> value
>>>>> empty.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com>
>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jun,
>>>>>>>>> 
>>>>>>>>>> Regarding the motivation, currently, we never documented that an
>>>>> empty
>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
>>>>>>>>> cleanup.policy empty, it actually breaks remote storage since it
>>> will
>>>>> stop
>>>>>>>>> the cleaning based on local retention size and time too. So,
>> leaving
>>>>>>>>> cleanup.policy empty is probably more like a user mistake than an
>>>>>>>>> intentional choice. Based on this assumption, the idea behind the
>>> KIP
>>>>> is
>>>>>>>>> to
>>>>>>>>> fail an empty cleanup.policy so that the user is aware of this
>>> likely
>>>>>>>>> mistake and provide a new option None for users wanting infinite
>>>>> retention
>>>>>>>>> to opt in explicitly.
>>>>>>>>> 
>>>>>>>>> While we never documented the empty cleanup.policy implies
>> infinite
>>>>>>>>> retention, we also never documented (or failed) the empty
>>>>> cleanup.policy
>>>>>>>>> is
>>>>>>>>> an invalid configuration. My thought is that this behavior has
>> been
>>>>>>>>> existed
>>>>>>>>> for years (maybe after "cleanup.policy" was introduced?), there
>>> could
>>>>> be
>>>>>>>>> users relying on the empty cleanup.policy for a long time. It is
>> not
>>>>> good
>>>>>>>>> to break the backward compatibility in a minor release version
>> (ex:
>>>>>>>>> v4.1.0). Even though we think we are fixing a long existing bug,
>> it
>>>>> could
>>>>>>>>> be a surprise to users.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
>> fixing
>>>>> the
>>>>>>>>> issue in remote storage. But then the user may never realize this
>> if
>>>>> it's
>>>>>>>>> a
>>>>>>>>> mistake.
>>>>>>>>> 
>>>>>>>>> Good point! I agree we need to fix it!
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Hi Chia-Ping,
>>>>>>>>> 
>>>>>>>>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0
>> we
>>>>>>>>> throw
>>>>>>>>> exception directly to make it fail fast
>>>>>>>>> 
>>>>>>>>> This suggestion looks good to me!
>>>>>>>>> 
>>>>>>>>> Thank you.
>>>>>>>>> Luke
>>>>>>>>> 
>>>>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
>> chia7712@apache.org
>>>> 
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> hi all,
>>>>>>>>>> 
>>>>>>>>>> Given Luke mentioned the existence of use cases, it is worth
>>>>> considering
>>>>>>>>>> the compatibility issue. In fact, I had previously encountered
>> this
>>>>> use
>>>>>>>>>> case but advised customers to utilize retention configurations
>>>>> instead.
>>>>>>>>>> 
>>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
>>> fixing
>>>>>>>>> the
>>>>>>>>>>> issue in remote storage. But then the user may never realize
>> this
>>> if
>>>>>>>>>> it's a
>>>>>>>>>>> mistake.
>>>>>>>>>> 
>>>>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for remote
>>>>>>>>> storage.
>>>>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage use
>> a
>>>>> valid
>>>>>>>>>> delete policy.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Chia-Ping
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>>>>>>> Hi, Luke,
>>>>>>>>>>> 
>>>>>>>>>>> Regarding the motivation, currently, we never documented that an
>>>>> empty
>>>>>>>>>>> cleanup.policy implies infinite retention. In fact, if one
>> leaves
>>>>>>>>>>> cleanup.policy empty, it actually breaks remote storage since it
>>>>> will
>>>>>>>>>> stop
>>>>>>>>>>> the cleaning based on local retention size and time too. So,
>>> leaving
>>>>>>>>>>> cleanup.policy empty is probably more like a user mistake than
>> an
>>>>>>>>>>> intentional choice. Based on this assumption, the idea behind
>> the
>>>>> KIP
>>>>>>>>> is
>>>>>>>>>> to
>>>>>>>>>>> fail an empty cleanup.policy so that the user is aware of this
>>>>> likely
>>>>>>>>>>> mistake and provide a new option None for users wanting infinite
>>>>>>>>>> retention
>>>>>>>>>>> to opt in explicitly.
>>>>>>>>>>> 
>>>>>>>>>>> We could also consider supporting an empty cleanup.policy by
>>> fixing
>>>>>>>>> the
>>>>>>>>>>> issue in remote storage. But then the user may never realize
>> this
>>> if
>>>>>>>>>> it's a
>>>>>>>>>>> mistake.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com>
>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>> 
>>>>>>>>>>>> Comments:
>>>>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
>>> considered
>>>>>>>>>> invalid
>>>>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
>>>>>>>>>>>> Could you explain more about this? Why is this trade-off
>>> justified?
>>>>>>>>>>>> If we think the empty cleanup.policy is invalid in kafka, why
>> do
>>> we
>>>>>>>>>> provide
>>>>>>>>>>>> an additional "none" option to users in this KIP?
>>>>>>>>>>>> FYI, there are indeed use cases that need to keep all history
>>> data
>>>>>>>>>> without
>>>>>>>>>>>> a cleanup policy.
>>>>>>>>>>>> 
>>>>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
>>>>>>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles"
>>> configs
>>>>>>>>>> since
>>>>>>>>>>>> they are invalid.
>>>>>>>>>>>> But about "cleanup.policy", I don't see a persuasive reason why
>>> we
>>>>>>>>>> should
>>>>>>>>>>>> break backward compatibility to change it.
>>>>>>>>>>>> "This provides a clear and direct way to configure Kafka to
>>> retain
>>>>>>>>> all
>>>>>>>>>> log
>>>>>>>>>>>> segments without any automatic cleanup."
>>>>>>>>>>>> This is the motivation we mentioned in the KIP, but to me,
>>> backward
>>>>>>>>>>>> compatibility is much more important than "a clear and direct
>> way
>>>>> to
>>>>>>>>>> config
>>>>>>>>>>>> kafka".
>>>>>>>>>>>> Do we really have to change the "cleanup.policy" config?
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>> Luke
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
>> <jun@confluent.io.invalid
>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Jun, chia
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed in
>>> the
>>>>>>>>>>>> updated
>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I've removed all deprecated-related sections. Additionally,
>> an
>>>>>>>>>>>> exception
>>>>>>>>>>>>>> will now be thrown if a
>>>>>>>>>>>>>> developer passes an empty validStrings list to the inNonEmpty
>>>>>>>>>> method.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty config,
>>>>>>>>> so
>>>>>>>>>>>> should
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> be non-empty too?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception directly.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
>> 上午12:28寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the reply. There are still references to the
>>>>>>>>>> deprecated
>>>>>>>>>>>>>> method.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka
>> 5.0"
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed
>> in
>>>>>>>>>> the
>>>>>>>>>>>>>> updated
>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03
>> 寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan",
>>>>>>>>>> could you
>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>> description to remind readers that "clean.policy=" should
>>>>>>>>> be
>>>>>>>>>>>>> replaced
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete and
>>>>>>>>>> compact.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
>>>>>>>>> 上午1:55寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could
>> you
>>>>>>>>>> update
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>> completely? There are still references to deprecation.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
>>>>>>>>> method,
>>>>>>>>>> as
>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have
>>>>>>>>> both
>>>>>>>>>>>>> methods:
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
>>>>>>>>> that
>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
>> 凌晨12:50
>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
>>>>>>>>> that
>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>>>>>>>> start.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
>> s7133700@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain
>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
>>>>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove
>>>>>>>>> the
>>>>>>>>>>>>> boolean
>>>>>>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日
>> 凌晨4:17
>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
>>>>>>>>> the
>>>>>>>>>>>> method
>>>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
>>>>>>>>> true.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
>>>>>>>>> explicitly
>>>>>>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
>> validation
>>>>>>>>>> phase
>>>>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
>>>>>>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by this
>>>>>>>>>> change
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
>>>>>>>>> storage
>>>>>>>>>>>> format
>>>>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
>>>>>>>>> already
>>>>>>>>>> fails
>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>> process.roles
>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
>> storage
>>>>>>>>>> format
>>>>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will
>>>>>>>>> now
>>>>>>>>>> be
>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage format
>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
>>>>>>>>>> 凌晨4:44
>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With
>> this
>>>>>>>>>> KIP,
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
>>>>>>>>>> exception.
>>>>>>>>>>>> It
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
>>>>>>>>>> Chia-Ping.
>>>>>>>>>>>>> If
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
>>>>>>>>>> mistake
>>>>>>>>>>>> than
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user
>>>>>>>>> will
>>>>>>>>>> never
>>>>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user
>>>>>>>>>> realize
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>>>>>>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
>>>>>>>>>> maintaining
>>>>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we
>> don't
>>>>>>>>>> allow
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
>>>>>>>>>> throwing
>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
>> streamline
>>>>>>>>>> the
>>>>>>>>>>>> code
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
>> 2025年4月23日
>>>>>>>>> 週三
>>>>>>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or
>>>>>>>>>> compact
>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
>>>>>>>>> segments
>>>>>>>>>> based
>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
>>>>>>>>>> should
>>>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>>>>>>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when
>> remote
>>>>>>>>>>>> storage
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>>>>>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
>>>>>>>>>> revised
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be
>> used
>>>>>>>>>> with
>>>>>>>>>>>> any
>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
>>>>>>>>>> cleanup.policy,
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
>>>>>>>>>>>> Additionally, a
>>>>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>>>>> 2025年4月22日
>>>>>>>>>>>>> 凌晨4:52
>>>>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
>>>>>>>>> example,
>>>>>>>>>> None
>>>>>>>>>>>>>> can't
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
>>>>>>>>>> document
>>>>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when
>> an
>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what
>> the
>>>>>>>>>> empty
>>>>>>>>>>>>> check
>>>>>>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>>>>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>>>>>>>>> KIP-1161:
>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
>>>>>>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
>>>>>>>>>> empty
>>>>>>>>>>>> list
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

hi all,

JR11. There is an existing config interceptor.classes that allows an empty
> list and it makes intuitive sense. So, it seems that it's ok to support
> empty lists in general. Given that, it's probably simpler to just allow
> log.cleanup.policy to be empty and properly support it?


yes, that is what I meant before. Additionally, the `LIST` type should not
accept "null". Instead, we should use an empty list as the default value.
For example, `sasl.oauthbearer.expected.audience` should never has "null"
value. That could eliminate the null check.

For another, there is an additional issue which could be handled by this
KIP. Some configs have "string" type, but they are handled as a LIST. For
example, `early.start.listeners`, `security.providers`. Could we change
their type to LIST to benefit from this KIP? I mean to make them
non-nullable.

Best,
Chia-Ping



Jun Rao <ju...@confluent.io.invalid> 於 2025年7月8日 週二 上午5:56寫道：

> Hi, Jiunn-Yang,
>
> Thanks for the reply.
>
> JR10. Regarding supporting the null value, I am not sure how one could set
> a null value in a config file. For example, if you set the following in a
> config file.
>   configkey=
> The value of configkey will be an empty list, instead of null, right?
>
> Null could make sense as the default value, but it just means that the
> config key is not set explicitly.
>
> JR11. There is an existing config interceptor.classes that allows an empty
> list and it makes intuitive sense. So, it seems that it's ok to support
> empty lists in general. Given that, it's probably simpler to just allow
> log.cleanup.policy to be empty and properly support it?
>
> Jun
>
>
>
> On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hello Jun,
> >
> > JR10: The sasl.oauthbearer.expected.audience is optional, that is right,
> > so I think
> > that we can allow null value, but not allow the empty
> >
> > JR11: In this KIP, the goal is to address configurations that should not
> > accept empty arrays.
> > We want to prevent users from passing in an empty list when it's not
> > valid. I believe there are two
> > main scenarios to consider:
> >
> > 1. Nullable but not empty:
> > The parameter allows either a null value or a non-empty list. In this
> > case,
> > if the user provides an empty list, we should throw a ConfigException.
> >
> > 2. Non-null and non-empty only: The parameter must always contain a
> > non-empty list — null is also
> > invalid. In this case, if the user provides either a null or an empty
> > list, we should throw a ConfigException.
> >
> > Therefore, I want to prevent scenarios where users can pass in an empty
> > list.
> >
> > JR12: I’ve updated NonEmptyListValidator to include an allowNull
> property.
> > This allows us to specify
> > whether a configuration permits null values or not.
> >
> > JR13: I have updated the table
> >
> >
> > > Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> > >
> > > Hi, Jiunn-Yang,
> > >
> > > Sorry for the late reply. A few more comments.
> > >
> > > JR10. The doc for sasl.oauthbearer.expected.audience says that it's
> > > optional. So an empty list seems to be valid.
> > >
> > > JR11. Thinking a bit more. If we are going to support empty lists in
> > > general, it's probably more consistent to just allow log.cleanup.policy
> > to
> > > be empty as Luke suggested, instead of introducing a new option None.
> > >
> > > JR12. NonEmptyListValidator allows the input to be null. It seems that
> > > only ssl.cipher.suites allows null. The remaining ones shouldn't be
> null.
> > >
> > > JR13. The table is very helpful. Could you also describe the old/new
> > > behavior for each config with an empty list or duplicate entry?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > >
> > > On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >
> > >> Hello all,
> > >>
> > >> I am manually bumping this thread. Any feedback would be appreciated.
> > >>
> > >> Best regards,
> > >> Jiunn-Yang
> > >>
> > >>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> > >>>
> > >>> Hello Jun,
> > >>>
> > >>> I have updated the KIP and introduced a new validator,
> > >> NonEmptyListValidator, which ensures that the
> > >>> provided list is either null or a non-empty list without duplicate
> > >> entries. I’ve also listed the configurations that will
> > >>> be validated using this logic. Please feel free to share any feedback
> > or
> > >> suggestions.
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> > >>>>
> > >>>> Hi, Jiunn-Yang,
> > >>>>
> > >>>> It seems there are quite a few other configs of type list (e.g.
> > several
> > >> SSL
> > >>>> related ones). It would be useful to understand if empty lists are
> > valid
> > >>>> for them or not.
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:
> > >>>>
> > >>>>> Hi, Luke,
> > >>>>>
> > >>>>> Thanks for the reply.
> > >>>>>
> > >>>>> "My thought is that this behavior has been existed
> > >>>>> for years (maybe after "cleanup.policy" was introduced?), there
> could
> > >> be
> > >>>>> users relying on the empty cleanup.policy for a long time. "
> > >>>>>
> > >>>>> If you do a google search on "kafka infinite retention", you will
> get
> > >>>>> links like
> > >>>>>
> > >>
> >
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> > >>>>> and
> > >>>>>
> > >>
> >
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> > >> ,
> > >>>>> both pointing to setting the retention time and retention size to
> -1
> > to
> > >>>>> achieve this goal. So, it seems that if a user intentionally wants
> > >> infinite
> > >>>>> retention, it's more likely they will use that approach instead of
> > >> setting
> > >>>>> cleanup.policy to empty. On the other hand, our API/tools make it
> > >> possible
> > >>>>> for users to make a mistake by inadvertently leaving the config
> value
> > >> empty.
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Jun
> > >>>>>
> > >>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com>
> > wrote:
> > >>>>>
> > >>>>>> Hi Jun,
> > >>>>>>
> > >>>>>>> Regarding the motivation, currently, we never documented that an
> > >> empty
> > >>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
> > >>>>>> cleanup.policy empty, it actually breaks remote storage since it
> > will
> > >> stop
> > >>>>>> the cleaning based on local retention size and time too. So,
> leaving
> > >>>>>> cleanup.policy empty is probably more like a user mistake than an
> > >>>>>> intentional choice. Based on this assumption, the idea behind the
> > KIP
> > >> is
> > >>>>>> to
> > >>>>>> fail an empty cleanup.policy so that the user is aware of this
> > likely
> > >>>>>> mistake and provide a new option None for users wanting infinite
> > >> retention
> > >>>>>> to opt in explicitly.
> > >>>>>>
> > >>>>>> While we never documented the empty cleanup.policy implies
> infinite
> > >>>>>> retention, we also never documented (or failed) the empty
> > >> cleanup.policy
> > >>>>>> is
> > >>>>>> an invalid configuration. My thought is that this behavior has
> been
> > >>>>>> existed
> > >>>>>> for years (maybe after "cleanup.policy" was introduced?), there
> > could
> > >> be
> > >>>>>> users relying on the empty cleanup.policy for a long time. It is
> not
> > >> good
> > >>>>>> to break the backward compatibility in a minor release version
> (ex:
> > >>>>>> v4.1.0). Even though we think we are fixing a long existing bug,
> it
> > >> could
> > >>>>>> be a surprise to users.
> > >>>>>>
> > >>>>>>
> > >>>>>>> We could also consider supporting an empty cleanup.policy by
> fixing
> > >> the
> > >>>>>> issue in remote storage. But then the user may never realize this
> if
> > >> it's
> > >>>>>> a
> > >>>>>> mistake.
> > >>>>>>
> > >>>>>> Good point! I agree we need to fix it!
> > >>>>>>
> > >>>>>>
> > >>>>>> Hi Chia-Ping,
> > >>>>>>
> > >>>>>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0
> we
> > >>>>>> throw
> > >>>>>> exception directly to make it fail fast
> > >>>>>>
> > >>>>>> This suggestion looks good to me!
> > >>>>>>
> > >>>>>> Thank you.
> > >>>>>> Luke
> > >>>>>>
> > >>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <
> chia7712@apache.org
> > >
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> hi all,
> > >>>>>>>
> > >>>>>>> Given Luke mentioned the existence of use cases, it is worth
> > >> considering
> > >>>>>>> the compatibility issue. In fact, I had previously encountered
> this
> > >> use
> > >>>>>>> case but advised customers to utilize retention configurations
> > >> instead.
> > >>>>>>>
> > >>>>>>>> We could also consider supporting an empty cleanup.policy by
> > fixing
> > >>>>>> the
> > >>>>>>>> issue in remote storage. But then the user may never realize
> this
> > if
> > >>>>>>> it's a
> > >>>>>>>> mistake.
> > >>>>>>>
> > >>>>>>> Good catch! The "empty" or "none" wouldn't make sense for remote
> > >>>>>> storage.
> > >>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage use
> a
> > >> valid
> > >>>>>>> delete policy.
> > >>>>>>>
> > >>>>>>> Best,
> > >>>>>>> Chia-Ping
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> > >>>>>>>> Hi, Luke,
> > >>>>>>>>
> > >>>>>>>> Regarding the motivation, currently, we never documented that an
> > >> empty
> > >>>>>>>> cleanup.policy implies infinite retention. In fact, if one
> leaves
> > >>>>>>>> cleanup.policy empty, it actually breaks remote storage since it
> > >> will
> > >>>>>>> stop
> > >>>>>>>> the cleaning based on local retention size and time too. So,
> > leaving
> > >>>>>>>> cleanup.policy empty is probably more like a user mistake than
> an
> > >>>>>>>> intentional choice. Based on this assumption, the idea behind
> the
> > >> KIP
> > >>>>>> is
> > >>>>>>> to
> > >>>>>>>> fail an empty cleanup.policy so that the user is aware of this
> > >> likely
> > >>>>>>>> mistake and provide a new option None for users wanting infinite
> > >>>>>>> retention
> > >>>>>>>> to opt in explicitly.
> > >>>>>>>>
> > >>>>>>>> We could also consider supporting an empty cleanup.policy by
> > fixing
> > >>>>>> the
> > >>>>>>>> issue in remote storage. But then the user may never realize
> this
> > if
> > >>>>>>> it's a
> > >>>>>>>> mistake.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com>
> > >> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>
> > >>>>>>>>> Comments:
> > >>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
> > considered
> > >>>>>>> invalid
> > >>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> > >>>>>>>>> Could you explain more about this? Why is this trade-off
> > justified?
> > >>>>>>>>> If we think the empty cleanup.policy is invalid in kafka, why
> do
> > we
> > >>>>>>> provide
> > >>>>>>>>> an additional "none" option to users in this KIP?
> > >>>>>>>>> FYI, there are indeed use cases that need to keep all history
> > data
> > >>>>>>> without
> > >>>>>>>>> a cleanup policy.
> > >>>>>>>>>
> > >>>>>>>>> 2. Following (1), I agree we should fail fast for empty
> > >>>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles"
> > configs
> > >>>>>>> since
> > >>>>>>>>> they are invalid.
> > >>>>>>>>> But about "cleanup.policy", I don't see a persuasive reason why
> > we
> > >>>>>>> should
> > >>>>>>>>> break backward compatibility to change it.
> > >>>>>>>>> "This provides a clear and direct way to configure Kafka to
> > retain
> > >>>>>> all
> > >>>>>>> log
> > >>>>>>>>> segments without any automatic cleanup."
> > >>>>>>>>> This is the motivation we mentioned in the KIP, but to me,
> > backward
> > >>>>>>>>> compatibility is much more important than "a clear and direct
> way
> > >> to
> > >>>>>>> config
> > >>>>>>>>> kafka".
> > >>>>>>>>> Do we really have to change the "cleanup.policy" config?
> > >>>>>>>>>
> > >>>>>>>>> Thanks.
> > >>>>>>>>> Luke
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao
> <jun@confluent.io.invalid
> > >
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Jun, chia
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for all the comments. They have all been addressed in
> > the
> > >>>>>>>>> updated
> > >>>>>>>>>>> KIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I've removed all deprecated-related sections. Additionally,
> an
> > >>>>>>>>> exception
> > >>>>>>>>>>> will now be thrown if a
> > >>>>>>>>>>> developer passes an empty validStrings list to the inNonEmpty
> > >>>>>>> method.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best Regards,
> > >>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty config,
> > >>>>>> so
> > >>>>>>>>> should
> > >>>>>>>>>> it
> > >>>>>>>>>>>> be non-empty too?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> For example, `inNonEmpty()` should throw exception directly.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二
> 上午12:28寫道：
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the reply. There are still references to the
> > >>>>>>> deprecated
> > >>>>>>>>>>> method.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> "stop relying on the deprecated methods"
> > >>>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka
> 5.0"
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed
> in
> > >>>>>>> the
> > >>>>>>>>>>> updated
> > >>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03
> 寫道：
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan",
> > >>>>>>> could you
> > >>>>>>>>>> add
> > >>>>>>>>>>>>>>> description to remind readers that "clean.policy=" should
> > >>>>>> be
> > >>>>>>>>>> replaced
> > >>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete and
> > >>>>>>> compact.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
> > >>>>>> 上午1:55寫道：
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could
> you
> > >>>>>>> update
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>> completely? There are still references to deprecation.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
> > >>>>>> method,
> > >>>>>>> as
> > >>>>>>>>>> there
> > >>>>>>>>>>>>> may
> > >>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> future scenarios where we want
> > >>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have
> > >>>>>> both
> > >>>>>>>>>> methods:
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>> (which allows empty lists)
> > >>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
> > >>>>>> that
> > >>>>>>>>> during
> > >>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> > >>>>>> start.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日
> 凌晨12:50
> > >>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
> > >>>>>> that
> > >>>>>>>>> during
> > >>>>>>>>>>>>>>>> upgrade,
> > >>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> > >>>>>> start.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <
> s7133700@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain
> > >>>>>>> backward
> > >>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> > >>>>>>>>> necessary.
> > >>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove
> > >>>>>> the
> > >>>>>>>>>> boolean
> > >>>>>>>>>>>>>>>>>>> parameter.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日
> 凌晨4:17
> > >>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
> > >>>>>> the
> > >>>>>>>>> method
> > >>>>>>>>>>>>> name
> > >>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
> > >>>>>> true.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> > >>>>>> s7133700@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
> > >>>>>> invalid
> > >>>>>>>>>>>>>>>>> configurations
> > >>>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> > >>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
> > >>>>>> explicitly
> > >>>>>>>>>> throw a
> > >>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format
> validation
> > >>>>>>> phase
> > >>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
> > >>>>>>> 凌晨1:17 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by this
> > >>>>>>> change
> > >>>>>>>>> is
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
> > >>>>>> storage
> > >>>>>>>>> format
> > >>>>>>>>>>>>>>>>>>> validation
> > >>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
> > >>>>>> already
> > >>>>>>> fails
> > >>>>>>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>> process.roles
> > >>>>>>>>> are
> > >>>>>>>>>>>>>> empty.
> > >>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>> just
> > >>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> > >>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> - Validation for
> > >>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>> will be
> > >>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the
> storage
> > >>>>>>> format
> > >>>>>>>>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will
> > >>>>>> now
> > >>>>>>> be
> > >>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>> invalid
> > >>>>>>>>>>>>>>>>>>>>>>> and will
> > >>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage format
> > >>>>>>> phase.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
> > >>>>>>> 凌晨4:44
> > >>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> > >>>>>>> group.coordinator.rebalance.protocols
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With
> this
> > >>>>>>> KIP,
> > >>>>>>>>> the
> > >>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
> > >>>>>>> exception.
> > >>>>>>>>> It
> > >>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>> useful
> > >>>>>>>>>>>>>>>>>>>>>>>> to document that.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
> > >>>>>>> Chia-Ping.
> > >>>>>>>>>> If
> > >>>>>>>>>>> a
> > >>>>>>>>>>>>>>>> user
> > >>>>>>>>>>>>>>>>>>>>>>> leaves
> > >>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
> > >>>>>>> mistake
> > >>>>>>>>> than
> > >>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>> intention.
> > >>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user
> > >>>>>> will
> > >>>>>>> never
> > >>>>>>>>>>>>>> realize
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user
> > >>>>>>> realize
> > >>>>>>>>>>> there
> > >>>>>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> > >>>>>>>>>>>>>>>> chia7712@gmail.com
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> > >>>>>>> maintaining
> > >>>>>>>>>>>>>>>>> compatibility
> > >>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>> important.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we
> don't
> > >>>>>>> allow
> > >>>>>>>>> an
> > >>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>> value
> > >>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
> > >>>>>>> throwing
> > >>>>>>>>> an
> > >>>>>>>>>>>>>>>>> exception
> > >>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could
> streamline
> > >>>>>>> the
> > >>>>>>>>> code
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> avoid
> > >>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於
> 2025年4月23日
> > >>>>>> 週三
> > >>>>>>>>>>> 上午6:33寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or
> > >>>>>>> compact
> > >>>>>>>>> any
> > >>>>>>>>>>>>>>>>>>> segments",
> > >>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
> > >>>>>> segments
> > >>>>>>> based
> > >>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
> > >>>>>>> should
> > >>>>>>>>>>>>> continue
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> delete
> > >>>>>>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> > >>>>>>>>>>>>> log.retention.ms.
> > >>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when
> remote
> > >>>>>>>>> storage
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>> enabled.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> > >>>>>>>>>> jun@confluent.io>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > >>>>>>>>> process.roles
> > >>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>> upgrade?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> > >>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
> > >>>>>>> revised
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be
> used
> > >>>>>>> with
> > >>>>>>>>> any
> > >>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>> policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> > >>>>>>> cleanup.policy,
> > >>>>>>>>> the
> > >>>>>>>>>>>>>>>> approach
> > >>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> > >>>>>>>>> Additionally, a
> > >>>>>>>>>>>>>>>> warning
> > >>>>>>>>>>>>>>>>>>>>>>>>> message
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> > >>>>>> 2025年4月22日
> > >>>>>>>>>> 凌晨4:52
> > >>>>>>>>>>>>> 寫道：
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
> > >>>>>> for
> > >>>>>>>>>>>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>> But
> > >>>>>>>>>>>>>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
> > >>>>>> example,
> > >>>>>>> None
> > >>>>>>>>>>> can't
> > >>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> > >>>>>>> document
> > >>>>>>>>>> that.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when
> an
> > >>>>>>>>> existing
> > >>>>>>>>>>>>>>>> config
> > >>>>>>>>>>>>>>>>>>> has
> > >>>>>>>>>>>>>>>>>>>>> an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what
> the
> > >>>>>>> empty
> > >>>>>>>>>> check
> > >>>>>>>>>>>>>>>> does.
> > >>>>>>>>>>>>>>>>>>> How
> > >>>>>>>>>>>>>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> > >>>>>>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> > >>>>>> KIP-1161:
> > >>>>>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> > >>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>> configuration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> > >>>>>> improvements:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
> > >>>>>>> empty
> > >>>>>>>>> list
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>> allowed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> > >>>>>>> cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply.

JR10. Regarding supporting the null value, I am not sure how one could set
a null value in a config file. For example, if you set the following in a
config file.
  configkey=
The value of configkey will be an empty list, instead of null, right?

Null could make sense as the default value, but it just means that the
config key is not set explicitly.

JR11. There is an existing config interceptor.classes that allows an empty
list and it makes intuitive sense. So, it seems that it's ok to support
empty lists in general. Given that, it's probably simpler to just allow
log.cleanup.policy to be empty and properly support it?

Jun



On Fri, Jul 4, 2025 at 5:13 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> JR10: The sasl.oauthbearer.expected.audience is optional, that is right,
> so I think
> that we can allow null value, but not allow the empty
>
> JR11: In this KIP, the goal is to address configurations that should not
> accept empty arrays.
> We want to prevent users from passing in an empty list when it's not
> valid. I believe there are two
> main scenarios to consider:
>
> 1. Nullable but not empty:
> The parameter allows either a null value or a non-empty list. In this
> case,
> if the user provides an empty list, we should throw a ConfigException.
>
> 2. Non-null and non-empty only: The parameter must always contain a
> non-empty list — null is also
> invalid. In this case, if the user provides either a null or an empty
> list, we should throw a ConfigException.
>
> Therefore, I want to prevent scenarios where users can pass in an empty
> list.
>
> JR12: I’ve updated NonEmptyListValidator to include an allowNull property.
> This allows us to specify
> whether a configuration permits null values or not.
>
> JR13: I have updated the table
>
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Sorry for the late reply. A few more comments.
> >
> > JR10. The doc for sasl.oauthbearer.expected.audience says that it's
> > optional. So an empty list seems to be valid.
> >
> > JR11. Thinking a bit more. If we are going to support empty lists in
> > general, it's probably more consistent to just allow log.cleanup.policy
> to
> > be empty as Luke suggested, instead of introducing a new option None.
> >
> > JR12. NonEmptyListValidator allows the input to be null. It seems that
> > only ssl.cipher.suites allows null. The remaining ones shouldn't be null.
> >
> > JR13. The table is very helpful. Could you also describe the old/new
> > behavior for each config with an empty list or duplicate entry?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello all,
> >>
> >> I am manually bumping this thread. Any feedback would be appreciated.
> >>
> >> Best regards,
> >> Jiunn-Yang
> >>
> >>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> >>>
> >>> Hello Jun,
> >>>
> >>> I have updated the KIP and introduced a new validator,
> >> NonEmptyListValidator, which ensures that the
> >>> provided list is either null or a non-empty list without duplicate
> >> entries. I’ve also listed the configurations that will
> >>> be validated using this logic. Please feel free to share any feedback
> or
> >> suggestions.
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> >>>>
> >>>> Hi, Jiunn-Yang,
> >>>>
> >>>> It seems there are quite a few other configs of type list (e.g.
> several
> >> SSL
> >>>> related ones). It would be useful to understand if empty lists are
> valid
> >>>> for them or not.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:
> >>>>
> >>>>> Hi, Luke,
> >>>>>
> >>>>> Thanks for the reply.
> >>>>>
> >>>>> "My thought is that this behavior has been existed
> >>>>> for years (maybe after "cleanup.policy" was introduced?), there could
> >> be
> >>>>> users relying on the empty cleanup.policy for a long time. "
> >>>>>
> >>>>> If you do a google search on "kafka infinite retention", you will get
> >>>>> links like
> >>>>>
> >>
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> >>>>> and
> >>>>>
> >>
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> >> ,
> >>>>> both pointing to setting the retention time and retention size to -1
> to
> >>>>> achieve this goal. So, it seems that if a user intentionally wants
> >> infinite
> >>>>> retention, it's more likely they will use that approach instead of
> >> setting
> >>>>> cleanup.policy to empty. On the other hand, our API/tools make it
> >> possible
> >>>>> for users to make a mistake by inadvertently leaving the config value
> >> empty.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com>
> wrote:
> >>>>>
> >>>>>> Hi Jun,
> >>>>>>
> >>>>>>> Regarding the motivation, currently, we never documented that an
> >> empty
> >>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
> >>>>>> cleanup.policy empty, it actually breaks remote storage since it
> will
> >> stop
> >>>>>> the cleaning based on local retention size and time too. So, leaving
> >>>>>> cleanup.policy empty is probably more like a user mistake than an
> >>>>>> intentional choice. Based on this assumption, the idea behind the
> KIP
> >> is
> >>>>>> to
> >>>>>> fail an empty cleanup.policy so that the user is aware of this
> likely
> >>>>>> mistake and provide a new option None for users wanting infinite
> >> retention
> >>>>>> to opt in explicitly.
> >>>>>>
> >>>>>> While we never documented the empty cleanup.policy implies infinite
> >>>>>> retention, we also never documented (or failed) the empty
> >> cleanup.policy
> >>>>>> is
> >>>>>> an invalid configuration. My thought is that this behavior has been
> >>>>>> existed
> >>>>>> for years (maybe after "cleanup.policy" was introduced?), there
> could
> >> be
> >>>>>> users relying on the empty cleanup.policy for a long time. It is not
> >> good
> >>>>>> to break the backward compatibility in a minor release version (ex:
> >>>>>> v4.1.0). Even though we think we are fixing a long existing bug, it
> >> could
> >>>>>> be a surprise to users.
> >>>>>>
> >>>>>>
> >>>>>>> We could also consider supporting an empty cleanup.policy by fixing
> >> the
> >>>>>> issue in remote storage. But then the user may never realize this if
> >> it's
> >>>>>> a
> >>>>>> mistake.
> >>>>>>
> >>>>>> Good point! I agree we need to fix it!
> >>>>>>
> >>>>>>
> >>>>>> Hi Chia-Ping,
> >>>>>>
> >>>>>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0 we
> >>>>>> throw
> >>>>>> exception directly to make it fail fast
> >>>>>>
> >>>>>> This suggestion looks good to me!
> >>>>>>
> >>>>>> Thank you.
> >>>>>> Luke
> >>>>>>
> >>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <chia7712@apache.org
> >
> >>>>>> wrote:
> >>>>>>
> >>>>>>> hi all,
> >>>>>>>
> >>>>>>> Given Luke mentioned the existence of use cases, it is worth
> >> considering
> >>>>>>> the compatibility issue. In fact, I had previously encountered this
> >> use
> >>>>>>> case but advised customers to utilize retention configurations
> >> instead.
> >>>>>>>
> >>>>>>>> We could also consider supporting an empty cleanup.policy by
> fixing
> >>>>>> the
> >>>>>>>> issue in remote storage. But then the user may never realize this
> if
> >>>>>>> it's a
> >>>>>>>> mistake.
> >>>>>>>
> >>>>>>> Good catch! The "empty" or "none" wouldn't make sense for remote
> >>>>>> storage.
> >>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage use a
> >> valid
> >>>>>>> delete policy.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Chia-Ping
> >>>>>>>
> >>>>>>>
> >>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> >>>>>>>> Hi, Luke,
> >>>>>>>>
> >>>>>>>> Regarding the motivation, currently, we never documented that an
> >> empty
> >>>>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
> >>>>>>>> cleanup.policy empty, it actually breaks remote storage since it
> >> will
> >>>>>>> stop
> >>>>>>>> the cleaning based on local retention size and time too. So,
> leaving
> >>>>>>>> cleanup.policy empty is probably more like a user mistake than an
> >>>>>>>> intentional choice. Based on this assumption, the idea behind the
> >> KIP
> >>>>>> is
> >>>>>>> to
> >>>>>>>> fail an empty cleanup.policy so that the user is aware of this
> >> likely
> >>>>>>>> mistake and provide a new option None for users wanting infinite
> >>>>>>> retention
> >>>>>>>> to opt in explicitly.
> >>>>>>>>
> >>>>>>>> We could also consider supporting an empty cleanup.policy by
> fixing
> >>>>>> the
> >>>>>>>> issue in remote storage. But then the user may never realize this
> if
> >>>>>>> it's a
> >>>>>>>> mistake.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com>
> >> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>
> >>>>>>>>> Thanks for the KIP.
> >>>>>>>>>
> >>>>>>>>> Comments:
> >>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is
> considered
> >>>>>>> invalid
> >>>>>>>>> in Kafka. Therefore, this trade-off is justified."
> >>>>>>>>> Could you explain more about this? Why is this trade-off
> justified?
> >>>>>>>>> If we think the empty cleanup.policy is invalid in kafka, why do
> we
> >>>>>>> provide
> >>>>>>>>> an additional "none" option to users in this KIP?
> >>>>>>>>> FYI, there are indeed use cases that need to keep all history
> data
> >>>>>>> without
> >>>>>>>>> a cleanup policy.
> >>>>>>>>>
> >>>>>>>>> 2. Following (1), I agree we should fail fast for empty
> >>>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles"
> configs
> >>>>>>> since
> >>>>>>>>> they are invalid.
> >>>>>>>>> But about "cleanup.policy", I don't see a persuasive reason why
> we
> >>>>>>> should
> >>>>>>>>> break backward compatibility to change it.
> >>>>>>>>> "This provides a clear and direct way to configure Kafka to
> retain
> >>>>>> all
> >>>>>>> log
> >>>>>>>>> segments without any automatic cleanup."
> >>>>>>>>> This is the motivation we mentioned in the KIP, but to me,
> backward
> >>>>>>>>> compatibility is much more important than "a clear and direct way
> >> to
> >>>>>>> config
> >>>>>>>>> kafka".
> >>>>>>>>> Do we really have to change the "cleanup.policy" config?
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>> Luke
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <jun@confluent.io.invalid
> >
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the updated KIP. It looks good to me.
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Jun, chia
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for all the comments. They have all been addressed in
> the
> >>>>>>>>> updated
> >>>>>>>>>>> KIP.
> >>>>>>>>>>>
> >>>>>>>>>>> I've removed all deprecated-related sections. Additionally, an
> >>>>>>>>> exception
> >>>>>>>>>>> will now be thrown if a
> >>>>>>>>>>> developer passes an empty validStrings list to the inNonEmpty
> >>>>>>> method.
> >>>>>>>>>>>
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>
> >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> >>>>>>>>>>>>
> >>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty config,
> >>>>>> so
> >>>>>>>>> should
> >>>>>>>>>> it
> >>>>>>>>>>>> be non-empty too?
> >>>>>>>>>>>>
> >>>>>>>>>>>> For example, `inNonEmpty()` should throw exception directly.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the reply. There are still references to the
> >>>>>>> deprecated
> >>>>>>>>>>> method.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> "stop relying on the deprecated methods"
> >>>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed in
> >>>>>>> the
> >>>>>>>>>>> updated
> >>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan",
> >>>>>>> could you
> >>>>>>>>>> add
> >>>>>>>>>>>>>>> description to remind readers that "clean.policy=" should
> >>>>>> be
> >>>>>>>>>> replaced
> >>>>>>>>>>>>> by
> >>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete and
> >>>>>>> compact.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
> >>>>>> 上午1:55寫道：
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you
> >>>>>>> update
> >>>>>>>>>> the
> >>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>> completely? There are still references to deprecation.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
> >>>>>> method,
> >>>>>>> as
> >>>>>>>>>> there
> >>>>>>>>>>>>> may
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> future scenarios where we want
> >>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have
> >>>>>> both
> >>>>>>>>>> methods:
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> (which allows empty lists)
> >>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
> >>>>>> that
> >>>>>>>>> during
> >>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> >>>>>> start.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50
> >>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
> >>>>>> that
> >>>>>>>>> during
> >>>>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> >>>>>> start.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain
> >>>>>>> backward
> >>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> >>>>>>>>> necessary.
> >>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove
> >>>>>> the
> >>>>>>>>>> boolean
> >>>>>>>>>>>>>>>>>>> parameter.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
> >>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
> >>>>>> the
> >>>>>>>>> method
> >>>>>>>>>>>>> name
> >>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
> >>>>>> true.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> >>>>>> s7133700@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
> >>>>>> invalid
> >>>>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
> >>>>>> explicitly
> >>>>>>>>>> throw a
> >>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format validation
> >>>>>>> phase
> >>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
> >>>>>>> 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by this
> >>>>>>> change
> >>>>>>>>> is
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
> >>>>>> storage
> >>>>>>>>> format
> >>>>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
> >>>>>> already
> >>>>>>> fails
> >>>>>>>>>> if
> >>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>> process.roles
> >>>>>>>>> are
> >>>>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> >>>>>>> s7133700@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> - Validation for
> >>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the storage
> >>>>>>> format
> >>>>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will
> >>>>>> now
> >>>>>>> be
> >>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage format
> >>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
> >>>>>>> 凌晨4:44
> >>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> >>>>>>> group.coordinator.rebalance.protocols
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With this
> >>>>>>> KIP,
> >>>>>>>>> the
> >>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
> >>>>>>> exception.
> >>>>>>>>> It
> >>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
> >>>>>>> Chia-Ping.
> >>>>>>>>>> If
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
> >>>>>>> mistake
> >>>>>>>>> than
> >>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user
> >>>>>> will
> >>>>>>> never
> >>>>>>>>>>>>>> realize
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user
> >>>>>>> realize
> >>>>>>>>>>> there
> >>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> >>>>>>>>>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> >>>>>>> maintaining
> >>>>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we don't
> >>>>>>> allow
> >>>>>>>>> an
> >>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
> >>>>>>> throwing
> >>>>>>>>> an
> >>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline
> >>>>>>> the
> >>>>>>>>> code
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日
> >>>>>> 週三
> >>>>>>>>>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or
> >>>>>>> compact
> >>>>>>>>> any
> >>>>>>>>>>>>>>>>>>> segments",
> >>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
> >>>>>> segments
> >>>>>>> based
> >>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
> >>>>>>> should
> >>>>>>>>>>>>> continue
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> >>>>>>>>>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote
> >>>>>>>>> storage
> >>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> >>>>>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>> process.roles
> >>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> >>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
> >>>>>>> revised
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
> >>>>>>> with
> >>>>>>>>> any
> >>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> >>>>>>> cleanup.policy,
> >>>>>>>>> the
> >>>>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> >>>>>>>>> Additionally, a
> >>>>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>>>> 2025年4月22日
> >>>>>>>>>> 凌晨4:52
> >>>>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
> >>>>>> for
> >>>>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
> >>>>>> example,
> >>>>>>> None
> >>>>>>>>>>> can't
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> >>>>>>> document
> >>>>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
> >>>>>>>>> existing
> >>>>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
> >>>>>>> empty
> >>>>>>>>>> check
> >>>>>>>>>>>>>>>> does.
> >>>>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> >>>>>>>>>> s7133700@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> >>>>>> KIP-1161:
> >>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>> https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> >>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> >>>>>> improvements:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
> >>>>>>> empty
> >>>>>>>>> list
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> >>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

JR10: The sasl.oauthbearer.expected.audience is optional, that is right, so I think
that we can allow null value, but not allow the empty 

JR11: In this KIP, the goal is to address configurations that should not accept empty arrays. 
We want to prevent users from passing in an empty list when it's not valid. I believe there are two 
main scenarios to consider: 

1. Nullable but not empty: 
The parameter allows either a null value or a non-empty list. In this case, 
if the user provides an empty list, we should throw a ConfigException. 

2. Non-null and non-empty only: The parameter must always contain a non-empty list — null is also 
invalid. In this case, if the user provides either a null or an empty list, we should throw a ConfigException.

Therefore, I want to prevent scenarios where users can pass in an empty list.

JR12: I’ve updated NonEmptyListValidator to include an allowNull property. This allows us to specify 
whether a configuration permits null values or not.

JR13: I have updated the table


> Jun Rao <ju...@confluent.io.INVALID> 於 2025年7月1日 清晨6:30 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Sorry for the late reply. A few more comments.
> 
> JR10. The doc for sasl.oauthbearer.expected.audience says that it's
> optional. So an empty list seems to be valid.
> 
> JR11. Thinking a bit more. If we are going to support empty lists in
> general, it's probably more consistent to just allow log.cleanup.policy to
> be empty as Luke suggested, instead of introducing a new option None.
> 
> JR12. NonEmptyListValidator allows the input to be null. It seems that
> only ssl.cipher.suites allows null. The remaining ones shouldn't be null.
> 
> JR13. The table is very helpful. Could you also describe the old/new
> behavior for each config with an empty list or duplicate entry?
> 
> Thanks,
> 
> Jun
> 
> 
> 
> On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello all,
>> 
>> I am manually bumping this thread. Any feedback would be appreciated.
>> 
>> Best regards,
>> Jiunn-Yang
>> 
>>> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
>>> 
>>> Hello Jun,
>>> 
>>> I have updated the KIP and introduced a new validator,
>> NonEmptyListValidator, which ensures that the
>>> provided list is either null or a non-empty list without duplicate
>> entries. I’ve also listed the configurations that will
>>> be validated using this logic. Please feel free to share any feedback or
>> suggestions.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
>>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> It seems there are quite a few other configs of type list (e.g. several
>> SSL
>>>> related ones). It would be useful to understand if empty lists are valid
>>>> for them or not.
>>>> 
>>>> Thanks,
>>>> 
>>>> Jun
>>>> 
>>>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:
>>>> 
>>>>> Hi, Luke,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> "My thought is that this behavior has been existed
>>>>> for years (maybe after "cleanup.policy" was introduced?), there could
>> be
>>>>> users relying on the empty cleanup.policy for a long time. "
>>>>> 
>>>>> If you do a google search on "kafka infinite retention", you will get
>>>>> links like
>>>>> 
>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>>>> and
>>>>> 
>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
>> ,
>>>>> both pointing to setting the retention time and retention size to -1 to
>>>>> achieve this goal. So, it seems that if a user intentionally wants
>> infinite
>>>>> retention, it's more likely they will use that approach instead of
>> setting
>>>>> cleanup.policy to empty. On the other hand, our API/tools make it
>> possible
>>>>> for users to make a mistake by inadvertently leaving the config value
>> empty.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com> wrote:
>>>>> 
>>>>>> Hi Jun,
>>>>>> 
>>>>>>> Regarding the motivation, currently, we never documented that an
>> empty
>>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
>>>>>> cleanup.policy empty, it actually breaks remote storage since it will
>> stop
>>>>>> the cleaning based on local retention size and time too. So, leaving
>>>>>> cleanup.policy empty is probably more like a user mistake than an
>>>>>> intentional choice. Based on this assumption, the idea behind the KIP
>> is
>>>>>> to
>>>>>> fail an empty cleanup.policy so that the user is aware of this likely
>>>>>> mistake and provide a new option None for users wanting infinite
>> retention
>>>>>> to opt in explicitly.
>>>>>> 
>>>>>> While we never documented the empty cleanup.policy implies infinite
>>>>>> retention, we also never documented (or failed) the empty
>> cleanup.policy
>>>>>> is
>>>>>> an invalid configuration. My thought is that this behavior has been
>>>>>> existed
>>>>>> for years (maybe after "cleanup.policy" was introduced?), there could
>> be
>>>>>> users relying on the empty cleanup.policy for a long time. It is not
>> good
>>>>>> to break the backward compatibility in a minor release version (ex:
>>>>>> v4.1.0). Even though we think we are fixing a long existing bug, it
>> could
>>>>>> be a surprise to users.
>>>>>> 
>>>>>> 
>>>>>>> We could also consider supporting an empty cleanup.policy by fixing
>> the
>>>>>> issue in remote storage. But then the user may never realize this if
>> it's
>>>>>> a
>>>>>> mistake.
>>>>>> 
>>>>>> Good point! I agree we need to fix it!
>>>>>> 
>>>>>> 
>>>>>> Hi Chia-Ping,
>>>>>> 
>>>>>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0 we
>>>>>> throw
>>>>>> exception directly to make it fail fast
>>>>>> 
>>>>>> This suggestion looks good to me!
>>>>>> 
>>>>>> Thank you.
>>>>>> Luke
>>>>>> 
>>>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <ch...@apache.org>
>>>>>> wrote:
>>>>>> 
>>>>>>> hi all,
>>>>>>> 
>>>>>>> Given Luke mentioned the existence of use cases, it is worth
>> considering
>>>>>>> the compatibility issue. In fact, I had previously encountered this
>> use
>>>>>>> case but advised customers to utilize retention configurations
>> instead.
>>>>>>> 
>>>>>>>> We could also consider supporting an empty cleanup.policy by fixing
>>>>>> the
>>>>>>>> issue in remote storage. But then the user may never realize this if
>>>>>>> it's a
>>>>>>>> mistake.
>>>>>>> 
>>>>>>> Good catch! The "empty" or "none" wouldn't make sense for remote
>>>>>> storage.
>>>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage use a
>> valid
>>>>>>> delete policy.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>> 
>>>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>>>> Hi, Luke,
>>>>>>>> 
>>>>>>>> Regarding the motivation, currently, we never documented that an
>> empty
>>>>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
>>>>>>>> cleanup.policy empty, it actually breaks remote storage since it
>> will
>>>>>>> stop
>>>>>>>> the cleaning based on local retention size and time too. So, leaving
>>>>>>>> cleanup.policy empty is probably more like a user mistake than an
>>>>>>>> intentional choice. Based on this assumption, the idea behind the
>> KIP
>>>>>> is
>>>>>>> to
>>>>>>>> fail an empty cleanup.policy so that the user is aware of this
>> likely
>>>>>>>> mistake and provide a new option None for users wanting infinite
>>>>>>> retention
>>>>>>>> to opt in explicitly.
>>>>>>>> 
>>>>>>>> We could also consider supporting an empty cleanup.policy by fixing
>>>>>> the
>>>>>>>> issue in remote storage. But then the user may never realize this if
>>>>>>> it's a
>>>>>>>> mistake.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com>
>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Thanks for the KIP.
>>>>>>>>> 
>>>>>>>>> Comments:
>>>>>>>>> 1. "it is acceptable because an empty cleanup.policy is considered
>>>>>>> invalid
>>>>>>>>> in Kafka. Therefore, this trade-off is justified."
>>>>>>>>> Could you explain more about this? Why is this trade-off justified?
>>>>>>>>> If we think the empty cleanup.policy is invalid in kafka, why do we
>>>>>>> provide
>>>>>>>>> an additional "none" option to users in this KIP?
>>>>>>>>> FYI, there are indeed use cases that need to keep all history data
>>>>>>> without
>>>>>>>>> a cleanup policy.
>>>>>>>>> 
>>>>>>>>> 2. Following (1), I agree we should fail fast for empty
>>>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles" configs
>>>>>>> since
>>>>>>>>> they are invalid.
>>>>>>>>> But about "cleanup.policy", I don't see a persuasive reason why we
>>>>>>> should
>>>>>>>>> break backward compatibility to change it.
>>>>>>>>> "This provides a clear and direct way to configure Kafka to retain
>>>>>> all
>>>>>>> log
>>>>>>>>> segments without any automatic cleanup."
>>>>>>>>> This is the motivation we mentioned in the KIP, but to me, backward
>>>>>>>>> compatibility is much more important than "a clear and direct way
>> to
>>>>>>> config
>>>>>>>>> kafka".
>>>>>>>>> Do we really have to change the "cleanup.policy" config?
>>>>>>>>> 
>>>>>>>>> Thanks.
>>>>>>>>> Luke
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the updated KIP. It looks good to me.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Jun, chia
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for all the comments. They have all been addressed in the
>>>>>>>>> updated
>>>>>>>>>>> KIP.
>>>>>>>>>>> 
>>>>>>>>>>> I've removed all deprecated-related sections. Additionally, an
>>>>>>>>> exception
>>>>>>>>>>> will now be thrown if a
>>>>>>>>>>> developer passes an empty validStrings list to the inNonEmpty
>>>>>>> method.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty config,
>>>>>> so
>>>>>>>>> should
>>>>>>>>>> it
>>>>>>>>>>>> be non-empty too?
>>>>>>>>>>>> 
>>>>>>>>>>>> For example, `inNonEmpty()` should throw exception directly.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply. There are still references to the
>>>>>>> deprecated
>>>>>>>>>>> method.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for all the comments. They have all been addressed in
>>>>>>> the
>>>>>>>>>>> updated
>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan",
>>>>>>> could you
>>>>>>>>>> add
>>>>>>>>>>>>>>> description to remind readers that "clean.policy=" should
>>>>>> be
>>>>>>>>>> replaced
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete and
>>>>>>> compact.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
>>>>>> 上午1:55寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you
>>>>>>> update
>>>>>>>>>> the
>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>> completely? There are still references to deprecation.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
>>>>>> method,
>>>>>>> as
>>>>>>>>>> there
>>>>>>>>>>>>> may
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have
>>>>>> both
>>>>>>>>>> methods:
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
>>>>>> that
>>>>>>>>> during
>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>>>>> start.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50
>>>>>> 寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
>>>>>> that
>>>>>>>>> during
>>>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>>>>> start.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain
>>>>>>> backward
>>>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
>>>>>>>>> necessary.
>>>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove
>>>>>> the
>>>>>>>>>> boolean
>>>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
>>>>>> the
>>>>>>>>> method
>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
>>>>>> true.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>>>>> s7133700@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
>>>>>> invalid
>>>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
>>>>>> explicitly
>>>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format validation
>>>>>>> phase
>>>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
>>>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by this
>>>>>>> change
>>>>>>>>> is
>>>>>>>>>>>>> that
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
>>>>>> storage
>>>>>>>>> format
>>>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
>>>>>> already
>>>>>>> fails
>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>> process.roles
>>>>>>>>> are
>>>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>>>>>>> s7133700@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the storage
>>>>>>> format
>>>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will
>>>>>> now
>>>>>>> be
>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage format
>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
>>>>>>> 凌晨4:44
>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>>>> group.coordinator.rebalance.protocols
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With this
>>>>>>> KIP,
>>>>>>>>> the
>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
>>>>>>> exception.
>>>>>>>>> It
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
>>>>>>> Chia-Ping.
>>>>>>>>>> If
>>>>>>>>>>> a
>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
>>>>>>> mistake
>>>>>>>>> than
>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user
>>>>>> will
>>>>>>> never
>>>>>>>>>>>>>> realize
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user
>>>>>>> realize
>>>>>>>>>>> there
>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>>>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
>>>>>>> maintaining
>>>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we don't
>>>>>>> allow
>>>>>>>>> an
>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
>>>>>>> throwing
>>>>>>>>> an
>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline
>>>>>>> the
>>>>>>>>> code
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日
>>>>>> 週三
>>>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or
>>>>>>> compact
>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
>>>>>> segments
>>>>>>> based
>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
>>>>>>> should
>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>>>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote
>>>>>>>>> storage
>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>>>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>> process.roles
>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
>>>>>>> revised
>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
>>>>>>> with
>>>>>>>>> any
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
>>>>>>> cleanup.policy,
>>>>>>>>> the
>>>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
>>>>>>>>> Additionally, a
>>>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>>>> 2025年4月22日
>>>>>>>>>> 凌晨4:52
>>>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
>>>>>> for
>>>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
>>>>>> example,
>>>>>>> None
>>>>>>>>>>> can't
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
>>>>>>> document
>>>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
>>>>>>>>> existing
>>>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
>>>>>>> empty
>>>>>>>>>> check
>>>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>>>>>>>>>> s7133700@gmail.com
>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>>>>>> KIP-1161:
>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
>>>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
>>>>>>> empty
>>>>>>>>> list
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Sorry for the late reply. A few more comments.

JR10. The doc for sasl.oauthbearer.expected.audience says that it's
optional. So an empty list seems to be valid.

JR11. Thinking a bit more. If we are going to support empty lists in
general, it's probably more consistent to just allow log.cleanup.policy to
be empty as Luke suggested, instead of introducing a new option None.

JR12. NonEmptyListValidator allows the input to be null. It seems that
only ssl.cipher.suites allows null. The remaining ones shouldn't be null.

JR13. The table is very helpful. Could you also describe the old/new
behavior for each config with an empty list or duplicate entry?

Thanks,

Jun



On Mon, Jun 30, 2025 at 6:02 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello all,
>
> I am manually bumping this thread. Any feedback would be appreciated.
>
> Best regards,
> Jiunn-Yang
>
> > 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> >
> > Hello Jun,
> >
> > I have updated the KIP and introduced a new validator,
> NonEmptyListValidator, which ensures that the
> > provided list is either null or a non-empty list without duplicate
> entries. I’ve also listed the configurations that will
> > be validated using this logic. Please feel free to share any feedback or
> suggestions.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> >> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> >>
> >> Hi, Jiunn-Yang,
> >>
> >> It seems there are quite a few other configs of type list (e.g. several
> SSL
> >> related ones). It would be useful to understand if empty lists are valid
> >> for them or not.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:
> >>
> >>> Hi, Luke,
> >>>
> >>> Thanks for the reply.
> >>>
> >>> "My thought is that this behavior has been existed
> >>> for years (maybe after "cleanup.policy" was introduced?), there could
> be
> >>> users relying on the empty cleanup.policy for a long time. "
> >>>
> >>> If you do a google search on "kafka infinite retention", you will get
> >>> links like
> >>>
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> >>> and
> >>>
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/
> ,
> >>> both pointing to setting the retention time and retention size to -1 to
> >>> achieve this goal. So, it seems that if a user intentionally wants
> infinite
> >>> retention, it's more likely they will use that approach instead of
> setting
> >>> cleanup.policy to empty. On the other hand, our API/tools make it
> possible
> >>> for users to make a mistake by inadvertently leaving the config value
> empty.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com> wrote:
> >>>
> >>>> Hi Jun,
> >>>>
> >>>>> Regarding the motivation, currently, we never documented that an
> empty
> >>>> cleanup.policy implies infinite retention. In fact, if one leaves
> >>>> cleanup.policy empty, it actually breaks remote storage since it will
> stop
> >>>> the cleaning based on local retention size and time too. So, leaving
> >>>> cleanup.policy empty is probably more like a user mistake than an
> >>>> intentional choice. Based on this assumption, the idea behind the KIP
> is
> >>>> to
> >>>> fail an empty cleanup.policy so that the user is aware of this likely
> >>>> mistake and provide a new option None for users wanting infinite
> retention
> >>>> to opt in explicitly.
> >>>>
> >>>> While we never documented the empty cleanup.policy implies infinite
> >>>> retention, we also never documented (or failed) the empty
> cleanup.policy
> >>>> is
> >>>> an invalid configuration. My thought is that this behavior has been
> >>>> existed
> >>>> for years (maybe after "cleanup.policy" was introduced?), there could
> be
> >>>> users relying on the empty cleanup.policy for a long time. It is not
> good
> >>>> to break the backward compatibility in a minor release version (ex:
> >>>> v4.1.0). Even though we think we are fixing a long existing bug, it
> could
> >>>> be a surprise to users.
> >>>>
> >>>>
> >>>>> We could also consider supporting an empty cleanup.policy by fixing
> the
> >>>> issue in remote storage. But then the user may never realize this if
> it's
> >>>> a
> >>>> mistake.
> >>>>
> >>>> Good point! I agree we need to fix it!
> >>>>
> >>>>
> >>>> Hi Chia-Ping,
> >>>>
> >>>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0 we
> >>>> throw
> >>>> exception directly to make it fail fast
> >>>>
> >>>> This suggestion looks good to me!
> >>>>
> >>>> Thank you.
> >>>> Luke
> >>>>
> >>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <ch...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> hi all,
> >>>>>
> >>>>> Given Luke mentioned the existence of use cases, it is worth
> considering
> >>>>> the compatibility issue. In fact, I had previously encountered this
> use
> >>>>> case but advised customers to utilize retention configurations
> instead.
> >>>>>
> >>>>>> We could also consider supporting an empty cleanup.policy by fixing
> >>>> the
> >>>>>> issue in remote storage. But then the user may never realize this if
> >>>>> it's a
> >>>>>> mistake.
> >>>>>
> >>>>> Good catch! The "empty" or "none" wouldn't make sense for remote
> >>>> storage.
> >>>>> I've opened KAFKA-19273 to ensure topics with tiered storage use a
> valid
> >>>>> delete policy.
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>>
> >>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
> >>>>>> Hi, Luke,
> >>>>>>
> >>>>>> Regarding the motivation, currently, we never documented that an
> empty
> >>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
> >>>>>> cleanup.policy empty, it actually breaks remote storage since it
> will
> >>>>> stop
> >>>>>> the cleaning based on local retention size and time too. So, leaving
> >>>>>> cleanup.policy empty is probably more like a user mistake than an
> >>>>>> intentional choice. Based on this assumption, the idea behind the
> KIP
> >>>> is
> >>>>> to
> >>>>>> fail an empty cleanup.policy so that the user is aware of this
> likely
> >>>>>> mistake and provide a new option None for users wanting infinite
> >>>>> retention
> >>>>>> to opt in explicitly.
> >>>>>>
> >>>>>> We could also consider supporting an empty cleanup.policy by fixing
> >>>> the
> >>>>>> issue in remote storage. But then the user may never realize this if
> >>>>> it's a
> >>>>>> mistake.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>>
> >>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Hi Jiunn-Yang,
> >>>>>>>
> >>>>>>> Thanks for the KIP.
> >>>>>>>
> >>>>>>> Comments:
> >>>>>>> 1. "it is acceptable because an empty cleanup.policy is considered
> >>>>> invalid
> >>>>>>> in Kafka. Therefore, this trade-off is justified."
> >>>>>>> Could you explain more about this? Why is this trade-off justified?
> >>>>>>> If we think the empty cleanup.policy is invalid in kafka, why do we
> >>>>> provide
> >>>>>>> an additional "none" option to users in this KIP?
> >>>>>>> FYI, there are indeed use cases that need to keep all history data
> >>>>> without
> >>>>>>> a cleanup policy.
> >>>>>>>
> >>>>>>> 2. Following (1), I agree we should fail fast for empty
> >>>>>>> "group.coordinator.rebalance.protocols" and "process.roles" configs
> >>>>> since
> >>>>>>> they are invalid.
> >>>>>>> But about "cleanup.policy", I don't see a persuasive reason why we
> >>>>> should
> >>>>>>> break backward compatibility to change it.
> >>>>>>> "This provides a clear and direct way to configure Kafka to retain
> >>>> all
> >>>>> log
> >>>>>>> segments without any automatic cleanup."
> >>>>>>> This is the motivation we mentioned in the KIP, but to me, backward
> >>>>>>> compatibility is much more important than "a clear and direct way
> to
> >>>>> config
> >>>>>>> kafka".
> >>>>>>> Do we really have to change the "cleanup.policy" config?
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>> Luke
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> Thanks for the updated KIP. It looks good to me.
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Jun, chia
> >>>>>>>>>
> >>>>>>>>> Thanks for all the comments. They have all been addressed in the
> >>>>>>> updated
> >>>>>>>>> KIP.
> >>>>>>>>>
> >>>>>>>>> I've removed all deprecated-related sections. Additionally, an
> >>>>>>> exception
> >>>>>>>>> will now be thrown if a
> >>>>>>>>> developer passes an empty validStrings list to the inNonEmpty
> >>>>> method.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> >>>>>>>>>>
> >>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>
> >>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty config,
> >>>> so
> >>>>>>> should
> >>>>>>>> it
> >>>>>>>>>> be non-empty too?
> >>>>>>>>>>
> >>>>>>>>>> For example, `inNonEmpty()` should throw exception directly.
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> Chia-Ping
> >>>>>>>>>>
> >>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> >>>>>>>>>>
> >>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the reply. There are still references to the
> >>>>> deprecated
> >>>>>>>>> method.
> >>>>>>>>>>>
> >>>>>>>>>>> "stop relying on the deprecated methods"
> >>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for all the comments. They have all been addressed in
> >>>>> the
> >>>>>>>>> updated
> >>>>>>>>>>>> KIP.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan",
> >>>>> could you
> >>>>>>>> add
> >>>>>>>>>>>>> description to remind readers that "clean.policy=" should
> >>>> be
> >>>>>>>> replaced
> >>>>>>>>>>> by
> >>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete and
> >>>>> compact.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
> >>>> 上午1:55寫道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you
> >>>>> update
> >>>>>>>> the
> >>>>>>>>>>>> KIP
> >>>>>>>>>>>>>> completely? There are still references to deprecation.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
> >>>> method,
> >>>>> as
> >>>>>>>> there
> >>>>>>>>>>> may
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>> future scenarios where we want
> >>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have
> >>>> both
> >>>>>>>> methods:
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>> (which allows empty lists)
> >>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
> >>>> that
> >>>>>>> during
> >>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> >>>> start.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50
> >>>> 寫道：
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
> >>>> that
> >>>>>>> during
> >>>>>>>>>>>>>> upgrade,
> >>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
> >>>> start.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain
> >>>>> backward
> >>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> >>>>>>> necessary.
> >>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove
> >>>> the
> >>>>>>>> boolean
> >>>>>>>>>>>>>>>>> parameter.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
> >>>>> 寫道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
> >>>> the
> >>>>>>> method
> >>>>>>>>>>> name
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
> >>>> true.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> >>>> s7133700@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
> >>>> invalid
> >>>>>>>>>>>>>>> configurations
> >>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
> >>>> explicitly
> >>>>>>>> throw a
> >>>>>>>>>>>>>>>>>>> ConfigException  during the storage format validation
> >>>>> phase
> >>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
> >>>>> 凌晨1:17 寫道：
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by this
> >>>>> change
> >>>>>>> is
> >>>>>>>>>>> that
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
> >>>> storage
> >>>>>>> format
> >>>>>>>>>>>>>>>>> validation
> >>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
> >>>> already
> >>>>> fails
> >>>>>>>> if
> >>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>> process.roles
> >>>>>>> are
> >>>>>>>>>>>> empty.
> >>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> >>>>> s7133700@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> - Validation for
> >>>>> group.coordinator.rebalance.protocols and
> >>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>> will be
> >>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the storage
> >>>>> format
> >>>>>>>>>>> phase.
> >>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will
> >>>> now
> >>>>> be
> >>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>> invalid
> >>>>>>>>>>>>>>>>>>>>> and will
> >>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage format
> >>>>> phase.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
> >>>>> 凌晨4:44
> >>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
> >>>>> group.coordinator.rebalance.protocols
> >>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> process.roles
> >>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With this
> >>>>> KIP,
> >>>>>>> the
> >>>>>>>>>>> user
> >>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
> >>>>> exception.
> >>>>>>> It
> >>>>>>>>>>> will
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
> >>>>> Chia-Ping.
> >>>>>>>> If
> >>>>>>>>> a
> >>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>> leaves
> >>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
> >>>>> mistake
> >>>>>>> than
> >>>>>>>>> an
> >>>>>>>>>>>>>>>>>>> intention.
> >>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user
> >>>> will
> >>>>> never
> >>>>>>>>>>>> realize
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user
> >>>>> realize
> >>>>>>>>> there
> >>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> >>>>>>>>>>>>>> chia7712@gmail.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> >>>>> maintaining
> >>>>>>>>>>>>>>> compatibility
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we don't
> >>>>> allow
> >>>>>>> an
> >>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
> >>>>> throwing
> >>>>>>> an
> >>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline
> >>>>> the
> >>>>>>> code
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>> avoid
> >>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日
> >>>> 週三
> >>>>>>>>> 上午6:33寫道：
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or
> >>>>> compact
> >>>>>>> any
> >>>>>>>>>>>>>>>>> segments",
> >>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
> >>>> segments
> >>>>> based
> >>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
> >>>>> should
> >>>>>>>>>>> continue
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> delete
> >>>>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> >>>>>>>>>>> log.retention.ms.
> >>>>>>>>>>>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote
> >>>>>>> storage
> >>>>>>>> is
> >>>>>>>>>>>>>>>>> enabled.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> >>>>>>>> jun@confluent.io>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
> >>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> >>>>>>> process.roles
> >>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> >>>>>>> s7133700@gmail.com
> >>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
> >>>>> revised
> >>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
> >>>>> with
> >>>>>>> any
> >>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
> >>>>> cleanup.policy,
> >>>>>>> the
> >>>>>>>>>>>>>> approach
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> >>>>>>> Additionally, a
> >>>>>>>>>>>>>> warning
> >>>>>>>>>>>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> >>>> 2025年4月22日
> >>>>>>>> 凌晨4:52
> >>>>>>>>>>> 寫道：
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
> >>>> for
> >>>>>>>>>>>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>> But
> >>>>>>>>>>>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
> >>>> example,
> >>>>> None
> >>>>>>>>> can't
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> >>>>> document
> >>>>>>>> that.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
> >>>>>>> existing
> >>>>>>>>>>>>>> config
> >>>>>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
> >>>>> empty
> >>>>>>>> check
> >>>>>>>>>>>>>> does.
> >>>>>>>>>>>>>>>>> How
> >>>>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> >>>>>>>> s7133700@gmail.com
> >>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> >>>> KIP-1161:
> >>>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>> https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> >>>>> cleanup.policy
> >>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> >>>> improvements:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
> >>>>> empty
> >>>>>>> list
> >>>>>>>>> is
> >>>>>>>>>>>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> >>>>> cleanup.policy.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello all,

I am manually bumping this thread. Any feedback would be appreciated.

Best regards,
Jiunn-Yang

> 黃竣陽 <s7...@gmail.com> 於 2025年5月28日 晚上11:21 寫道：
> 
> Hello Jun,
> 
> I have updated the KIP and introduced a new validator, NonEmptyListValidator, which ensures that the 
> provided list is either null or a non-empty list without duplicate entries. I’ve also listed the configurations that will 
> be validated using this logic. Please feel free to share any feedback or suggestions.
> 
> Best Regards,
> Jiunn-Yang
> 
>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
>> 
>> Hi, Jiunn-Yang,
>> 
>> It seems there are quite a few other configs of type list (e.g. several SSL
>> related ones). It would be useful to understand if empty lists are valid
>> for them or not.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:
>> 
>>> Hi, Luke,
>>> 
>>> Thanks for the reply.
>>> 
>>> "My thought is that this behavior has been existed
>>> for years (maybe after "cleanup.policy" was introduced?), there could be
>>> users relying on the empty cleanup.policy for a long time. "
>>> 
>>> If you do a google search on "kafka infinite retention", you will get
>>> links like
>>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>>> and
>>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/,
>>> both pointing to setting the retention time and retention size to -1 to
>>> achieve this goal. So, it seems that if a user intentionally wants infinite
>>> retention, it's more likely they will use that approach instead of setting
>>> cleanup.policy to empty. On the other hand, our API/tools make it possible
>>> for users to make a mistake by inadvertently leaving the config value empty.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com> wrote:
>>> 
>>>> Hi Jun,
>>>> 
>>>>> Regarding the motivation, currently, we never documented that an empty
>>>> cleanup.policy implies infinite retention. In fact, if one leaves
>>>> cleanup.policy empty, it actually breaks remote storage since it will stop
>>>> the cleaning based on local retention size and time too. So, leaving
>>>> cleanup.policy empty is probably more like a user mistake than an
>>>> intentional choice. Based on this assumption, the idea behind the KIP is
>>>> to
>>>> fail an empty cleanup.policy so that the user is aware of this likely
>>>> mistake and provide a new option None for users wanting infinite retention
>>>> to opt in explicitly.
>>>> 
>>>> While we never documented the empty cleanup.policy implies infinite
>>>> retention, we also never documented (or failed) the empty cleanup.policy
>>>> is
>>>> an invalid configuration. My thought is that this behavior has been
>>>> existed
>>>> for years (maybe after "cleanup.policy" was introduced?), there could be
>>>> users relying on the empty cleanup.policy for a long time. It is not good
>>>> to break the backward compatibility in a minor release version (ex:
>>>> v4.1.0). Even though we think we are fixing a long existing bug, it could
>>>> be a surprise to users.
>>>> 
>>>> 
>>>>> We could also consider supporting an empty cleanup.policy by fixing the
>>>> issue in remote storage. But then the user may never realize this if it's
>>>> a
>>>> mistake.
>>>> 
>>>> Good point! I agree we need to fix it!
>>>> 
>>>> 
>>>> Hi Chia-Ping,
>>>> 
>>>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0 we
>>>> throw
>>>> exception directly to make it fail fast
>>>> 
>>>> This suggestion looks good to me!
>>>> 
>>>> Thank you.
>>>> Luke
>>>> 
>>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <ch...@apache.org>
>>>> wrote:
>>>> 
>>>>> hi all,
>>>>> 
>>>>> Given Luke mentioned the existence of use cases, it is worth considering
>>>>> the compatibility issue. In fact, I had previously encountered this use
>>>>> case but advised customers to utilize retention configurations instead.
>>>>> 
>>>>>> We could also consider supporting an empty cleanup.policy by fixing
>>>> the
>>>>>> issue in remote storage. But then the user may never realize this if
>>>>> it's a
>>>>>> mistake.
>>>>> 
>>>>> Good catch! The "empty" or "none" wouldn't make sense for remote
>>>> storage.
>>>>> I've opened KAFKA-19273 to ensure topics with tiered storage use a valid
>>>>> delete policy.
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> 
>>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>>> Hi, Luke,
>>>>>> 
>>>>>> Regarding the motivation, currently, we never documented that an empty
>>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
>>>>>> cleanup.policy empty, it actually breaks remote storage since it will
>>>>> stop
>>>>>> the cleaning based on local retention size and time too. So, leaving
>>>>>> cleanup.policy empty is probably more like a user mistake than an
>>>>>> intentional choice. Based on this assumption, the idea behind the KIP
>>>> is
>>>>> to
>>>>>> fail an empty cleanup.policy so that the user is aware of this likely
>>>>>> mistake and provide a new option None for users wanting infinite
>>>>> retention
>>>>>> to opt in explicitly.
>>>>>> 
>>>>>> We could also consider supporting an empty cleanup.policy by fixing
>>>> the
>>>>>> issue in remote storage. But then the user may never realize this if
>>>>> it's a
>>>>>> mistake.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> 
>>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the KIP.
>>>>>>> 
>>>>>>> Comments:
>>>>>>> 1. "it is acceptable because an empty cleanup.policy is considered
>>>>> invalid
>>>>>>> in Kafka. Therefore, this trade-off is justified."
>>>>>>> Could you explain more about this? Why is this trade-off justified?
>>>>>>> If we think the empty cleanup.policy is invalid in kafka, why do we
>>>>> provide
>>>>>>> an additional "none" option to users in this KIP?
>>>>>>> FYI, there are indeed use cases that need to keep all history data
>>>>> without
>>>>>>> a cleanup policy.
>>>>>>> 
>>>>>>> 2. Following (1), I agree we should fail fast for empty
>>>>>>> "group.coordinator.rebalance.protocols" and "process.roles" configs
>>>>> since
>>>>>>> they are invalid.
>>>>>>> But about "cleanup.policy", I don't see a persuasive reason why we
>>>>> should
>>>>>>> break backward compatibility to change it.
>>>>>>> "This provides a clear and direct way to configure Kafka to retain
>>>> all
>>>>> log
>>>>>>> segments without any automatic cleanup."
>>>>>>> This is the motivation we mentioned in the KIP, but to me, backward
>>>>>>> compatibility is much more important than "a clear and direct way to
>>>>> config
>>>>>>> kafka".
>>>>>>> Do we really have to change the "cleanup.policy" config?
>>>>>>> 
>>>>>>> Thanks.
>>>>>>> Luke
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid>
>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the updated KIP. It looks good to me.
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jun, chia
>>>>>>>>> 
>>>>>>>>> Thanks for all the comments. They have all been addressed in the
>>>>>>> updated
>>>>>>>>> KIP.
>>>>>>>>> 
>>>>>>>>> I've removed all deprecated-related sections. Additionally, an
>>>>>>> exception
>>>>>>>>> will now be thrown if a
>>>>>>>>> developer passes an empty validStrings list to the inNonEmpty
>>>>> method.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
>>>>>>>>>> 
>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty config,
>>>> so
>>>>>>> should
>>>>>>>> it
>>>>>>>>>> be non-empty too?
>>>>>>>>>> 
>>>>>>>>>> For example, `inNonEmpty()` should throw exception directly.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Chia-Ping
>>>>>>>>>> 
>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
>>>>>>>>>> 
>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the reply. There are still references to the
>>>>> deprecated
>>>>>>>>> method.
>>>>>>>>>>> 
>>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for all the comments. They have all been addressed in
>>>>> the
>>>>>>>>> updated
>>>>>>>>>>>> KIP.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan",
>>>>> could you
>>>>>>>> add
>>>>>>>>>>>>> description to remind readers that "clean.policy=" should
>>>> be
>>>>>>>> replaced
>>>>>>>>>>> by
>>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete and
>>>>> compact.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
>>>> 上午1:55寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you
>>>>> update
>>>>>>>> the
>>>>>>>>>>>> KIP
>>>>>>>>>>>>>> completely? There are still references to deprecation.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
>>>> method,
>>>>> as
>>>>>>>> there
>>>>>>>>>>> may
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have
>>>> both
>>>>>>>> methods:
>>>>>>>>>>> in
>>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
>>>> that
>>>>>>> during
>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>>> start.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50
>>>> 寫道：
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
>>>> that
>>>>>>> during
>>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>>> start.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain
>>>>> backward
>>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
>>>>>>> necessary.
>>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove
>>>> the
>>>>>>>> boolean
>>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
>>>>> 寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
>>>> the
>>>>>>> method
>>>>>>>>>>> name
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
>>>> true.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>>> s7133700@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
>>>> invalid
>>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
>>>> explicitly
>>>>>>>> throw a
>>>>>>>>>>>>>>>>>>> ConfigException  during the storage format validation
>>>>> phase
>>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
>>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by this
>>>>> change
>>>>>>> is
>>>>>>>>>>> that
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
>>>> storage
>>>>>>> format
>>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
>>>> already
>>>>> fails
>>>>>>>> if
>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>> process.roles
>>>>>>> are
>>>>>>>>>>>> empty.
>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>>>>> s7133700@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> - Validation for
>>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the storage
>>>>> format
>>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will
>>>> now
>>>>> be
>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage format
>>>>> phase.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
>>>>> 凌晨4:44
>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>>> group.coordinator.rebalance.protocols
>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With this
>>>>> KIP,
>>>>>>> the
>>>>>>>>>>> user
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
>>>>> exception.
>>>>>>> It
>>>>>>>>>>> will
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
>>>>> Chia-Ping.
>>>>>>>> If
>>>>>>>>> a
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
>>>>> mistake
>>>>>>> than
>>>>>>>>> an
>>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user
>>>> will
>>>>> never
>>>>>>>>>>>> realize
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user
>>>>> realize
>>>>>>>>> there
>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
>>>>> maintaining
>>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we don't
>>>>> allow
>>>>>>> an
>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
>>>>> throwing
>>>>>>> an
>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline
>>>>> the
>>>>>>> code
>>>>>>>>>>> and
>>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日
>>>> 週三
>>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or
>>>>> compact
>>>>>>> any
>>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
>>>> segments
>>>>> based
>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
>>>>> should
>>>>>>>>>>> continue
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote
>>>>>>> storage
>>>>>>>> is
>>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>>> process.roles
>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
>>>>>>> s7133700@gmail.com
>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
>>>>> revised
>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
>>>>> with
>>>>>>> any
>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
>>>>> cleanup.policy,
>>>>>>> the
>>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
>>>>>>> Additionally, a
>>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>>> 2025年4月22日
>>>>>>>> 凌晨4:52
>>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
>>>> for
>>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
>>>> example,
>>>>> None
>>>>>>>>> can't
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
>>>>> document
>>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
>>>>>>> existing
>>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
>>>>> empty
>>>>>>>> check
>>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>>>>>>>> s7133700@gmail.com
>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>>>> KIP-1161:
>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
>>>>> cleanup.policy
>>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
>>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
>>>>> empty
>>>>>>> list
>>>>>>>>> is
>>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
>>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hello Jun,

I have updated the KIP and introduced a new validator, NonEmptyListValidator, which ensures that the 
provided list is either null or a non-empty list without duplicate entries. I’ve also listed the configurations that will 
be validated using this logic. Please feel free to share any feedback or suggestions.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月23日 凌晨2:29 寫道：
> 
> Hi, Jiunn-Yang,
> 
> It seems there are quite a few other configs of type list (e.g. several SSL
> related ones). It would be useful to understand if empty lists are valid
> for them or not.
> 
> Thanks,
> 
> Jun
> 
> On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:
> 
>> Hi, Luke,
>> 
>> Thanks for the reply.
>> 
>> "My thought is that this behavior has been existed
>> for years (maybe after "cleanup.policy" was introduced?), there could be
>> users relying on the empty cleanup.policy for a long time. "
>> 
>> If you do a google search on "kafka infinite retention", you will get
>> links like
>> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
>> and
>> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/,
>> both pointing to setting the retention time and retention size to -1 to
>> achieve this goal. So, it seems that if a user intentionally wants infinite
>> retention, it's more likely they will use that approach instead of setting
>> cleanup.policy to empty. On the other hand, our API/tools make it possible
>> for users to make a mistake by inadvertently leaving the config value empty.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com> wrote:
>> 
>>> Hi Jun,
>>> 
>>>> Regarding the motivation, currently, we never documented that an empty
>>> cleanup.policy implies infinite retention. In fact, if one leaves
>>> cleanup.policy empty, it actually breaks remote storage since it will stop
>>> the cleaning based on local retention size and time too. So, leaving
>>> cleanup.policy empty is probably more like a user mistake than an
>>> intentional choice. Based on this assumption, the idea behind the KIP is
>>> to
>>> fail an empty cleanup.policy so that the user is aware of this likely
>>> mistake and provide a new option None for users wanting infinite retention
>>> to opt in explicitly.
>>> 
>>> While we never documented the empty cleanup.policy implies infinite
>>> retention, we also never documented (or failed) the empty cleanup.policy
>>> is
>>> an invalid configuration. My thought is that this behavior has been
>>> existed
>>> for years (maybe after "cleanup.policy" was introduced?), there could be
>>> users relying on the empty cleanup.policy for a long time. It is not good
>>> to break the backward compatibility in a minor release version (ex:
>>> v4.1.0). Even though we think we are fixing a long existing bug, it could
>>> be a surprise to users.
>>> 
>>> 
>>>> We could also consider supporting an empty cleanup.policy by fixing the
>>> issue in remote storage. But then the user may never realize this if it's
>>> a
>>> mistake.
>>> 
>>> Good point! I agree we need to fix it!
>>> 
>>> 
>>> Hi Chia-Ping,
>>> 
>>>> We can print warnings for empty cleanup.policy in 4.x and in 5.0 we
>>> throw
>>> exception directly to make it fail fast
>>> 
>>> This suggestion looks good to me!
>>> 
>>> Thank you.
>>> Luke
>>> 
>>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <ch...@apache.org>
>>> wrote:
>>> 
>>>> hi all,
>>>> 
>>>> Given Luke mentioned the existence of use cases, it is worth considering
>>>> the compatibility issue. In fact, I had previously encountered this use
>>>> case but advised customers to utilize retention configurations instead.
>>>> 
>>>>> We could also consider supporting an empty cleanup.policy by fixing
>>> the
>>>>> issue in remote storage. But then the user may never realize this if
>>>> it's a
>>>>> mistake.
>>>> 
>>>> Good catch! The "empty" or "none" wouldn't make sense for remote
>>> storage.
>>>> I've opened KAFKA-19273 to ensure topics with tiered storage use a valid
>>>> delete policy.
>>>> 
>>>> Best,
>>>> Chia-Ping
>>>> 
>>>> 
>>>> On 2025/05/12 19:06:10 Jun Rao wrote:
>>>>> Hi, Luke,
>>>>> 
>>>>> Regarding the motivation, currently, we never documented that an empty
>>>>> cleanup.policy implies infinite retention. In fact, if one leaves
>>>>> cleanup.policy empty, it actually breaks remote storage since it will
>>>> stop
>>>>> the cleaning based on local retention size and time too. So, leaving
>>>>> cleanup.policy empty is probably more like a user mistake than an
>>>>> intentional choice. Based on this assumption, the idea behind the KIP
>>> is
>>>> to
>>>>> fail an empty cleanup.policy so that the user is aware of this likely
>>>>> mistake and provide a new option None for users wanting infinite
>>>> retention
>>>>> to opt in explicitly.
>>>>> 
>>>>> We could also consider supporting an empty cleanup.policy by fixing
>>> the
>>>>> issue in remote storage. But then the user may never realize this if
>>>> it's a
>>>>> mistake.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jun
>>>>> 
>>>>> 
>>>>> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com> wrote:
>>>>> 
>>>>>> Hi Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the KIP.
>>>>>> 
>>>>>> Comments:
>>>>>> 1. "it is acceptable because an empty cleanup.policy is considered
>>>> invalid
>>>>>> in Kafka. Therefore, this trade-off is justified."
>>>>>> Could you explain more about this? Why is this trade-off justified?
>>>>>> If we think the empty cleanup.policy is invalid in kafka, why do we
>>>> provide
>>>>>> an additional "none" option to users in this KIP?
>>>>>> FYI, there are indeed use cases that need to keep all history data
>>>> without
>>>>>> a cleanup policy.
>>>>>> 
>>>>>> 2. Following (1), I agree we should fail fast for empty
>>>>>> "group.coordinator.rebalance.protocols" and "process.roles" configs
>>>> since
>>>>>> they are invalid.
>>>>>> But about "cleanup.policy", I don't see a persuasive reason why we
>>>> should
>>>>>> break backward compatibility to change it.
>>>>>> "This provides a clear and direct way to configure Kafka to retain
>>> all
>>>> log
>>>>>> segments without any automatic cleanup."
>>>>>> This is the motivation we mentioned in the KIP, but to me, backward
>>>>>> compatibility is much more important than "a clear and direct way to
>>>> config
>>>>>> kafka".
>>>>>> Do we really have to change the "cleanup.policy" config?
>>>>>> 
>>>>>> Thanks.
>>>>>> Luke
>>>>>> 
>>>>>> 
>>>>>> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid>
>>>> wrote:
>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the updated KIP. It looks good to me.
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hi Jun, chia
>>>>>>>> 
>>>>>>>> Thanks for all the comments. They have all been addressed in the
>>>>>> updated
>>>>>>>> KIP.
>>>>>>>> 
>>>>>>>> I've removed all deprecated-related sections. Additionally, an
>>>>>> exception
>>>>>>>> will now be thrown if a
>>>>>>>> developer passes an empty validStrings list to the inNonEmpty
>>>> method.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
>>>>>>>>> 
>>>>>>>>> hi Jiunn-Yang
>>>>>>>>> 
>>>>>>>>> The `inNonEmpty` is used to prevent users pass empty config,
>>> so
>>>>>> should
>>>>>>> it
>>>>>>>>> be non-empty too?
>>>>>>>>> 
>>>>>>>>> For example, `inNonEmpty()` should throw exception directly.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply. There are still references to the
>>>> deprecated
>>>>>>>> method.
>>>>>>>>>> 
>>>>>>>>>> "stop relying on the deprecated methods"
>>>>>>>>>> "Finally, the deprecated method will be removed in Kafka 5.0"
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for all the comments. They have all been addressed in
>>>> the
>>>>>>>> updated
>>>>>>>>>>> KIP.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>> in the "Compatibility, Deprecation, and Migration Plan",
>>>> could you
>>>>>>> add
>>>>>>>>>>>> description to remind readers that "clean.policy=" should
>>> be
>>>>>>> replaced
>>>>>>>>>> by
>>>>>>>>>>>> "clean.policy=none" if they do want to disable delete and
>>>> compact.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
>>> 上午1:55寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> It's fine not to deprecate ValidList#in for now. Could you
>>>> update
>>>>>>> the
>>>>>>>>>>> KIP
>>>>>>>>>>>>> completely? There are still references to deprecation.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I don’t think we should deprecate the ValidList#in
>>> method,
>>>> as
>>>>>>> there
>>>>>>>>>> may
>>>>>>>>>>>>> be
>>>>>>>>>>>>>> future scenarios where we want
>>>>>>>>>>>>>> to allow list configs to be empty. It’s useful to have
>>> both
>>>>>>> methods:
>>>>>>>>>> in
>>>>>>>>>>>>>> (which allows empty lists)
>>>>>>>>>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
>>> that
>>>>>> during
>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>> start.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I’ve updated the KIP accordingly.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50
>>> 寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the reply. Sounds good.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Just a minor comment. It would be useful to document
>>> that
>>>>>> during
>>>>>>>>>>>>> upgrade,
>>>>>>>>>>>>>>> if cleanup.policy is empty, the broker will fail to
>>> start.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Since ValidList is a public API, we need to maintain
>>>> backward
>>>>>>>>>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
>>>>>> necessary.
>>>>>>>>>>>>>>>> We can update the signature of isNonEmpty() to remove
>>> the
>>>>>>> boolean
>>>>>>>>>>>>>>>> parameter.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
>>>> 寫道：
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
>>> the
>>>>>> method
>>>>>>>>>> name
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
>>> true.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>>> s7133700@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> “In this case, the change does not introduce new
>>> invalid
>>>>>>>>>>>>>> configurations
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>>>>>>>>>> The main behavioral difference is that we now
>>> explicitly
>>>>>>> throw a
>>>>>>>>>>>>>>>>>> ConfigException  during the storage format validation
>>>> phase
>>>>>>>>>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
>>>> 凌晨1:17 寫道：
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> "The main behavioral difference introduced by this
>>>> change
>>>>>> is
>>>>>>>>>> that
>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> ConfigException  will now be thrown during the
>>> storage
>>>>>> format
>>>>>>>>>>>>>>>> validation
>>>>>>>>>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> It seems that currently formating the storage
>>> already
>>>> fails
>>>>>>> if
>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>> process.roles
>>>>>> are
>>>>>>>>>>> empty.
>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>> gets a different exception.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>>>> s7133700@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> - Validation for
>>>> group.coordinator.rebalance.protocols and
>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>>>>>>>> moved from the server startup phase to the storage
>>>> format
>>>>>>>>>> phase.
>>>>>>>>>>>>>>>>>>>> - For cleanup.policy, setting an empty list will
>>> now
>>>> be
>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>> invalid
>>>>>>>>>>>>>>>>>>>> and will
>>>>>>>>>>>>>>>>>>>> result in a ConfigException during storage format
>>>> phase.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
>>>> 凌晨4:44
>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Q2. It's true that
>>>> group.coordinator.rebalance.protocols
>>>>>>> and
>>>>>>>>>>>>>>>>>>>> process.roles
>>>>>>>>>>>>>>>>>>>>> configurations can't be empty right now. With this
>>>> KIP,
>>>>>> the
>>>>>>>>>> user
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>> probably get a different (and more accurate)
>>>> exception.
>>>>>> It
>>>>>>>>>> will
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
>>>> Chia-Ping.
>>>>>>> If
>>>>>>>> a
>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>> leaves
>>>>>>>>>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
>>>> mistake
>>>>>> than
>>>>>>>> an
>>>>>>>>>>>>>>>>>> intention.
>>>>>>>>>>>>>>>>>>>>> If we automatically treat it as None, the user
>>> will
>>>> never
>>>>>>>>>>> realize
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> mistake. Throwing an exception will let the user
>>>> realize
>>>>>>>> there
>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>> mistake.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>>>>>>>>>>>>> chia7712@gmail.com
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP, and I understand that
>>>> maintaining
>>>>>>>>>>>>>> compatibility
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> However, according to the documentation, we don't
>>>> allow
>>>>>> an
>>>>>>>>>>> empty
>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
>>>> throwing
>>>>>> an
>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> empty value in version 4.x? This could streamline
>>>> the
>>>>>> code
>>>>>>>>>> and
>>>>>>>>>>>>>> avoid
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日
>>> 週三
>>>>>>>> 上午6:33寫道：
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Regarding "The none policy will not delete or
>>>> compact
>>>>>> any
>>>>>>>>>>>>>>>> segments",
>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>> should be more accurate. We won't delete
>>> segments
>>>> based
>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
>>>> should
>>>>>>>>>> continue
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> delete
>>>>>>>>>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>>>>>>>>>> log.retention.ms.
>>>>>>>>>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>> risk running out of local disk space when remote
>>>>>> storage
>>>>>>> is
>>>>>>>>>>>>>>>> enabled.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>>>>>>> jun@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>>>>>> process.roles
>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
>>>>>> s7133700@gmail.com
>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
>>>> revised
>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
>>>> with
>>>>>> any
>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>>>>>>>>>> For users currently using an empty
>>>> cleanup.policy,
>>>>>> the
>>>>>>>>>>>>> approach
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
>>>>>> Additionally, a
>>>>>>>>>>>>> warning
>>>>>>>>>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>>> 2025年4月22日
>>>>>>> 凌晨4:52
>>>>>>>>>> 寫道：
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
>>> for
>>>>>>>>>>>>> cleanup.policy.
>>>>>>>>>>>>>>>> But
>>>>>>>>>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> all value combinations are valid. For
>>> example,
>>>> None
>>>>>>>> can't
>>>>>>>>>>> be
>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
>>>> document
>>>>>>> that.
>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
>>>>>> existing
>>>>>>>>>>>>> config
>>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
>>>> empty
>>>>>>> check
>>>>>>>>>>>>> does.
>>>>>>>>>>>>>>>> How
>>>>>>>>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>>>>>>> s7133700@gmail.com
>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>>> KIP-1161:
>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>> https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to improve the
>>>> cleanup.policy
>>>>>>>>>>>>>> configuration.
>>>>>>>>>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
>>> improvements:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
>>>> empty
>>>>>> list
>>>>>>>> is
>>>>>>>>>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
>>>> cleanup.policy.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

It seems there are quite a few other configs of type list (e.g. several SSL
related ones). It would be useful to understand if empty lists are valid
for them or not.

Thanks,

Jun

On Tue, May 13, 2025 at 10:12 AM Jun Rao <ju...@confluent.io> wrote:

> Hi, Luke,
>
> Thanks for the reply.
>
> "My thought is that this behavior has been existed
> for years (maybe after "cleanup.policy" was introduced?), there could be
> users relying on the empty cleanup.policy for a long time. "
>
> If you do a google search on "kafka infinite retention", you will get
> links like
> https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
> and
> https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/,
> both pointing to setting the retention time and retention size to -1 to
> achieve this goal. So, it seems that if a user intentionally wants infinite
> retention, it's more likely they will use that approach instead of setting
> cleanup.policy to empty. On the other hand, our API/tools make it possible
> for users to make a mistake by inadvertently leaving the config value empty.
>
> Thanks,
>
> Jun
>
> On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com> wrote:
>
>> Hi Jun,
>>
>> > Regarding the motivation, currently, we never documented that an empty
>> cleanup.policy implies infinite retention. In fact, if one leaves
>> cleanup.policy empty, it actually breaks remote storage since it will stop
>> the cleaning based on local retention size and time too. So, leaving
>> cleanup.policy empty is probably more like a user mistake than an
>> intentional choice. Based on this assumption, the idea behind the KIP is
>> to
>> fail an empty cleanup.policy so that the user is aware of this likely
>> mistake and provide a new option None for users wanting infinite retention
>> to opt in explicitly.
>>
>> While we never documented the empty cleanup.policy implies infinite
>> retention, we also never documented (or failed) the empty cleanup.policy
>> is
>> an invalid configuration. My thought is that this behavior has been
>> existed
>> for years (maybe after "cleanup.policy" was introduced?), there could be
>> users relying on the empty cleanup.policy for a long time. It is not good
>> to break the backward compatibility in a minor release version (ex:
>> v4.1.0). Even though we think we are fixing a long existing bug, it could
>> be a surprise to users.
>>
>>
>> > We could also consider supporting an empty cleanup.policy by fixing the
>> issue in remote storage. But then the user may never realize this if it's
>> a
>> mistake.
>>
>> Good point! I agree we need to fix it!
>>
>>
>> Hi Chia-Ping,
>>
>> > We can print warnings for empty cleanup.policy in 4.x and in 5.0 we
>> throw
>> exception directly to make it fail fast
>>
>> This suggestion looks good to me!
>>
>> Thank you.
>> Luke
>>
>> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <ch...@apache.org>
>> wrote:
>>
>> > hi all,
>> >
>> > Given Luke mentioned the existence of use cases, it is worth considering
>> > the compatibility issue. In fact, I had previously encountered this use
>> > case but advised customers to utilize retention configurations instead.
>> >
>> > > We could also consider supporting an empty cleanup.policy by fixing
>> the
>> > > issue in remote storage. But then the user may never realize this if
>> > it's a
>> > > mistake.
>> >
>> > Good catch! The "empty" or "none" wouldn't make sense for remote
>> storage.
>> > I've opened KAFKA-19273 to ensure topics with tiered storage use a valid
>> > delete policy.
>> >
>> > Best,
>> > Chia-Ping
>> >
>> >
>> > On 2025/05/12 19:06:10 Jun Rao wrote:
>> > > Hi, Luke,
>> > >
>> > > Regarding the motivation, currently, we never documented that an empty
>> > > cleanup.policy implies infinite retention. In fact, if one leaves
>> > > cleanup.policy empty, it actually breaks remote storage since it will
>> > stop
>> > > the cleaning based on local retention size and time too. So, leaving
>> > > cleanup.policy empty is probably more like a user mistake than an
>> > > intentional choice. Based on this assumption, the idea behind the KIP
>> is
>> > to
>> > > fail an empty cleanup.policy so that the user is aware of this likely
>> > > mistake and provide a new option None for users wanting infinite
>> > retention
>> > > to opt in explicitly.
>> > >
>> > > We could also consider supporting an empty cleanup.policy by fixing
>> the
>> > > issue in remote storage. But then the user may never realize this if
>> > it's a
>> > > mistake.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > >
>> > > On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com> wrote:
>> > >
>> > > > Hi Jiunn-Yang,
>> > > >
>> > > > Thanks for the KIP.
>> > > >
>> > > > Comments:
>> > > > 1. "it is acceptable because an empty cleanup.policy is considered
>> > invalid
>> > > > in Kafka. Therefore, this trade-off is justified."
>> > > > Could you explain more about this? Why is this trade-off justified?
>> > > > If we think the empty cleanup.policy is invalid in kafka, why do we
>> > provide
>> > > > an additional "none" option to users in this KIP?
>> > > > FYI, there are indeed use cases that need to keep all history data
>> > without
>> > > > a cleanup policy.
>> > > >
>> > > > 2. Following (1), I agree we should fail fast for empty
>> > > > "group.coordinator.rebalance.protocols" and "process.roles" configs
>> > since
>> > > > they are invalid.
>> > > > But about "cleanup.policy", I don't see a persuasive reason why we
>> > should
>> > > > break backward compatibility to change it.
>> > > > "This provides a clear and direct way to configure Kafka to retain
>> all
>> > log
>> > > > segments without any automatic cleanup."
>> > > > This is the motivation we mentioned in the KIP, but to me, backward
>> > > > compatibility is much more important than "a clear and direct way to
>> > config
>> > > > kafka".
>> > > > Do we really have to change the "cleanup.policy" config?
>> > > >
>> > > > Thanks.
>> > > > Luke
>> > > >
>> > > >
>> > > > On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid>
>> > wrote:
>> > > >
>> > > > > Hi, Jiunn-Yang,
>> > > > >
>> > > > > Thanks for the updated KIP. It looks good to me.
>> > > > >
>> > > > > Jun
>> > > > >
>> > > > > On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>> > > > >
>> > > > > > Hi Jun, chia
>> > > > > >
>> > > > > > Thanks for all the comments. They have all been addressed in the
>> > > > updated
>> > > > > > KIP.
>> > > > > >
>> > > > > > I've removed all deprecated-related sections. Additionally, an
>> > > > exception
>> > > > > > will now be thrown if a
>> > > > > > developer passes an empty validStrings list to the inNonEmpty
>> > method.
>> > > > > >
>> > > > > > Best Regards,
>> > > > > > Jiunn-Yang
>> > > > > >
>> > > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
>> > > > > > >
>> > > > > > > hi Jiunn-Yang
>> > > > > > >
>> > > > > > > The `inNonEmpty` is used to prevent users pass empty config,
>> so
>> > > > should
>> > > > > it
>> > > > > > > be non-empty too?
>> > > > > > >
>> > > > > > > For example, `inNonEmpty()` should throw exception directly.
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Chia-Ping
>> > > > > > >
>> > > > > > > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
>> > > > > > >
>> > > > > > >> Hi, Jiunn-Yang,
>> > > > > > >>
>> > > > > > >> Thanks for the reply. There are still references to the
>> > deprecated
>> > > > > > method.
>> > > > > > >>
>> > > > > > >> "stop relying on the deprecated methods"
>> > > > > > >> "Finally, the deprecated method will be removed in Kafka 5.0"
>> > > > > > >>
>> > > > > > >> Thanks,
>> > > > > > >>
>> > > > > > >> Jun
>> > > > > > >>
>> > > > > > >> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>> > > > > > >>
>> > > > > > >>> Hi Jun, chia,
>> > > > > > >>>
>> > > > > > >>> Thanks for all the comments. They have all been addressed in
>> > the
>> > > > > > updated
>> > > > > > >>> KIP.
>> > > > > > >>>
>> > > > > > >>> Best Regards,
>> > > > > > >>> Jiunn-Yang
>> > > > > > >>>
>> > > > > > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
>> > > > > > >>>>
>> > > > > > >>>> hi Jiunn-Yang
>> > > > > > >>>>
>> > > > > > >>>> in the "Compatibility, Deprecation, and Migration Plan",
>> > could you
>> > > > > add
>> > > > > > >>>> description to remind readers that "clean.policy=" should
>> be
>> > > > > replaced
>> > > > > > >> by
>> > > > > > >>>> "clean.policy=none" if they do want to disable delete and
>> > compact.
>> > > > > > >>>>
>> > > > > > >>>> Best,
>> > > > > > >>>> Chia-Ping
>> > > > > > >>>>
>> > > > > > >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六
>> 上午1:55寫道：
>> > > > > > >>>>
>> > > > > > >>>>> Hi, Jiunn-Yang,
>> > > > > > >>>>>
>> > > > > > >>>>> It's fine not to deprecate ValidList#in for now. Could you
>> > update
>> > > > > the
>> > > > > > >>> KIP
>> > > > > > >>>>> completely? There are still references to deprecation.
>> > > > > > >>>>>
>> > > > > > >>>>> Thanks,
>> > > > > > >>>>>
>> > > > > > >>>>> Jun
>> > > > > > >>>>>
>> > > > > > >>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
>> > wrote:
>> > > > > > >>>>>
>> > > > > > >>>>>> Hi Jun,
>> > > > > > >>>>>>
>> > > > > > >>>>>> I don’t think we should deprecate the ValidList#in
>> method,
>> > as
>> > > > > there
>> > > > > > >> may
>> > > > > > >>>>> be
>> > > > > > >>>>>> future scenarios where we want
>> > > > > > >>>>>> to allow list configs to be empty. It’s useful to have
>> both
>> > > > > methods:
>> > > > > > >> in
>> > > > > > >>>>>> (which allows empty lists)
>> > > > > > >>>>>> and inNonEmpty (which enforces non-empty lists).
>> > > > > > >>>>>>
>> > > > > > >>>>>>> Just a minor comment. It would be useful to document
>> that
>> > > > during
>> > > > > > >>>>> upgrade,
>> > > > > > >>>>>>> if cleanup.policy is empty, the broker will fail to
>> start.
>> > > > > > >>>>>>
>> > > > > > >>>>>> I’ve updated the KIP accordingly.
>> > > > > > >>>>>>
>> > > > > > >>>>>> Best Regards,
>> > > > > > >>>>>> Jiunn-Yang
>> > > > > > >>>>>>
>> > > > > > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50
>> 寫道：
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Hi, Jiunn-Yang,
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Thanks for the reply. Sounds good.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Just a minor comment. It would be useful to document
>> that
>> > > > during
>> > > > > > >>>>> upgrade,
>> > > > > > >>>>>>> if cleanup.policy is empty, the broker will fail to
>> start.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Jun
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
>> > wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> Hello Jun,
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Since ValidList is a public API, we need to maintain
>> > backward
>> > > > > > >>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
>> > > > necessary.
>> > > > > > >>>>>>>> We can update the signature of isNonEmpty() to remove
>> the
>> > > > > boolean
>> > > > > > >>>>>>>> parameter.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Best Regards,
>> > > > > > >>>>>>>> Jiunn-Yang
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
>> > 寫道：
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> Hi, Jiunn-Yang,
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> Thanks for the reply.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
>> the
>> > > > method
>> > > > > > >> name
>> > > > > > >>>>> is
>> > > > > > >>>>>>>>> inNonEmpty. Also, it's not clear when it's set to
>> true.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> Thanks,
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> Jun
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
>> s7133700@gmail.com>
>> > > > > wrote:
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> Hi Jun,
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> Thank you for pointing that out.
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> I have revised the KIP as follows:
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> “In this case, the change does not introduce new
>> invalid
>> > > > > > >>>>>> configurations
>> > > > > > >>>>>>>> or
>> > > > > > >>>>>>>>>> prevent any currently valid setup.
>> > > > > > >>>>>>>>>> The main behavioral difference is that we now
>> explicitly
>> > > > > throw a
>> > > > > > >>>>>>>>>> ConfigException  during the storage format validation
>> > phase
>> > > > > > >>>>>>>>>> instead of relying on the current behavior.”
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> This reflects the correct behavior.
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> Best Regards,
>> > > > > > >>>>>>>>>> Jiunn-Yang
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
>> > 凌晨1:17 寫道：
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> Hi, Jiunn-Yang,
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> "The main behavioral difference introduced by this
>> > change
>> > > > is
>> > > > > > >> that
>> > > > > > >>> a
>> > > > > > >>>>>>>>>>> ConfigException  will now be thrown during the
>> storage
>> > > > format
>> > > > > > >>>>>>>> validation
>> > > > > > >>>>>>>>>>> phase, rather than during server startup."
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> It seems that currently formating the storage
>> already
>> > fails
>> > > > > if
>> > > > > > >>>>>>>>>>> group.coordinator.rebalance.protocols and
>> process.roles
>> > > > are
>> > > > > > >>> empty.
>> > > > > > >>>>>> It
>> > > > > > >>>>>>>>>> just
>> > > > > > >>>>>>>>>>> gets a different exception.
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> Thanks,
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> Jun
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
>> > s7133700@gmail.com>
>> > > > > > wrote:
>> > > > > > >>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> Hi Jun, chia,
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> Thanks for your feedback.
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> I add a new section for this change:
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> - Validation for
>> > group.coordinator.rebalance.protocols and
>> > > > > > >>>>>>>> process.roles
>> > > > > > >>>>>>>>>>>> will be
>> > > > > > >>>>>>>>>>>> moved from the server startup phase to the storage
>> > format
>> > > > > > >> phase.
>> > > > > > >>>>>>>>>>>> - For cleanup.policy, setting an empty list will
>> now
>> > be
>> > > > > > >>> considered
>> > > > > > >>>>>>>>>> invalid
>> > > > > > >>>>>>>>>>>> and will
>> > > > > > >>>>>>>>>>>> result in a ConfigException during storage format
>> > phase.
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>> Best Regards,
>> > > > > > >>>>>>>>>>>> Jiunn-Yang
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
>> > 凌晨4:44
>> > > > 寫道：
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Thanks for the reply.
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Q2. It's true that
>> > group.coordinator.rebalance.protocols
>> > > > > and
>> > > > > > >>>>>>>>>>>> process.roles
>> > > > > > >>>>>>>>>>>>> configurations can't be empty right now. With this
>> > KIP,
>> > > > the
>> > > > > > >> user
>> > > > > > >>>>>> will
>> > > > > > >>>>>>>>>>>>> probably get a different (and more accurate)
>> > exception.
>> > > > It
>> > > > > > >> will
>> > > > > > >>>>> be
>> > > > > > >>>>>>>>>> useful
>> > > > > > >>>>>>>>>>>>> to document that.
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
>> > Chia-Ping.
>> > > > > If
>> > > > > > a
>> > > > > > >>>>> user
>> > > > > > >>>>>>>>>>>> leaves
>> > > > > > >>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
>> > mistake
>> > > > than
>> > > > > > an
>> > > > > > >>>>>>>>>> intention.
>> > > > > > >>>>>>>>>>>>> If we automatically treat it as None, the user
>> will
>> > never
>> > > > > > >>> realize
>> > > > > > >>>>>> the
>> > > > > > >>>>>>>>>>>>> mistake. Throwing an exception will let the user
>> > realize
>> > > > > > there
>> > > > > > >>>>> is a
>> > > > > > >>>>>>>>>>>> mistake.
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> Jun
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>> > > > > > >>>>> chia7712@gmail.com
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>>>>>> wrote:
>> > > > > > >>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>> hi Jiunn-Yang,
>> > > > > > >>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>> Thanks for the KIP, and I understand that
>> > maintaining
>> > > > > > >>>>>> compatibility
>> > > > > > >>>>>>>> is
>> > > > > > >>>>>>>>>>>>>> important.
>> > > > > > >>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>> However, according to the documentation, we don't
>> > allow
>> > > > an
>> > > > > > >>> empty
>> > > > > > >>>>>>>> value
>> > > > > > >>>>>>>>>>>> for
>> > > > > > >>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
>> > throwing
>> > > > an
>> > > > > > >>>>>> exception
>> > > > > > >>>>>>>>>> for
>> > > > > > >>>>>>>>>>>> an
>> > > > > > >>>>>>>>>>>>>> empty value in version 4.x? This could streamline
>> > the
>> > > > code
>> > > > > > >> and
>> > > > > > >>>>>> avoid
>> > > > > > >>>>>>>>>> an
>> > > > > > >>>>>>>>>>>>>> extra deprecation cycle.
>> > > > > > >>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>> Best,
>> > > > > > >>>>>>>>>>>>>> Chia-Ping
>> > > > > > >>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日
>> 週三
>> > > > > > 上午6:33寫道：
>> > > > > > >>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>> > > > > > >>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>> Regarding "The none policy will not delete or
>> > compact
>> > > > any
>> > > > > > >>>>>>>> segments",
>> > > > > > >>>>>>>>>> we
>> > > > > > >>>>>>>>>>>>>>> should be more accurate. We won't delete
>> segments
>> > based
>> > > > > on
>> > > > > > >>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
>> > should
>> > > > > > >> continue
>> > > > > > >>>>> to
>> > > > > > >>>>>>>>>> delete
>> > > > > > >>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>> > > > > > >> log.retention.ms.
>> > > > > > >>>>>>>>>>>> Otherwise,
>> > > > > > >>>>>>>>>>>>>>> we
>> > > > > > >>>>>>>>>>>>>>> risk running out of local disk space when remote
>> > > > storage
>> > > > > is
>> > > > > > >>>>>>>> enabled.
>> > > > > > >>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>> Thanks,
>> > > > > > >>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>> Jun
>> > > > > > >>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
>> > > > > jun@confluent.io>
>> > > > > > >>>>>> wrote:
>> > > > > > >>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>> > > > > > >>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>> Thanks for the reply.
>> > > > > > >>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>> Q2. What about existing empty values for
>> > > > > > >>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
>> > > > process.roles
>> > > > > > >>> during
>> > > > > > >>>>>>>>>>>> upgrade?
>> > > > > > >>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>> Jun
>> > > > > > >>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
>> > > > s7133700@gmail.com
>> > > > > >
>> > > > > > >>>>> wrote:
>> > > > > > >>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>> Hello Jun,
>> > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>> Thanks for review this KIP.
>> > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>> Q1 & Q3:
>> > > > > > >>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
>> > revised
>> > > > > the
>> > > > > > >>>>>>>>>>>>>> cleanup.policy
>> > > > > > >>>>>>>>>>>>>>>>> documentation
>> > > > > > >>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
>> > with
>> > > > any
>> > > > > > >>> other
>> > > > > > >>>>>>>>>> policy.
>> > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>> Q2:
>> > > > > > >>>>>>>>>>>>>>>>> For users currently using an empty
>> > cleanup.policy,
>> > > > the
>> > > > > > >>>>> approach
>> > > > > > >>>>>>>> is
>> > > > > > >>>>>>>>>> to
>> > > > > > >>>>>>>>>>>>>>>>> apply the none policy
>> > > > > > >>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
>> > > > Additionally, a
>> > > > > > >>>>> warning
>> > > > > > >>>>>>>>>>>>>> message
>> > > > > > >>>>>>>>>>>>>>>>> will be emitted to inform users
>> > > > > > >>>>>>>>>>>>>>>>> of the upcoming change.
>> > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>> Best Regards,
>> > > > > > >>>>>>>>>>>>>>>>> Jiunn-Yang
>> > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
>> 2025年4月22日
>> > > > > 凌晨4:52
>> > > > > > >> 寫道：
>> > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>> > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>> > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None
>> for
>> > > > > > >>>>> cleanup.policy.
>> > > > > > >>>>>>>> But
>> > > > > > >>>>>>>>>>>>>> now
>> > > > > > >>>>>>>>>>>>>>>>> not
>> > > > > > >>>>>>>>>>>>>>>>>> all value combinations are valid. For
>> example,
>> > None
>> > > > > > can't
>> > > > > > >>> be
>> > > > > > >>>>>>>> used
>> > > > > > >>>>>>>>>>>>>> with
>> > > > > > >>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
>> > document
>> > > > > that.
>> > > > > > >>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
>> > > > existing
>> > > > > > >>>>> config
>> > > > > > >>>>>>>> has
>> > > > > > >>>>>>>>>> an
>> > > > > > >>>>>>>>>>>>>>>>> empty
>> > > > > > >>>>>>>>>>>>>>>>>> list.
>> > > > > > >>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
>> > empty
>> > > > > check
>> > > > > > >>>>> does.
>> > > > > > >>>>>>>> How
>> > > > > > >>>>>>>>>>>>>>> about
>> > > > > > >>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>> > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>> Thanks,
>> > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>> Jun
>> > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
>> > > > > s7133700@gmail.com
>> > > > > > >
>> > > > > > >>>>>> wrote:
>> > > > > > >>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>>> Hello everyone,
>> > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>> KIP-1161:
>> > > > > > >>>>>> cleanup.policy
>> > > > > > >>>>>>>>>>>>>>>>> shouldn't
>> > > > > > >>>>>>>>>>>>>>>>>>> be empty
>> > > > > > >>>>>>>>>>>>>>>>>>> <
>> https://cwiki.apache.org/confluence/x/HArXF>
>> > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>>> This proposal aims to improve the
>> > cleanup.policy
>> > > > > > >>>>>> configuration.
>> > > > > > >>>>>>>>>>>>>>>>> Currently,
>> > > > > > >>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>> > > > > > >>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
>> improvements:
>> > > > > > >>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
>> > empty
>> > > > list
>> > > > > > is
>> > > > > > >>>>>>>> allowed.
>> > > > > > >>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
>> > cleanup.policy.
>> > > > > > >>>>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
>> > > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
>> > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>>>
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Luke,

Thanks for the reply.

"My thought is that this behavior has been existed
for years (maybe after "cleanup.policy" was introduced?), there could be
users relying on the empty cleanup.policy for a long time. "

If you do a google search on "kafka infinite retention", you will get links
like
https://stackoverflow.com/questions/39735036/make-kafka-topic-log-retention-permanent
and
https://www.reddit.com/r/apachekafka/comments/mpz4bp/kafka_retention_question/,
both pointing to setting the retention time and retention size to -1 to
achieve this goal. So, it seems that if a user intentionally wants infinite
retention, it's more likely they will use that approach instead of setting
cleanup.policy to empty. On the other hand, our API/tools make it possible
for users to make a mistake by inadvertently leaving the config value empty.

Thanks,

Jun

On Mon, May 12, 2025 at 11:38 PM Luke Chen <sh...@gmail.com> wrote:

> Hi Jun,
>
> > Regarding the motivation, currently, we never documented that an empty
> cleanup.policy implies infinite retention. In fact, if one leaves
> cleanup.policy empty, it actually breaks remote storage since it will stop
> the cleaning based on local retention size and time too. So, leaving
> cleanup.policy empty is probably more like a user mistake than an
> intentional choice. Based on this assumption, the idea behind the KIP is to
> fail an empty cleanup.policy so that the user is aware of this likely
> mistake and provide a new option None for users wanting infinite retention
> to opt in explicitly.
>
> While we never documented the empty cleanup.policy implies infinite
> retention, we also never documented (or failed) the empty cleanup.policy is
> an invalid configuration. My thought is that this behavior has been existed
> for years (maybe after "cleanup.policy" was introduced?), there could be
> users relying on the empty cleanup.policy for a long time. It is not good
> to break the backward compatibility in a minor release version (ex:
> v4.1.0). Even though we think we are fixing a long existing bug, it could
> be a surprise to users.
>
>
> > We could also consider supporting an empty cleanup.policy by fixing the
> issue in remote storage. But then the user may never realize this if it's a
> mistake.
>
> Good point! I agree we need to fix it!
>
>
> Hi Chia-Ping,
>
> > We can print warnings for empty cleanup.policy in 4.x and in 5.0 we throw
> exception directly to make it fail fast
>
> This suggestion looks good to me!
>
> Thank you.
> Luke
>
> On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <ch...@apache.org>
> wrote:
>
> > hi all,
> >
> > Given Luke mentioned the existence of use cases, it is worth considering
> > the compatibility issue. In fact, I had previously encountered this use
> > case but advised customers to utilize retention configurations instead.
> >
> > > We could also consider supporting an empty cleanup.policy by fixing the
> > > issue in remote storage. But then the user may never realize this if
> > it's a
> > > mistake.
> >
> > Good catch! The "empty" or "none" wouldn't make sense for remote storage.
> > I've opened KAFKA-19273 to ensure topics with tiered storage use a valid
> > delete policy.
> >
> > Best,
> > Chia-Ping
> >
> >
> > On 2025/05/12 19:06:10 Jun Rao wrote:
> > > Hi, Luke,
> > >
> > > Regarding the motivation, currently, we never documented that an empty
> > > cleanup.policy implies infinite retention. In fact, if one leaves
> > > cleanup.policy empty, it actually breaks remote storage since it will
> > stop
> > > the cleaning based on local retention size and time too. So, leaving
> > > cleanup.policy empty is probably more like a user mistake than an
> > > intentional choice. Based on this assumption, the idea behind the KIP
> is
> > to
> > > fail an empty cleanup.policy so that the user is aware of this likely
> > > mistake and provide a new option None for users wanting infinite
> > retention
> > > to opt in explicitly.
> > >
> > > We could also consider supporting an empty cleanup.policy by fixing the
> > > issue in remote storage. But then the user may never realize this if
> > it's a
> > > mistake.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com> wrote:
> > >
> > > > Hi Jiunn-Yang,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > Comments:
> > > > 1. "it is acceptable because an empty cleanup.policy is considered
> > invalid
> > > > in Kafka. Therefore, this trade-off is justified."
> > > > Could you explain more about this? Why is this trade-off justified?
> > > > If we think the empty cleanup.policy is invalid in kafka, why do we
> > provide
> > > > an additional "none" option to users in this KIP?
> > > > FYI, there are indeed use cases that need to keep all history data
> > without
> > > > a cleanup policy.
> > > >
> > > > 2. Following (1), I agree we should fail fast for empty
> > > > "group.coordinator.rebalance.protocols" and "process.roles" configs
> > since
> > > > they are invalid.
> > > > But about "cleanup.policy", I don't see a persuasive reason why we
> > should
> > > > break backward compatibility to change it.
> > > > "This provides a clear and direct way to configure Kafka to retain
> all
> > log
> > > > segments without any automatic cleanup."
> > > > This is the motivation we mentioned in the KIP, but to me, backward
> > > > compatibility is much more important than "a clear and direct way to
> > config
> > > > kafka".
> > > > Do we really have to change the "cleanup.policy" config?
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > > On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid>
> > wrote:
> > > >
> > > > > Hi, Jiunn-Yang,
> > > > >
> > > > > Thanks for the updated KIP. It looks good to me.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >
> > > > > > Hi Jun, chia
> > > > > >
> > > > > > Thanks for all the comments. They have all been addressed in the
> > > > updated
> > > > > > KIP.
> > > > > >
> > > > > > I've removed all deprecated-related sections. Additionally, an
> > > > exception
> > > > > > will now be thrown if a
> > > > > > developer passes an empty validStrings list to the inNonEmpty
> > method.
> > > > > >
> > > > > > Best Regards,
> > > > > > Jiunn-Yang
> > > > > >
> > > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> > > > > > >
> > > > > > > hi Jiunn-Yang
> > > > > > >
> > > > > > > The `inNonEmpty` is used to prevent users pass empty config, so
> > > > should
> > > > > it
> > > > > > > be non-empty too?
> > > > > > >
> > > > > > > For example, `inNonEmpty()` should throw exception directly.
> > > > > > >
> > > > > > > Best,
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> > > > > > >
> > > > > > >> Hi, Jiunn-Yang,
> > > > > > >>
> > > > > > >> Thanks for the reply. There are still references to the
> > deprecated
> > > > > > method.
> > > > > > >>
> > > > > > >> "stop relying on the deprecated methods"
> > > > > > >> "Finally, the deprecated method will be removed in Kafka 5.0"
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >> Jun
> > > > > > >>
> > > > > > >> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > > > > > >>
> > > > > > >>> Hi Jun, chia,
> > > > > > >>>
> > > > > > >>> Thanks for all the comments. They have all been addressed in
> > the
> > > > > > updated
> > > > > > >>> KIP.
> > > > > > >>>
> > > > > > >>> Best Regards,
> > > > > > >>> Jiunn-Yang
> > > > > > >>>
> > > > > > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> > > > > > >>>>
> > > > > > >>>> hi Jiunn-Yang
> > > > > > >>>>
> > > > > > >>>> in the "Compatibility, Deprecation, and Migration Plan",
> > could you
> > > > > add
> > > > > > >>>> description to remind readers that "clean.policy=" should be
> > > > > replaced
> > > > > > >> by
> > > > > > >>>> "clean.policy=none" if they do want to disable delete and
> > compact.
> > > > > > >>>>
> > > > > > >>>> Best,
> > > > > > >>>> Chia-Ping
> > > > > > >>>>
> > > > > > >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> > > > > > >>>>
> > > > > > >>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>
> > > > > > >>>>> It's fine not to deprecate ValidList#in for now. Could you
> > update
> > > > > the
> > > > > > >>> KIP
> > > > > > >>>>> completely? There are still references to deprecation.
> > > > > > >>>>>
> > > > > > >>>>> Thanks,
> > > > > > >>>>>
> > > > > > >>>>> Jun
> > > > > > >>>>>
> > > > > > >>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Hi Jun,
> > > > > > >>>>>>
> > > > > > >>>>>> I don’t think we should deprecate the ValidList#in method,
> > as
> > > > > there
> > > > > > >> may
> > > > > > >>>>> be
> > > > > > >>>>>> future scenarios where we want
> > > > > > >>>>>> to allow list configs to be empty. It’s useful to have
> both
> > > > > methods:
> > > > > > >> in
> > > > > > >>>>>> (which allows empty lists)
> > > > > > >>>>>> and inNonEmpty (which enforces non-empty lists).
> > > > > > >>>>>>
> > > > > > >>>>>>> Just a minor comment. It would be useful to document that
> > > > during
> > > > > > >>>>> upgrade,
> > > > > > >>>>>>> if cleanup.policy is empty, the broker will fail to
> start.
> > > > > > >>>>>>
> > > > > > >>>>>> I’ve updated the KIP accordingly.
> > > > > > >>>>>>
> > > > > > >>>>>> Best Regards,
> > > > > > >>>>>> Jiunn-Yang
> > > > > > >>>>>>
> > > > > > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50
> 寫道：
> > > > > > >>>>>>>
> > > > > > >>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks for the reply. Sounds good.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Just a minor comment. It would be useful to document that
> > > > during
> > > > > > >>>>> upgrade,
> > > > > > >>>>>>> if cleanup.policy is empty, the broker will fail to
> start.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Jun
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Hello Jun,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Since ValidList is a public API, we need to maintain
> > backward
> > > > > > >>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> > > > necessary.
> > > > > > >>>>>>>> We can update the signature of isNonEmpty() to remove
> the
> > > > > boolean
> > > > > > >>>>>>>> parameter.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Best Regards,
> > > > > > >>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
> > 寫道：
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Do we really need isEmptyAllowed? It's awkward since
> the
> > > > method
> > > > > > >> name
> > > > > > >>>>> is
> > > > > > >>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Jun
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <
> s7133700@gmail.com>
> > > > > wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Hi Jun,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Thank you for pointing that out.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> I have revised the KIP as follows:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> “In this case, the change does not introduce new
> invalid
> > > > > > >>>>>> configurations
> > > > > > >>>>>>>> or
> > > > > > >>>>>>>>>> prevent any currently valid setup.
> > > > > > >>>>>>>>>> The main behavioral difference is that we now
> explicitly
> > > > > throw a
> > > > > > >>>>>>>>>> ConfigException  during the storage format validation
> > phase
> > > > > > >>>>>>>>>> instead of relying on the current behavior.”
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> This reflects the correct behavior.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
> > 凌晨1:17 寫道：
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> "The main behavioral difference introduced by this
> > change
> > > > is
> > > > > > >> that
> > > > > > >>> a
> > > > > > >>>>>>>>>>> ConfigException  will now be thrown during the
> storage
> > > > format
> > > > > > >>>>>>>> validation
> > > > > > >>>>>>>>>>> phase, rather than during server startup."
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> It seems that currently formating the storage already
> > fails
> > > > > if
> > > > > > >>>>>>>>>>> group.coordinator.rebalance.protocols and
> process.roles
> > > > are
> > > > > > >>> empty.
> > > > > > >>>>>> It
> > > > > > >>>>>>>>>> just
> > > > > > >>>>>>>>>>> gets a different exception.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> > s7133700@gmail.com>
> > > > > > wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Hi Jun, chia,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thanks for your feedback.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I add a new section for this change:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> - Validation for
> > group.coordinator.rebalance.protocols and
> > > > > > >>>>>>>> process.roles
> > > > > > >>>>>>>>>>>> will be
> > > > > > >>>>>>>>>>>> moved from the server startup phase to the storage
> > format
> > > > > > >> phase.
> > > > > > >>>>>>>>>>>> - For cleanup.policy, setting an empty list will now
> > be
> > > > > > >>> considered
> > > > > > >>>>>>>>>> invalid
> > > > > > >>>>>>>>>>>> and will
> > > > > > >>>>>>>>>>>> result in a ConfigException during storage format
> > phase.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
> > 凌晨4:44
> > > > 寫道：
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Q2. It's true that
> > group.coordinator.rebalance.protocols
> > > > > and
> > > > > > >>>>>>>>>>>> process.roles
> > > > > > >>>>>>>>>>>>> configurations can't be empty right now. With this
> > KIP,
> > > > the
> > > > > > >> user
> > > > > > >>>>>> will
> > > > > > >>>>>>>>>>>>> probably get a different (and more accurate)
> > exception.
> > > > It
> > > > > > >> will
> > > > > > >>>>> be
> > > > > > >>>>>>>>>> useful
> > > > > > >>>>>>>>>>>>> to document that.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
> > Chia-Ping.
> > > > > If
> > > > > > a
> > > > > > >>>>> user
> > > > > > >>>>>>>>>>>> leaves
> > > > > > >>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
> > mistake
> > > > than
> > > > > > an
> > > > > > >>>>>>>>>> intention.
> > > > > > >>>>>>>>>>>>> If we automatically treat it as None, the user will
> > never
> > > > > > >>> realize
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>>>> mistake. Throwing an exception will let the user
> > realize
> > > > > > there
> > > > > > >>>>> is a
> > > > > > >>>>>>>>>>>> mistake.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> > > > > > >>>>> chia7712@gmail.com
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> hi Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> > maintaining
> > > > > > >>>>>> compatibility
> > > > > > >>>>>>>> is
> > > > > > >>>>>>>>>>>>>> important.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> However, according to the documentation, we don't
> > allow
> > > > an
> > > > > > >>> empty
> > > > > > >>>>>>>> value
> > > > > > >>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
> > throwing
> > > > an
> > > > > > >>>>>> exception
> > > > > > >>>>>>>>>> for
> > > > > > >>>>>>>>>>>> an
> > > > > > >>>>>>>>>>>>>> empty value in version 4.x? This could streamline
> > the
> > > > code
> > > > > > >> and
> > > > > > >>>>>> avoid
> > > > > > >>>>>>>>>> an
> > > > > > >>>>>>>>>>>>>> extra deprecation cycle.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日
> 週三
> > > > > > 上午6:33寫道：
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Regarding "The none policy will not delete or
> > compact
> > > > any
> > > > > > >>>>>>>> segments",
> > > > > > >>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>> should be more accurate. We won't delete segments
> > based
> > > > > on
> > > > > > >>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
> > should
> > > > > > >> continue
> > > > > > >>>>> to
> > > > > > >>>>>>>>>> delete
> > > > > > >>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> > > > > > >> log.retention.ms.
> > > > > > >>>>>>>>>>>> Otherwise,
> > > > > > >>>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>> risk running out of local disk space when remote
> > > > storage
> > > > > is
> > > > > > >>>>>>>> enabled.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> > > > > jun@confluent.io>
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > > > > > >>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > > > process.roles
> > > > > > >>> during
> > > > > > >>>>>>>>>>>> upgrade?
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> > > > s7133700@gmail.com
> > > > > >
> > > > > > >>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Hello Jun,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Q1 & Q3:
> > > > > > >>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
> > revised
> > > > > the
> > > > > > >>>>>>>>>>>>>> cleanup.policy
> > > > > > >>>>>>>>>>>>>>>>> documentation
> > > > > > >>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
> > with
> > > > any
> > > > > > >>> other
> > > > > > >>>>>>>>>> policy.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Q2:
> > > > > > >>>>>>>>>>>>>>>>> For users currently using an empty
> > cleanup.policy,
> > > > the
> > > > > > >>>>> approach
> > > > > > >>>>>>>> is
> > > > > > >>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>> apply the none policy
> > > > > > >>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> > > > Additionally, a
> > > > > > >>>>> warning
> > > > > > >>>>>>>>>>>>>> message
> > > > > > >>>>>>>>>>>>>>>>> will be emitted to inform users
> > > > > > >>>>>>>>>>>>>>>>> of the upcoming change.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於
> 2025年4月22日
> > > > > 凌晨4:52
> > > > > > >> 寫道：
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> > > > > > >>>>> cleanup.policy.
> > > > > > >>>>>>>> But
> > > > > > >>>>>>>>>>>>>> now
> > > > > > >>>>>>>>>>>>>>>>> not
> > > > > > >>>>>>>>>>>>>>>>>> all value combinations are valid. For example,
> > None
> > > > > > can't
> > > > > > >>> be
> > > > > > >>>>>>>> used
> > > > > > >>>>>>>>>>>>>> with
> > > > > > >>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> > document
> > > > > that.
> > > > > > >>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
> > > > existing
> > > > > > >>>>> config
> > > > > > >>>>>>>> has
> > > > > > >>>>>>>>>> an
> > > > > > >>>>>>>>>>>>>>>>> empty
> > > > > > >>>>>>>>>>>>>>>>>> list.
> > > > > > >>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
> > empty
> > > > > check
> > > > > > >>>>> does.
> > > > > > >>>>>>>> How
> > > > > > >>>>>>>>>>>>>>> about
> > > > > > >>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> > > > > s7133700@gmail.com
> > > > > > >
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> KIP-1161:
> > > > > > >>>>>> cleanup.policy
> > > > > > >>>>>>>>>>>>>>>>> shouldn't
> > > > > > >>>>>>>>>>>>>>>>>>> be empty
> > > > > > >>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF
> >
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> > cleanup.policy
> > > > > > >>>>>> configuration.
> > > > > > >>>>>>>>>>>>>>>>> Currently,
> > > > > > >>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > > > > > >>>>>>>>>>>>>>>>>>> Therefore, there are two proposed
> improvements:
> > > > > > >>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
> > empty
> > > > list
> > > > > > is
> > > > > > >>>>>>>> allowed.
> > > > > > >>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> > cleanup.policy.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Luke Chen <sh...@gmail.com>]

Hi Jun,

> Regarding the motivation, currently, we never documented that an empty
cleanup.policy implies infinite retention. In fact, if one leaves
cleanup.policy empty, it actually breaks remote storage since it will stop
the cleaning based on local retention size and time too. So, leaving
cleanup.policy empty is probably more like a user mistake than an
intentional choice. Based on this assumption, the idea behind the KIP is to
fail an empty cleanup.policy so that the user is aware of this likely
mistake and provide a new option None for users wanting infinite retention
to opt in explicitly.

While we never documented the empty cleanup.policy implies infinite
retention, we also never documented (or failed) the empty cleanup.policy is
an invalid configuration. My thought is that this behavior has been existed
for years (maybe after "cleanup.policy" was introduced?), there could be
users relying on the empty cleanup.policy for a long time. It is not good
to break the backward compatibility in a minor release version (ex:
v4.1.0). Even though we think we are fixing a long existing bug, it could
be a surprise to users.


> We could also consider supporting an empty cleanup.policy by fixing the
issue in remote storage. But then the user may never realize this if it's a
mistake.

Good point! I agree we need to fix it!


Hi Chia-Ping,

> We can print warnings for empty cleanup.policy in 4.x and in 5.0 we throw
exception directly to make it fail fast

This suggestion looks good to me!

Thank you.
Luke

On Tue, May 13, 2025 at 2:14 PM Chia-Ping Tsai <ch...@apache.org> wrote:

> hi all,
>
> Given Luke mentioned the existence of use cases, it is worth considering
> the compatibility issue. In fact, I had previously encountered this use
> case but advised customers to utilize retention configurations instead.
>
> > We could also consider supporting an empty cleanup.policy by fixing the
> > issue in remote storage. But then the user may never realize this if
> it's a
> > mistake.
>
> Good catch! The "empty" or "none" wouldn't make sense for remote storage.
> I've opened KAFKA-19273 to ensure topics with tiered storage use a valid
> delete policy.
>
> Best,
> Chia-Ping
>
>
> On 2025/05/12 19:06:10 Jun Rao wrote:
> > Hi, Luke,
> >
> > Regarding the motivation, currently, we never documented that an empty
> > cleanup.policy implies infinite retention. In fact, if one leaves
> > cleanup.policy empty, it actually breaks remote storage since it will
> stop
> > the cleaning based on local retention size and time too. So, leaving
> > cleanup.policy empty is probably more like a user mistake than an
> > intentional choice. Based on this assumption, the idea behind the KIP is
> to
> > fail an empty cleanup.policy so that the user is aware of this likely
> > mistake and provide a new option None for users wanting infinite
> retention
> > to opt in explicitly.
> >
> > We could also consider supporting an empty cleanup.policy by fixing the
> > issue in remote storage. But then the user may never realize this if
> it's a
> > mistake.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com> wrote:
> >
> > > Hi Jiunn-Yang,
> > >
> > > Thanks for the KIP.
> > >
> > > Comments:
> > > 1. "it is acceptable because an empty cleanup.policy is considered
> invalid
> > > in Kafka. Therefore, this trade-off is justified."
> > > Could you explain more about this? Why is this trade-off justified?
> > > If we think the empty cleanup.policy is invalid in kafka, why do we
> provide
> > > an additional "none" option to users in this KIP?
> > > FYI, there are indeed use cases that need to keep all history data
> without
> > > a cleanup policy.
> > >
> > > 2. Following (1), I agree we should fail fast for empty
> > > "group.coordinator.rebalance.protocols" and "process.roles" configs
> since
> > > they are invalid.
> > > But about "cleanup.policy", I don't see a persuasive reason why we
> should
> > > break backward compatibility to change it.
> > > "This provides a clear and direct way to configure Kafka to retain all
> log
> > > segments without any automatic cleanup."
> > > This is the motivation we mentioned in the KIP, but to me, backward
> > > compatibility is much more important than "a clear and direct way to
> config
> > > kafka".
> > > Do we really have to change the "cleanup.policy" config?
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > > On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid>
> wrote:
> > >
> > > > Hi, Jiunn-Yang,
> > > >
> > > > Thanks for the updated KIP. It looks good to me.
> > > >
> > > > Jun
> > > >
> > > > On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >
> > > > > Hi Jun, chia
> > > > >
> > > > > Thanks for all the comments. They have all been addressed in the
> > > updated
> > > > > KIP.
> > > > >
> > > > > I've removed all deprecated-related sections. Additionally, an
> > > exception
> > > > > will now be thrown if a
> > > > > developer passes an empty validStrings list to the inNonEmpty
> method.
> > > > >
> > > > > Best Regards,
> > > > > Jiunn-Yang
> > > > >
> > > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> > > > > >
> > > > > > hi Jiunn-Yang
> > > > > >
> > > > > > The `inNonEmpty` is used to prevent users pass empty config, so
> > > should
> > > > it
> > > > > > be non-empty too?
> > > > > >
> > > > > > For example, `inNonEmpty()` should throw exception directly.
> > > > > >
> > > > > > Best,
> > > > > > Chia-Ping
> > > > > >
> > > > > > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> > > > > >
> > > > > >> Hi, Jiunn-Yang,
> > > > > >>
> > > > > >> Thanks for the reply. There are still references to the
> deprecated
> > > > > method.
> > > > > >>
> > > > > >> "stop relying on the deprecated methods"
> > > > > >> "Finally, the deprecated method will be removed in Kafka 5.0"
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Jun
> > > > > >>
> > > > > >> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > > >>
> > > > > >>> Hi Jun, chia,
> > > > > >>>
> > > > > >>> Thanks for all the comments. They have all been addressed in
> the
> > > > > updated
> > > > > >>> KIP.
> > > > > >>>
> > > > > >>> Best Regards,
> > > > > >>> Jiunn-Yang
> > > > > >>>
> > > > > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> > > > > >>>>
> > > > > >>>> hi Jiunn-Yang
> > > > > >>>>
> > > > > >>>> in the "Compatibility, Deprecation, and Migration Plan",
> could you
> > > > add
> > > > > >>>> description to remind readers that "clean.policy=" should be
> > > > replaced
> > > > > >> by
> > > > > >>>> "clean.policy=none" if they do want to disable delete and
> compact.
> > > > > >>>>
> > > > > >>>> Best,
> > > > > >>>> Chia-Ping
> > > > > >>>>
> > > > > >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> > > > > >>>>
> > > > > >>>>> Hi, Jiunn-Yang,
> > > > > >>>>>
> > > > > >>>>> It's fine not to deprecate ValidList#in for now. Could you
> update
> > > > the
> > > > > >>> KIP
> > > > > >>>>> completely? There are still references to deprecation.
> > > > > >>>>>
> > > > > >>>>> Thanks,
> > > > > >>>>>
> > > > > >>>>> Jun
> > > > > >>>>>
> > > > > >>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > > > > >>>>>
> > > > > >>>>>> Hi Jun,
> > > > > >>>>>>
> > > > > >>>>>> I don’t think we should deprecate the ValidList#in method,
> as
> > > > there
> > > > > >> may
> > > > > >>>>> be
> > > > > >>>>>> future scenarios where we want
> > > > > >>>>>> to allow list configs to be empty. It’s useful to have both
> > > > methods:
> > > > > >> in
> > > > > >>>>>> (which allows empty lists)
> > > > > >>>>>> and inNonEmpty (which enforces non-empty lists).
> > > > > >>>>>>
> > > > > >>>>>>> Just a minor comment. It would be useful to document that
> > > during
> > > > > >>>>> upgrade,
> > > > > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > > > > >>>>>>
> > > > > >>>>>> I’ve updated the KIP accordingly.
> > > > > >>>>>>
> > > > > >>>>>> Best Regards,
> > > > > >>>>>> Jiunn-Yang
> > > > > >>>>>>
> > > > > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> > > > > >>>>>>>
> > > > > >>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks for the reply. Sounds good.
> > > > > >>>>>>>
> > > > > >>>>>>> Just a minor comment. It would be useful to document that
> > > during
> > > > > >>>>> upgrade,
> > > > > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > > > > >>>>>>>
> > > > > >>>>>>> Jun
> > > > > >>>>>>>
> > > > > >>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Hello Jun,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Since ValidList is a public API, we need to maintain
> backward
> > > > > >>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> > > necessary.
> > > > > >>>>>>>> We can update the signature of isNonEmpty() to remove the
> > > > boolean
> > > > > >>>>>>>> parameter.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Best Regards,
> > > > > >>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17
> 寫道：
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks for the reply.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the
> > > method
> > > > > >> name
> > > > > >>>>> is
> > > > > >>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Jun
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
> > > > wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Hi Jun,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Thank you for pointing that out.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> I have revised the KIP as follows:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> “In this case, the change does not introduce new invalid
> > > > > >>>>>> configurations
> > > > > >>>>>>>> or
> > > > > >>>>>>>>>> prevent any currently valid setup.
> > > > > >>>>>>>>>> The main behavioral difference is that we now explicitly
> > > > throw a
> > > > > >>>>>>>>>> ConfigException  during the storage format validation
> phase
> > > > > >>>>>>>>>> instead of relying on the current behavior.”
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> This reflects the correct behavior.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日
> 凌晨1:17 寫道：
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> "The main behavioral difference introduced by this
> change
> > > is
> > > > > >> that
> > > > > >>> a
> > > > > >>>>>>>>>>> ConfigException  will now be thrown during the storage
> > > format
> > > > > >>>>>>>> validation
> > > > > >>>>>>>>>>> phase, rather than during server startup."
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> It seems that currently formating the storage already
> fails
> > > > if
> > > > > >>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> > > are
> > > > > >>> empty.
> > > > > >>>>>> It
> > > > > >>>>>>>>>> just
> > > > > >>>>>>>>>>> gets a different exception.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <
> s7133700@gmail.com>
> > > > > wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Hi Jun, chia,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Thanks for your feedback.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I add a new section for this change:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> - Validation for
> group.coordinator.rebalance.protocols and
> > > > > >>>>>>>> process.roles
> > > > > >>>>>>>>>>>> will be
> > > > > >>>>>>>>>>>> moved from the server startup phase to the storage
> format
> > > > > >> phase.
> > > > > >>>>>>>>>>>> - For cleanup.policy, setting an empty list will now
> be
> > > > > >>> considered
> > > > > >>>>>>>>>> invalid
> > > > > >>>>>>>>>>>> and will
> > > > > >>>>>>>>>>>> result in a ConfigException during storage format
> phase.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日
> 凌晨4:44
> > > 寫道：
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Thanks for the reply.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Q2. It's true that
> group.coordinator.rebalance.protocols
> > > > and
> > > > > >>>>>>>>>>>> process.roles
> > > > > >>>>>>>>>>>>> configurations can't be empty right now. With this
> KIP,
> > > the
> > > > > >> user
> > > > > >>>>>> will
> > > > > >>>>>>>>>>>>> probably get a different (and more accurate)
> exception.
> > > It
> > > > > >> will
> > > > > >>>>> be
> > > > > >>>>>>>>>> useful
> > > > > >>>>>>>>>>>>> to document that.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with
> Chia-Ping.
> > > > If
> > > > > a
> > > > > >>>>> user
> > > > > >>>>>>>>>>>> leaves
> > > > > >>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a
> mistake
> > > than
> > > > > an
> > > > > >>>>>>>>>> intention.
> > > > > >>>>>>>>>>>>> If we automatically treat it as None, the user will
> never
> > > > > >>> realize
> > > > > >>>>>> the
> > > > > >>>>>>>>>>>>> mistake. Throwing an exception will let the user
> realize
> > > > > there
> > > > > >>>>> is a
> > > > > >>>>>>>>>>>> mistake.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> > > > > >>>>> chia7712@gmail.com
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> hi Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Thanks for the KIP, and I understand that
> maintaining
> > > > > >>>>>> compatibility
> > > > > >>>>>>>> is
> > > > > >>>>>>>>>>>>>> important.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> However, according to the documentation, we don't
> allow
> > > an
> > > > > >>> empty
> > > > > >>>>>>>> value
> > > > > >>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider
> throwing
> > > an
> > > > > >>>>>> exception
> > > > > >>>>>>>>>> for
> > > > > >>>>>>>>>>>> an
> > > > > >>>>>>>>>>>>>> empty value in version 4.x? This could streamline
> the
> > > code
> > > > > >> and
> > > > > >>>>>> avoid
> > > > > >>>>>>>>>> an
> > > > > >>>>>>>>>>>>>> extra deprecation cycle.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
> > > > > 上午6:33寫道：
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Regarding "The none policy will not delete or
> compact
> > > any
> > > > > >>>>>>>> segments",
> > > > > >>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>> should be more accurate. We won't delete segments
> based
> > > > on
> > > > > >>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we
> should
> > > > > >> continue
> > > > > >>>>> to
> > > > > >>>>>>>>>> delete
> > > > > >>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> > > > > >> log.retention.ms.
> > > > > >>>>>>>>>>>> Otherwise,
> > > > > >>>>>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>> risk running out of local disk space when remote
> > > storage
> > > > is
> > > > > >>>>>>>> enabled.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> > > > jun@confluent.io>
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > > > > >>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > > process.roles
> > > > > >>> during
> > > > > >>>>>>>>>>>> upgrade?
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> > > s7133700@gmail.com
> > > > >
> > > > > >>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Hello Jun,
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Q1 & Q3:
> > > > > >>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and
> revised
> > > > the
> > > > > >>>>>>>>>>>>>> cleanup.policy
> > > > > >>>>>>>>>>>>>>>>> documentation
> > > > > >>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used
> with
> > > any
> > > > > >>> other
> > > > > >>>>>>>>>> policy.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Q2:
> > > > > >>>>>>>>>>>>>>>>> For users currently using an empty
> cleanup.policy,
> > > the
> > > > > >>>>> approach
> > > > > >>>>>>>> is
> > > > > >>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>> apply the none policy
> > > > > >>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> > > Additionally, a
> > > > > >>>>> warning
> > > > > >>>>>>>>>>>>>> message
> > > > > >>>>>>>>>>>>>>>>> will be emitted to inform users
> > > > > >>>>>>>>>>>>>>>>> of the upcoming change.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
> > > > 凌晨4:52
> > > > > >> 寫道：
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> > > > > >>>>> cleanup.policy.
> > > > > >>>>>>>> But
> > > > > >>>>>>>>>>>>>> now
> > > > > >>>>>>>>>>>>>>>>> not
> > > > > >>>>>>>>>>>>>>>>>> all value combinations are valid. For example,
> None
> > > > > can't
> > > > > >>> be
> > > > > >>>>>>>> used
> > > > > >>>>>>>>>>>>>> with
> > > > > >>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to
> document
> > > > that.
> > > > > >>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
> > > existing
> > > > > >>>>> config
> > > > > >>>>>>>> has
> > > > > >>>>>>>>>> an
> > > > > >>>>>>>>>>>>>>>>> empty
> > > > > >>>>>>>>>>>>>>>>>> list.
> > > > > >>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the
> empty
> > > > check
> > > > > >>>>> does.
> > > > > >>>>>>>> How
> > > > > >>>>>>>>>>>>>>> about
> > > > > >>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> > > > s7133700@gmail.com
> > > > > >
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> > > > > >>>>>> cleanup.policy
> > > > > >>>>>>>>>>>>>>>>> shouldn't
> > > > > >>>>>>>>>>>>>>>>>>> be empty
> > > > > >>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> This proposal aims to improve the
> cleanup.policy
> > > > > >>>>>> configuration.
> > > > > >>>>>>>>>>>>>>>>> Currently,
> > > > > >>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > > > > >>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> > > > > >>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an
> empty
> > > list
> > > > > is
> > > > > >>>>>>>> allowed.
> > > > > >>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for
> cleanup.policy.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@apache.org>]

hi all,

Given Luke mentioned the existence of use cases, it is worth considering the compatibility issue. In fact, I had previously encountered this use case but advised customers to utilize retention configurations instead.

> We could also consider supporting an empty cleanup.policy by fixing the
> issue in remote storage. But then the user may never realize this if it's a
> mistake.

Good catch! The "empty" or "none" wouldn't make sense for remote storage. I've opened KAFKA-19273 to ensure topics with tiered storage use a valid delete policy.

Best,
Chia-Ping


On 2025/05/12 19:06:10 Jun Rao wrote:
> Hi, Luke,
> 
> Regarding the motivation, currently, we never documented that an empty
> cleanup.policy implies infinite retention. In fact, if one leaves
> cleanup.policy empty, it actually breaks remote storage since it will stop
> the cleaning based on local retention size and time too. So, leaving
> cleanup.policy empty is probably more like a user mistake than an
> intentional choice. Based on this assumption, the idea behind the KIP is to
> fail an empty cleanup.policy so that the user is aware of this likely
> mistake and provide a new option None for users wanting infinite retention
> to opt in explicitly.
> 
> We could also consider supporting an empty cleanup.policy by fixing the
> issue in remote storage. But then the user may never realize this if it's a
> mistake.
> 
> Thanks,
> 
> Jun
> 
> 
> On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com> wrote:
> 
> > Hi Jiunn-Yang,
> >
> > Thanks for the KIP.
> >
> > Comments:
> > 1. "it is acceptable because an empty cleanup.policy is considered invalid
> > in Kafka. Therefore, this trade-off is justified."
> > Could you explain more about this? Why is this trade-off justified?
> > If we think the empty cleanup.policy is invalid in kafka, why do we provide
> > an additional "none" option to users in this KIP?
> > FYI, there are indeed use cases that need to keep all history data without
> > a cleanup policy.
> >
> > 2. Following (1), I agree we should fail fast for empty
> > "group.coordinator.rebalance.protocols" and "process.roles" configs since
> > they are invalid.
> > But about "cleanup.policy", I don't see a persuasive reason why we should
> > break backward compatibility to change it.
> > "This provides a clear and direct way to configure Kafka to retain all log
> > segments without any automatic cleanup."
> > This is the motivation we mentioned in the KIP, but to me, backward
> > compatibility is much more important than "a clear and direct way to config
> > kafka".
> > Do we really have to change the "cleanup.policy" config?
> >
> > Thanks.
> > Luke
> >
> >
> > On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid> wrote:
> >
> > > Hi, Jiunn-Yang,
> > >
> > > Thanks for the updated KIP. It looks good to me.
> > >
> > > Jun
> > >
> > > On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >
> > > > Hi Jun, chia
> > > >
> > > > Thanks for all the comments. They have all been addressed in the
> > updated
> > > > KIP.
> > > >
> > > > I've removed all deprecated-related sections. Additionally, an
> > exception
> > > > will now be thrown if a
> > > > developer passes an empty validStrings list to the inNonEmpty method.
> > > >
> > > > Best Regards,
> > > > Jiunn-Yang
> > > >
> > > > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> > > > >
> > > > > hi Jiunn-Yang
> > > > >
> > > > > The `inNonEmpty` is used to prevent users pass empty config, so
> > should
> > > it
> > > > > be non-empty too?
> > > > >
> > > > > For example, `inNonEmpty()` should throw exception directly.
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> > > > >
> > > > >> Hi, Jiunn-Yang,
> > > > >>
> > > > >> Thanks for the reply. There are still references to the deprecated
> > > > method.
> > > > >>
> > > > >> "stop relying on the deprecated methods"
> > > > >> "Finally, the deprecated method will be removed in Kafka 5.0"
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Jun
> > > > >>
> > > > >> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >>
> > > > >>> Hi Jun, chia,
> > > > >>>
> > > > >>> Thanks for all the comments. They have all been addressed in the
> > > > updated
> > > > >>> KIP.
> > > > >>>
> > > > >>> Best Regards,
> > > > >>> Jiunn-Yang
> > > > >>>
> > > > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> > > > >>>>
> > > > >>>> hi Jiunn-Yang
> > > > >>>>
> > > > >>>> in the "Compatibility, Deprecation, and Migration Plan", could you
> > > add
> > > > >>>> description to remind readers that "clean.policy=" should be
> > > replaced
> > > > >> by
> > > > >>>> "clean.policy=none" if they do want to disable delete and compact.
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Chia-Ping
> > > > >>>>
> > > > >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> > > > >>>>
> > > > >>>>> Hi, Jiunn-Yang,
> > > > >>>>>
> > > > >>>>> It's fine not to deprecate ValidList#in for now. Could you update
> > > the
> > > > >>> KIP
> > > > >>>>> completely? There are still references to deprecation.
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>>
> > > > >>>>> Jun
> > > > >>>>>
> > > > >>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >>>>>
> > > > >>>>>> Hi Jun,
> > > > >>>>>>
> > > > >>>>>> I don’t think we should deprecate the ValidList#in method, as
> > > there
> > > > >> may
> > > > >>>>> be
> > > > >>>>>> future scenarios where we want
> > > > >>>>>> to allow list configs to be empty. It’s useful to have both
> > > methods:
> > > > >> in
> > > > >>>>>> (which allows empty lists)
> > > > >>>>>> and inNonEmpty (which enforces non-empty lists).
> > > > >>>>>>
> > > > >>>>>>> Just a minor comment. It would be useful to document that
> > during
> > > > >>>>> upgrade,
> > > > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > > > >>>>>>
> > > > >>>>>> I’ve updated the KIP accordingly.
> > > > >>>>>>
> > > > >>>>>> Best Regards,
> > > > >>>>>> Jiunn-Yang
> > > > >>>>>>
> > > > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> > > > >>>>>>>
> > > > >>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>
> > > > >>>>>>> Thanks for the reply. Sounds good.
> > > > >>>>>>>
> > > > >>>>>>> Just a minor comment. It would be useful to document that
> > during
> > > > >>>>> upgrade,
> > > > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > > > >>>>>>>
> > > > >>>>>>> Jun
> > > > >>>>>>>
> > > > >>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Hello Jun,
> > > > >>>>>>>>
> > > > >>>>>>>> Since ValidList is a public API, we need to maintain backward
> > > > >>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> > necessary.
> > > > >>>>>>>> We can update the signature of isNonEmpty() to remove the
> > > boolean
> > > > >>>>>>>> parameter.
> > > > >>>>>>>>
> > > > >>>>>>>> Best Regards,
> > > > >>>>>>>> Jiunn-Yang
> > > > >>>>>>>>
> > > > >>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> > > > >>>>>>>>>
> > > > >>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks for the reply.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the
> > method
> > > > >> name
> > > > >>>>> is
> > > > >>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Jun
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
> > > wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hi Jun,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thank you for pointing that out.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I have revised the KIP as follows:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> “In this case, the change does not introduce new invalid
> > > > >>>>>> configurations
> > > > >>>>>>>> or
> > > > >>>>>>>>>> prevent any currently valid setup.
> > > > >>>>>>>>>> The main behavioral difference is that we now explicitly
> > > throw a
> > > > >>>>>>>>>> ConfigException  during the storage format validation phase
> > > > >>>>>>>>>> instead of relying on the current behavior.”
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> This reflects the correct behavior.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Best Regards,
> > > > >>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> "The main behavioral difference introduced by this change
> > is
> > > > >> that
> > > > >>> a
> > > > >>>>>>>>>>> ConfigException  will now be thrown during the storage
> > format
> > > > >>>>>>>> validation
> > > > >>>>>>>>>>> phase, rather than during server startup."
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> It seems that currently formating the storage already fails
> > > if
> > > > >>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> > are
> > > > >>> empty.
> > > > >>>>>> It
> > > > >>>>>>>>>> just
> > > > >>>>>>>>>>> gets a different exception.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Jun
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com>
> > > > wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi Jun, chia,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks for your feedback.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I add a new section for this change:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
> > > > >>>>>>>> process.roles
> > > > >>>>>>>>>>>> will be
> > > > >>>>>>>>>>>> moved from the server startup phase to the storage format
> > > > >> phase.
> > > > >>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
> > > > >>> considered
> > > > >>>>>>>>>> invalid
> > > > >>>>>>>>>>>> and will
> > > > >>>>>>>>>>>> result in a ConfigException during storage format phase.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44
> > 寫道：
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks for the reply.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols
> > > and
> > > > >>>>>>>>>>>> process.roles
> > > > >>>>>>>>>>>>> configurations can't be empty right now. With this KIP,
> > the
> > > > >> user
> > > > >>>>>> will
> > > > >>>>>>>>>>>>> probably get a different (and more accurate) exception.
> > It
> > > > >> will
> > > > >>>>> be
> > > > >>>>>>>>>> useful
> > > > >>>>>>>>>>>>> to document that.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping.
> > > If
> > > > a
> > > > >>>>> user
> > > > >>>>>>>>>>>> leaves
> > > > >>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake
> > than
> > > > an
> > > > >>>>>>>>>> intention.
> > > > >>>>>>>>>>>>> If we automatically treat it as None, the user will never
> > > > >>> realize
> > > > >>>>>> the
> > > > >>>>>>>>>>>>> mistake. Throwing an exception will let the user realize
> > > > there
> > > > >>>>> is a
> > > > >>>>>>>>>>>> mistake.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> > > > >>>>> chia7712@gmail.com
> > > > >>>>>>>
> > > > >>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> hi Jiunn-Yang,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
> > > > >>>>>> compatibility
> > > > >>>>>>>> is
> > > > >>>>>>>>>>>>>> important.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> However, according to the documentation, we don't allow
> > an
> > > > >>> empty
> > > > >>>>>>>> value
> > > > >>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing
> > an
> > > > >>>>>> exception
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>> an
> > > > >>>>>>>>>>>>>> empty value in version 4.x? This could streamline the
> > code
> > > > >> and
> > > > >>>>>> avoid
> > > > >>>>>>>>>> an
> > > > >>>>>>>>>>>>>> extra deprecation cycle.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
> > > > 上午6:33寫道：
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact
> > any
> > > > >>>>>>>> segments",
> > > > >>>>>>>>>> we
> > > > >>>>>>>>>>>>>>> should be more accurate. We won't delete segments based
> > > on
> > > > >>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
> > > > >> continue
> > > > >>>>> to
> > > > >>>>>>>>>> delete
> > > > >>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> > > > >> log.retention.ms.
> > > > >>>>>>>>>>>> Otherwise,
> > > > >>>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>> risk running out of local disk space when remote
> > storage
> > > is
> > > > >>>>>>>> enabled.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> > > jun@confluent.io>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks for the reply.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > > > >>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> > process.roles
> > > > >>> during
> > > > >>>>>>>>>>>> upgrade?
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> > s7133700@gmail.com
> > > >
> > > > >>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hello Jun,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Q1 & Q3:
> > > > >>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised
> > > the
> > > > >>>>>>>>>>>>>> cleanup.policy
> > > > >>>>>>>>>>>>>>>>> documentation
> > > > >>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with
> > any
> > > > >>> other
> > > > >>>>>>>>>> policy.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Q2:
> > > > >>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy,
> > the
> > > > >>>>> approach
> > > > >>>>>>>> is
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> apply the none policy
> > > > >>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> > Additionally, a
> > > > >>>>> warning
> > > > >>>>>>>>>>>>>> message
> > > > >>>>>>>>>>>>>>>>> will be emitted to inform users
> > > > >>>>>>>>>>>>>>>>> of the upcoming change.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
> > > 凌晨4:52
> > > > >> 寫道：
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> > > > >>>>> cleanup.policy.
> > > > >>>>>>>> But
> > > > >>>>>>>>>>>>>> now
> > > > >>>>>>>>>>>>>>>>> not
> > > > >>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None
> > > > can't
> > > > >>> be
> > > > >>>>>>>> used
> > > > >>>>>>>>>>>>>> with
> > > > >>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document
> > > that.
> > > > >>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
> > existing
> > > > >>>>> config
> > > > >>>>>>>> has
> > > > >>>>>>>>>> an
> > > > >>>>>>>>>>>>>>>>> empty
> > > > >>>>>>>>>>>>>>>>>> list.
> > > > >>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty
> > > check
> > > > >>>>> does.
> > > > >>>>>>>> How
> > > > >>>>>>>>>>>>>>> about
> > > > >>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> > > s7133700@gmail.com
> > > > >
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> > > > >>>>>> cleanup.policy
> > > > >>>>>>>>>>>>>>>>> shouldn't
> > > > >>>>>>>>>>>>>>>>>>> be empty
> > > > >>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> > > > >>>>>> configuration.
> > > > >>>>>>>>>>>>>>>>> Currently,
> > > > >>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > > > >>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> > > > >>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty
> > list
> > > > is
> > > > >>>>>>>> allowed.
> > > > >>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Luke,

Regarding the motivation, currently, we never documented that an empty
cleanup.policy implies infinite retention. In fact, if one leaves
cleanup.policy empty, it actually breaks remote storage since it will stop
the cleaning based on local retention size and time too. So, leaving
cleanup.policy empty is probably more like a user mistake than an
intentional choice. Based on this assumption, the idea behind the KIP is to
fail an empty cleanup.policy so that the user is aware of this likely
mistake and provide a new option None for users wanting infinite retention
to opt in explicitly.

We could also consider supporting an empty cleanup.policy by fixing the
issue in remote storage. But then the user may never realize this if it's a
mistake.

Thanks,

Jun


On Sun, May 11, 2025 at 7:53 PM Luke Chen <sh...@gmail.com> wrote:

> Hi Jiunn-Yang,
>
> Thanks for the KIP.
>
> Comments:
> 1. "it is acceptable because an empty cleanup.policy is considered invalid
> in Kafka. Therefore, this trade-off is justified."
> Could you explain more about this? Why is this trade-off justified?
> If we think the empty cleanup.policy is invalid in kafka, why do we provide
> an additional "none" option to users in this KIP?
> FYI, there are indeed use cases that need to keep all history data without
> a cleanup policy.
>
> 2. Following (1), I agree we should fail fast for empty
> "group.coordinator.rebalance.protocols" and "process.roles" configs since
> they are invalid.
> But about "cleanup.policy", I don't see a persuasive reason why we should
> break backward compatibility to change it.
> "This provides a clear and direct way to configure Kafka to retain all log
> segments without any automatic cleanup."
> This is the motivation we mentioned in the KIP, but to me, backward
> compatibility is much more important than "a clear and direct way to config
> kafka".
> Do we really have to change the "cleanup.policy" config?
>
> Thanks.
> Luke
>
>
> On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid> wrote:
>
> > Hi, Jiunn-Yang,
> >
> > Thanks for the updated KIP. It looks good to me.
> >
> > Jun
> >
> > On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> > > Hi Jun, chia
> > >
> > > Thanks for all the comments. They have all been addressed in the
> updated
> > > KIP.
> > >
> > > I've removed all deprecated-related sections. Additionally, an
> exception
> > > will now be thrown if a
> > > developer passes an empty validStrings list to the inNonEmpty method.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> > > >
> > > > hi Jiunn-Yang
> > > >
> > > > The `inNonEmpty` is used to prevent users pass empty config, so
> should
> > it
> > > > be non-empty too?
> > > >
> > > > For example, `inNonEmpty()` should throw exception directly.
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> > > >
> > > >> Hi, Jiunn-Yang,
> > > >>
> > > >> Thanks for the reply. There are still references to the deprecated
> > > method.
> > > >>
> > > >> "stop relying on the deprecated methods"
> > > >> "Finally, the deprecated method will be removed in Kafka 5.0"
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>
> > > >>> Hi Jun, chia,
> > > >>>
> > > >>> Thanks for all the comments. They have all been addressed in the
> > > updated
> > > >>> KIP.
> > > >>>
> > > >>> Best Regards,
> > > >>> Jiunn-Yang
> > > >>>
> > > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> > > >>>>
> > > >>>> hi Jiunn-Yang
> > > >>>>
> > > >>>> in the "Compatibility, Deprecation, and Migration Plan", could you
> > add
> > > >>>> description to remind readers that "clean.policy=" should be
> > replaced
> > > >> by
> > > >>>> "clean.policy=none" if they do want to disable delete and compact.
> > > >>>>
> > > >>>> Best,
> > > >>>> Chia-Ping
> > > >>>>
> > > >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> > > >>>>
> > > >>>>> Hi, Jiunn-Yang,
> > > >>>>>
> > > >>>>> It's fine not to deprecate ValidList#in for now. Could you update
> > the
> > > >>> KIP
> > > >>>>> completely? There are still references to deprecation.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>>
> > > >>>>> Jun
> > > >>>>>
> > > >>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>>>>
> > > >>>>>> Hi Jun,
> > > >>>>>>
> > > >>>>>> I don’t think we should deprecate the ValidList#in method, as
> > there
> > > >> may
> > > >>>>> be
> > > >>>>>> future scenarios where we want
> > > >>>>>> to allow list configs to be empty. It’s useful to have both
> > methods:
> > > >> in
> > > >>>>>> (which allows empty lists)
> > > >>>>>> and inNonEmpty (which enforces non-empty lists).
> > > >>>>>>
> > > >>>>>>> Just a minor comment. It would be useful to document that
> during
> > > >>>>> upgrade,
> > > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > > >>>>>>
> > > >>>>>> I’ve updated the KIP accordingly.
> > > >>>>>>
> > > >>>>>> Best Regards,
> > > >>>>>> Jiunn-Yang
> > > >>>>>>
> > > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> > > >>>>>>>
> > > >>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>
> > > >>>>>>> Thanks for the reply. Sounds good.
> > > >>>>>>>
> > > >>>>>>> Just a minor comment. It would be useful to document that
> during
> > > >>>>> upgrade,
> > > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > > >>>>>>>
> > > >>>>>>> Jun
> > > >>>>>>>
> > > >>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hello Jun,
> > > >>>>>>>>
> > > >>>>>>>> Since ValidList is a public API, we need to maintain backward
> > > >>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is
> necessary.
> > > >>>>>>>> We can update the signature of isNonEmpty() to remove the
> > boolean
> > > >>>>>>>> parameter.
> > > >>>>>>>>
> > > >>>>>>>> Best Regards,
> > > >>>>>>>> Jiunn-Yang
> > > >>>>>>>>
> > > >>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> > > >>>>>>>>>
> > > >>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>
> > > >>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the
> method
> > > >> name
> > > >>>>> is
> > > >>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>>
> > > >>>>>>>>> Jun
> > > >>>>>>>>>
> > > >>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi Jun,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thank you for pointing that out.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I have revised the KIP as follows:
> > > >>>>>>>>>>
> > > >>>>>>>>>> “In this case, the change does not introduce new invalid
> > > >>>>>> configurations
> > > >>>>>>>> or
> > > >>>>>>>>>> prevent any currently valid setup.
> > > >>>>>>>>>> The main behavioral difference is that we now explicitly
> > throw a
> > > >>>>>>>>>> ConfigException  during the storage format validation phase
> > > >>>>>>>>>> instead of relying on the current behavior.”
> > > >>>>>>>>>>
> > > >>>>>>>>>> This reflects the correct behavior.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best Regards,
> > > >>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> "The main behavioral difference introduced by this change
> is
> > > >> that
> > > >>> a
> > > >>>>>>>>>>> ConfigException  will now be thrown during the storage
> format
> > > >>>>>>>> validation
> > > >>>>>>>>>>> phase, rather than during server startup."
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> It seems that currently formating the storage already fails
> > if
> > > >>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> are
> > > >>> empty.
> > > >>>>>> It
> > > >>>>>>>>>> just
> > > >>>>>>>>>>> gets a different exception.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Jun
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com>
> > > wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi Jun, chia,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for your feedback.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I add a new section for this change:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
> > > >>>>>>>> process.roles
> > > >>>>>>>>>>>> will be
> > > >>>>>>>>>>>> moved from the server startup phase to the storage format
> > > >> phase.
> > > >>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
> > > >>> considered
> > > >>>>>>>>>> invalid
> > > >>>>>>>>>>>> and will
> > > >>>>>>>>>>>> result in a ConfigException during storage format phase.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44
> 寫道：
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols
> > and
> > > >>>>>>>>>>>> process.roles
> > > >>>>>>>>>>>>> configurations can't be empty right now. With this KIP,
> the
> > > >> user
> > > >>>>>> will
> > > >>>>>>>>>>>>> probably get a different (and more accurate) exception.
> It
> > > >> will
> > > >>>>> be
> > > >>>>>>>>>> useful
> > > >>>>>>>>>>>>> to document that.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping.
> > If
> > > a
> > > >>>>> user
> > > >>>>>>>>>>>> leaves
> > > >>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake
> than
> > > an
> > > >>>>>>>>>> intention.
> > > >>>>>>>>>>>>> If we automatically treat it as None, the user will never
> > > >>> realize
> > > >>>>>> the
> > > >>>>>>>>>>>>> mistake. Throwing an exception will let the user realize
> > > there
> > > >>>>> is a
> > > >>>>>>>>>>>> mistake.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> > > >>>>> chia7712@gmail.com
> > > >>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> hi Jiunn-Yang,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
> > > >>>>>> compatibility
> > > >>>>>>>> is
> > > >>>>>>>>>>>>>> important.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> However, according to the documentation, we don't allow
> an
> > > >>> empty
> > > >>>>>>>> value
> > > >>>>>>>>>>>> for
> > > >>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing
> an
> > > >>>>>> exception
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>> an
> > > >>>>>>>>>>>>>> empty value in version 4.x? This could streamline the
> code
> > > >> and
> > > >>>>>> avoid
> > > >>>>>>>>>> an
> > > >>>>>>>>>>>>>> extra deprecation cycle.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
> > > 上午6:33寫道：
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact
> any
> > > >>>>>>>> segments",
> > > >>>>>>>>>> we
> > > >>>>>>>>>>>>>>> should be more accurate. We won't delete segments based
> > on
> > > >>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
> > > >> continue
> > > >>>>> to
> > > >>>>>>>>>> delete
> > > >>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> > > >> log.retention.ms.
> > > >>>>>>>>>>>> Otherwise,
> > > >>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>> risk running out of local disk space when remote
> storage
> > is
> > > >>>>>>>> enabled.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> > jun@confluent.io>
> > > >>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > > >>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and
> process.roles
> > > >>> during
> > > >>>>>>>>>>>> upgrade?
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <
> s7133700@gmail.com
> > >
> > > >>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Q1 & Q3:
> > > >>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised
> > the
> > > >>>>>>>>>>>>>> cleanup.policy
> > > >>>>>>>>>>>>>>>>> documentation
> > > >>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with
> any
> > > >>> other
> > > >>>>>>>>>> policy.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Q2:
> > > >>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy,
> the
> > > >>>>> approach
> > > >>>>>>>> is
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> apply the none policy
> > > >>>>>>>>>>>>>>>>> during the preProcessParsedConfig step.
> Additionally, a
> > > >>>>> warning
> > > >>>>>>>>>>>>>> message
> > > >>>>>>>>>>>>>>>>> will be emitted to inform users
> > > >>>>>>>>>>>>>>>>> of the upcoming change.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
> > 凌晨4:52
> > > >> 寫道：
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> > > >>>>> cleanup.policy.
> > > >>>>>>>> But
> > > >>>>>>>>>>>>>> now
> > > >>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None
> > > can't
> > > >>> be
> > > >>>>>>>> used
> > > >>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document
> > that.
> > > >>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an
> existing
> > > >>>>> config
> > > >>>>>>>> has
> > > >>>>>>>>>> an
> > > >>>>>>>>>>>>>>>>> empty
> > > >>>>>>>>>>>>>>>>>> list.
> > > >>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty
> > check
> > > >>>>> does.
> > > >>>>>>>> How
> > > >>>>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> > s7133700@gmail.com
> > > >
> > > >>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> > > >>>>>> cleanup.policy
> > > >>>>>>>>>>>>>>>>> shouldn't
> > > >>>>>>>>>>>>>>>>>>> be empty
> > > >>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> > > >>>>>> configuration.
> > > >>>>>>>>>>>>>>>>> Currently,
> > > >>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > > >>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> > > >>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty
> list
> > > is
> > > >>>>>>>> allowed.
> > > >>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Luke Chen <sh...@gmail.com>]

Hi Jiunn-Yang,

Thanks for the KIP.

Comments:
1. "it is acceptable because an empty cleanup.policy is considered invalid
in Kafka. Therefore, this trade-off is justified."
Could you explain more about this? Why is this trade-off justified?
If we think the empty cleanup.policy is invalid in kafka, why do we provide
an additional "none" option to users in this KIP?
FYI, there are indeed use cases that need to keep all history data without
a cleanup policy.

2. Following (1), I agree we should fail fast for empty
"group.coordinator.rebalance.protocols" and "process.roles" configs since
they are invalid.
But about "cleanup.policy", I don't see a persuasive reason why we should
break backward compatibility to change it.
"This provides a clear and direct way to configure Kafka to retain all log
segments without any automatic cleanup."
This is the motivation we mentioned in the KIP, but to me, backward
compatibility is much more important than "a clear and direct way to config
kafka".
Do we really have to change the "cleanup.policy" config?

Thanks.
Luke


On Wed, May 7, 2025 at 2:17 AM Jun Rao <ju...@confluent.io.invalid> wrote:

> Hi, Jiunn-Yang,
>
> Thanks for the updated KIP. It looks good to me.
>
> Jun
>
> On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hi Jun, chia
> >
> > Thanks for all the comments. They have all been addressed in the updated
> > KIP.
> >
> > I've removed all deprecated-related sections. Additionally, an exception
> > will now be thrown if a
> > developer passes an empty validStrings list to the inNonEmpty method.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> > >
> > > hi Jiunn-Yang
> > >
> > > The `inNonEmpty` is used to prevent users pass empty config, so should
> it
> > > be non-empty too?
> > >
> > > For example, `inNonEmpty()` should throw exception directly.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> > >
> > >> Hi, Jiunn-Yang,
> > >>
> > >> Thanks for the reply. There are still references to the deprecated
> > method.
> > >>
> > >> "stop relying on the deprecated methods"
> > >> "Finally, the deprecated method will be removed in Kafka 5.0"
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>
> > >>> Hi Jun, chia,
> > >>>
> > >>> Thanks for all the comments. They have all been addressed in the
> > updated
> > >>> KIP.
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> > >>>>
> > >>>> hi Jiunn-Yang
> > >>>>
> > >>>> in the "Compatibility, Deprecation, and Migration Plan", could you
> add
> > >>>> description to remind readers that "clean.policy=" should be
> replaced
> > >> by
> > >>>> "clean.policy=none" if they do want to disable delete and compact.
> > >>>>
> > >>>> Best,
> > >>>> Chia-Ping
> > >>>>
> > >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> > >>>>
> > >>>>> Hi, Jiunn-Yang,
> > >>>>>
> > >>>>> It's fine not to deprecate ValidList#in for now. Could you update
> the
> > >>> KIP
> > >>>>> completely? There are still references to deprecation.
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Jun
> > >>>>>
> > >>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>
> > >>>>>> Hi Jun,
> > >>>>>>
> > >>>>>> I don’t think we should deprecate the ValidList#in method, as
> there
> > >> may
> > >>>>> be
> > >>>>>> future scenarios where we want
> > >>>>>> to allow list configs to be empty. It’s useful to have both
> methods:
> > >> in
> > >>>>>> (which allows empty lists)
> > >>>>>> and inNonEmpty (which enforces non-empty lists).
> > >>>>>>
> > >>>>>>> Just a minor comment. It would be useful to document that during
> > >>>>> upgrade,
> > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > >>>>>>
> > >>>>>> I’ve updated the KIP accordingly.
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>> Jiunn-Yang
> > >>>>>>
> > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> > >>>>>>>
> > >>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>
> > >>>>>>> Thanks for the reply. Sounds good.
> > >>>>>>>
> > >>>>>>> Just a minor comment. It would be useful to document that during
> > >>>>> upgrade,
> > >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> > >>>>>>>
> > >>>>>>> Jun
> > >>>>>>>
> > >>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> Hello Jun,
> > >>>>>>>>
> > >>>>>>>> Since ValidList is a public API, we need to maintain backward
> > >>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
> > >>>>>>>> We can update the signature of isNonEmpty() to remove the
> boolean
> > >>>>>>>> parameter.
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Jiunn-Yang
> > >>>>>>>>
> > >>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> > >>>>>>>>>
> > >>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for the reply.
> > >>>>>>>>>
> > >>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the method
> > >> name
> > >>>>> is
> > >>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>>
> > >>>>>>>>> Jun
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Jun,
> > >>>>>>>>>>
> > >>>>>>>>>> Thank you for pointing that out.
> > >>>>>>>>>>
> > >>>>>>>>>> I have revised the KIP as follows:
> > >>>>>>>>>>
> > >>>>>>>>>> “In this case, the change does not introduce new invalid
> > >>>>>> configurations
> > >>>>>>>> or
> > >>>>>>>>>> prevent any currently valid setup.
> > >>>>>>>>>> The main behavioral difference is that we now explicitly
> throw a
> > >>>>>>>>>> ConfigException  during the storage format validation phase
> > >>>>>>>>>> instead of relying on the current behavior.”
> > >>>>>>>>>>
> > >>>>>>>>>> This reflects the correct behavior.
> > >>>>>>>>>>
> > >>>>>>>>>> Best Regards,
> > >>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>
> > >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>
> > >>>>>>>>>>> "The main behavioral difference introduced by this change is
> > >> that
> > >>> a
> > >>>>>>>>>>> ConfigException  will now be thrown during the storage format
> > >>>>>>>> validation
> > >>>>>>>>>>> phase, rather than during server startup."
> > >>>>>>>>>>>
> > >>>>>>>>>>> It seems that currently formating the storage already fails
> if
> > >>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
> > >>> empty.
> > >>>>>> It
> > >>>>>>>>>> just
> > >>>>>>>>>>> gets a different exception.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Jun
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for your feedback.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I add a new section for this change:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
> > >>>>>>>> process.roles
> > >>>>>>>>>>>> will be
> > >>>>>>>>>>>> moved from the server startup phase to the storage format
> > >> phase.
> > >>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
> > >>> considered
> > >>>>>>>>>> invalid
> > >>>>>>>>>>>> and will
> > >>>>>>>>>>>> result in a ConfigException during storage format phase.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols
> and
> > >>>>>>>>>>>> process.roles
> > >>>>>>>>>>>>> configurations can't be empty right now. With this KIP, the
> > >> user
> > >>>>>> will
> > >>>>>>>>>>>>> probably get a different (and more accurate) exception. It
> > >> will
> > >>>>> be
> > >>>>>>>>>> useful
> > >>>>>>>>>>>>> to document that.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping.
> If
> > a
> > >>>>> user
> > >>>>>>>>>>>> leaves
> > >>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than
> > an
> > >>>>>>>>>> intention.
> > >>>>>>>>>>>>> If we automatically treat it as None, the user will never
> > >>> realize
> > >>>>>> the
> > >>>>>>>>>>>>> mistake. Throwing an exception will let the user realize
> > there
> > >>>>> is a
> > >>>>>>>>>>>> mistake.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> > >>>>> chia7712@gmail.com
> > >>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> hi Jiunn-Yang,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
> > >>>>>> compatibility
> > >>>>>>>> is
> > >>>>>>>>>>>>>> important.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> However, according to the documentation, we don't allow an
> > >>> empty
> > >>>>>>>> value
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
> > >>>>>> exception
> > >>>>>>>>>> for
> > >>>>>>>>>>>> an
> > >>>>>>>>>>>>>> empty value in version 4.x? This could streamline the code
> > >> and
> > >>>>>> avoid
> > >>>>>>>>>> an
> > >>>>>>>>>>>>>> extra deprecation cycle.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
> > 上午6:33寫道：
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact any
> > >>>>>>>> segments",
> > >>>>>>>>>> we
> > >>>>>>>>>>>>>>> should be more accurate. We won't delete segments based
> on
> > >>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
> > >> continue
> > >>>>> to
> > >>>>>>>>>> delete
> > >>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> > >> log.retention.ms.
> > >>>>>>>>>>>> Otherwise,
> > >>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>> risk running out of local disk space when remote storage
> is
> > >>>>>>>> enabled.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <
> jun@confluent.io>
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Q2. What about existing empty values for
> > >>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> > >>> during
> > >>>>>>>>>>>> upgrade?
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7133700@gmail.com
> >
> > >>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for review this KIP.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Q1 & Q3:
> > >>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised
> the
> > >>>>>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>> documentation
> > >>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
> > >>> other
> > >>>>>>>>>> policy.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Q2:
> > >>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
> > >>>>> approach
> > >>>>>>>> is
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> apply the none policy
> > >>>>>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
> > >>>>> warning
> > >>>>>>>>>>>>>> message
> > >>>>>>>>>>>>>>>>> will be emitted to inform users
> > >>>>>>>>>>>>>>>>> of the upcoming change.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日
> 凌晨4:52
> > >> 寫道：
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> > >>>>> cleanup.policy.
> > >>>>>>>> But
> > >>>>>>>>>>>>>> now
> > >>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None
> > can't
> > >>> be
> > >>>>>>>> used
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document
> that.
> > >>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
> > >>>>> config
> > >>>>>>>> has
> > >>>>>>>>>> an
> > >>>>>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>>>>> list.
> > >>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty
> check
> > >>>>> does.
> > >>>>>>>> How
> > >>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <
> s7133700@gmail.com
> > >
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> > >>>>>> cleanup.policy
> > >>>>>>>>>>>>>>>>> shouldn't
> > >>>>>>>>>>>>>>>>>>> be empty
> > >>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> > >>>>>> configuration.
> > >>>>>>>>>>>>>>>>> Currently,
> > >>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> > >>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> > >>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list
> > is
> > >>>>>>>> allowed.
> > >>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the updated KIP. It looks good to me.

Jun

On Tue, May 6, 2025 at 4:58 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hi Jun, chia
>
> Thanks for all the comments. They have all been addressed in the updated
> KIP.
>
> I've removed all deprecated-related sections. Additionally, an exception
> will now be thrown if a
> developer passes an empty validStrings list to the inNonEmpty method.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> >
> > hi Jiunn-Yang
> >
> > The `inNonEmpty` is used to prevent users pass empty config, so should it
> > be non-empty too?
> >
> > For example, `inNonEmpty()` should throw exception directly.
> >
> > Best,
> > Chia-Ping
> >
> > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> >
> >> Hi, Jiunn-Yang,
> >>
> >> Thanks for the reply. There are still references to the deprecated
> method.
> >>
> >> "stop relying on the deprecated methods"
> >> "Finally, the deprecated method will be removed in Kafka 5.0"
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hi Jun, chia,
> >>>
> >>> Thanks for all the comments. They have all been addressed in the
> updated
> >>> KIP.
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> >>>>
> >>>> hi Jiunn-Yang
> >>>>
> >>>> in the "Compatibility, Deprecation, and Migration Plan", could you add
> >>>> description to remind readers that "clean.policy=" should be replaced
> >> by
> >>>> "clean.policy=none" if they do want to disable delete and compact.
> >>>>
> >>>> Best,
> >>>> Chia-Ping
> >>>>
> >>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> >>>>
> >>>>> Hi, Jiunn-Yang,
> >>>>>
> >>>>> It's fine not to deprecate ValidList#in for now. Could you update the
> >>> KIP
> >>>>> completely? There are still references to deprecation.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>
> >>>>>> Hi Jun,
> >>>>>>
> >>>>>> I don’t think we should deprecate the ValidList#in method, as there
> >> may
> >>>>> be
> >>>>>> future scenarios where we want
> >>>>>> to allow list configs to be empty. It’s useful to have both methods:
> >> in
> >>>>>> (which allows empty lists)
> >>>>>> and inNonEmpty (which enforces non-empty lists).
> >>>>>>
> >>>>>>> Just a minor comment. It would be useful to document that during
> >>>>> upgrade,
> >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> >>>>>>
> >>>>>> I’ve updated the KIP accordingly.
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> >>>>>>>
> >>>>>>> Hi, Jiunn-Yang,
> >>>>>>>
> >>>>>>> Thanks for the reply. Sounds good.
> >>>>>>>
> >>>>>>> Just a minor comment. It would be useful to document that during
> >>>>> upgrade,
> >>>>>>> if cleanup.policy is empty, the broker will fail to start.
> >>>>>>>
> >>>>>>> Jun
> >>>>>>>
> >>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> Hello Jun,
> >>>>>>>>
> >>>>>>>> Since ValidList is a public API, we need to maintain backward
> >>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
> >>>>>>>> We can update the signature of isNonEmpty() to remove the boolean
> >>>>>>>> parameter.
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>>
> >>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> >>>>>>>>>
> >>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>
> >>>>>>>>> Thanks for the reply.
> >>>>>>>>>
> >>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the method
> >> name
> >>>>> is
> >>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Jun
> >>>>>>>>>
> >>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Jun,
> >>>>>>>>>>
> >>>>>>>>>> Thank you for pointing that out.
> >>>>>>>>>>
> >>>>>>>>>> I have revised the KIP as follows:
> >>>>>>>>>>
> >>>>>>>>>> “In this case, the change does not introduce new invalid
> >>>>>> configurations
> >>>>>>>> or
> >>>>>>>>>> prevent any currently valid setup.
> >>>>>>>>>> The main behavioral difference is that we now explicitly throw a
> >>>>>>>>>> ConfigException  during the storage format validation phase
> >>>>>>>>>> instead of relying on the current behavior.”
> >>>>>>>>>>
> >>>>>>>>>> This reflects the correct behavior.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>
> >>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> >>>>>>>>>>>
> >>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>
> >>>>>>>>>>> "The main behavioral difference introduced by this change is
> >> that
> >>> a
> >>>>>>>>>>> ConfigException  will now be thrown during the storage format
> >>>>>>>> validation
> >>>>>>>>>>> phase, rather than during server startup."
> >>>>>>>>>>>
> >>>>>>>>>>> It seems that currently formating the storage already fails if
> >>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
> >>> empty.
> >>>>>> It
> >>>>>>>>>> just
> >>>>>>>>>>> gets a different exception.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Jun, chia,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for your feedback.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I add a new section for this change:
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
> >>>>>>>> process.roles
> >>>>>>>>>>>> will be
> >>>>>>>>>>>> moved from the server startup phase to the storage format
> >> phase.
> >>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
> >>> considered
> >>>>>>>>>> invalid
> >>>>>>>>>>>> and will
> >>>>>>>>>>>> result in a ConfigException during storage format phase.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
> >>>>>>>>>>>> process.roles
> >>>>>>>>>>>>> configurations can't be empty right now. With this KIP, the
> >> user
> >>>>>> will
> >>>>>>>>>>>>> probably get a different (and more accurate) exception. It
> >> will
> >>>>> be
> >>>>>>>>>> useful
> >>>>>>>>>>>>> to document that.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If
> a
> >>>>> user
> >>>>>>>>>>>> leaves
> >>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than
> an
> >>>>>>>>>> intention.
> >>>>>>>>>>>>> If we automatically treat it as None, the user will never
> >>> realize
> >>>>>> the
> >>>>>>>>>>>>> mistake. Throwing an exception will let the user realize
> there
> >>>>> is a
> >>>>>>>>>>>> mistake.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> >>>>> chia7712@gmail.com
> >>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
> >>>>>> compatibility
> >>>>>>>> is
> >>>>>>>>>>>>>> important.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> However, according to the documentation, we don't allow an
> >>> empty
> >>>>>>>> value
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
> >>>>>> exception
> >>>>>>>>>> for
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>> empty value in version 4.x? This could streamline the code
> >> and
> >>>>>> avoid
> >>>>>>>>>> an
> >>>>>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三
> 上午6:33寫道：
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact any
> >>>>>>>> segments",
> >>>>>>>>>> we
> >>>>>>>>>>>>>>> should be more accurate. We won't delete segments based on
> >>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
> >> continue
> >>>>> to
> >>>>>>>>>> delete
> >>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
> >> log.retention.ms.
> >>>>>>>>>>>> Otherwise,
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>> risk running out of local disk space when remote storage is
> >>>>>>>> enabled.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io>
> >>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Q2. What about existing empty values for
> >>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> >>> during
> >>>>>>>>>>>> upgrade?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised the
> >>>>>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
> >>> other
> >>>>>>>>>> policy.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
> >>>>> approach
> >>>>>>>> is
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
> >>>>> warning
> >>>>>>>>>>>>>> message
> >>>>>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52
> >> 寫道：
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> >>>>> cleanup.policy.
> >>>>>>>> But
> >>>>>>>>>>>>>> now
> >>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None
> can't
> >>> be
> >>>>>>>> used
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document that.
> >>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
> >>>>> config
> >>>>>>>> has
> >>>>>>>>>> an
> >>>>>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check
> >>>>> does.
> >>>>>>>> How
> >>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7133700@gmail.com
> >
> >>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> >>>>>> cleanup.policy
> >>>>>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> >>>>>> configuration.
> >>>>>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>>>>> this setting should not be left empty.
> >>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> >>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list
> is
> >>>>>>>> allowed.
> >>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi Jun, chia

Thanks for all the comments. They have all been addressed in the updated KIP.

I've removed all deprecated-related sections. Additionally, an exception will now be thrown if a 
developer passes an empty validStrings list to the inNonEmpty method.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月6日 凌晨12:46 寫道：
> 
> hi Jiunn-Yang
> 
> The `inNonEmpty` is used to prevent users pass empty config, so should it
> be non-empty too?
> 
> For example, `inNonEmpty()` should throw exception directly.
> 
> Best,
> Chia-Ping
> 
> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：
> 
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the reply. There are still references to the deprecated method.
>> 
>> "stop relying on the deprecated methods"
>> "Finally, the deprecated method will be removed in Kafka 5.0"
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hi Jun, chia,
>>> 
>>> Thanks for all the comments. They have all been addressed in the updated
>>> KIP.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
>>>> 
>>>> hi Jiunn-Yang
>>>> 
>>>> in the "Compatibility, Deprecation, and Migration Plan", could you add
>>>> description to remind readers that "clean.policy=" should be replaced
>> by
>>>> "clean.policy=none" if they do want to disable delete and compact.
>>>> 
>>>> Best,
>>>> Chia-Ping
>>>> 
>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
>>>> 
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> It's fine not to deprecate ValidList#in for now. Could you update the
>>> KIP
>>>>> completely? There are still references to deprecation.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hi Jun,
>>>>>> 
>>>>>> I don’t think we should deprecate the ValidList#in method, as there
>> may
>>>>> be
>>>>>> future scenarios where we want
>>>>>> to allow list configs to be empty. It’s useful to have both methods:
>> in
>>>>>> (which allows empty lists)
>>>>>> and inNonEmpty (which enforces non-empty lists).
>>>>>> 
>>>>>>> Just a minor comment. It would be useful to document that during
>>>>> upgrade,
>>>>>>> if cleanup.policy is empty, the broker will fail to start.
>>>>>> 
>>>>>> I’ve updated the KIP accordingly.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
>>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the reply. Sounds good.
>>>>>>> 
>>>>>>> Just a minor comment. It would be useful to document that during
>>>>> upgrade,
>>>>>>> if cleanup.policy is empty, the broker will fail to start.
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hello Jun,
>>>>>>>> 
>>>>>>>> Since ValidList is a public API, we need to maintain backward
>>>>>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
>>>>>>>> We can update the signature of isNonEmpty() to remove the boolean
>>>>>>>> parameter.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
>>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply.
>>>>>>>>> 
>>>>>>>>> Do we really need isEmptyAllowed? It's awkward since the method
>> name
>>>>> is
>>>>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Jun,
>>>>>>>>>> 
>>>>>>>>>> Thank you for pointing that out.
>>>>>>>>>> 
>>>>>>>>>> I have revised the KIP as follows:
>>>>>>>>>> 
>>>>>>>>>> “In this case, the change does not introduce new invalid
>>>>>> configurations
>>>>>>>> or
>>>>>>>>>> prevent any currently valid setup.
>>>>>>>>>> The main behavioral difference is that we now explicitly throw a
>>>>>>>>>> ConfigException  during the storage format validation phase
>>>>>>>>>> instead of relying on the current behavior.”
>>>>>>>>>> 
>>>>>>>>>> This reflects the correct behavior.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
>>>>>>>>>>> 
>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> "The main behavioral difference introduced by this change is
>> that
>>> a
>>>>>>>>>>> ConfigException  will now be thrown during the storage format
>>>>>>>> validation
>>>>>>>>>>> phase, rather than during server startup."
>>>>>>>>>>> 
>>>>>>>>>>> It seems that currently formating the storage already fails if
>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
>>> empty.
>>>>>> It
>>>>>>>>>> just
>>>>>>>>>>> gets a different exception.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Jun, chia,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for your feedback.
>>>>>>>>>>>> 
>>>>>>>>>>>> I add a new section for this change:
>>>>>>>>>>>> 
>>>>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
>>>>>>>> process.roles
>>>>>>>>>>>> will be
>>>>>>>>>>>> moved from the server startup phase to the storage format
>> phase.
>>>>>>>>>>>> - For cleanup.policy, setting an empty list will now be
>>> considered
>>>>>>>>>> invalid
>>>>>>>>>>>> and will
>>>>>>>>>>>> result in a ConfigException during storage format phase.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
>>>>>>>>>>>> process.roles
>>>>>>>>>>>>> configurations can't be empty right now. With this KIP, the
>> user
>>>>>> will
>>>>>>>>>>>>> probably get a different (and more accurate) exception. It
>> will
>>>>> be
>>>>>>>>>> useful
>>>>>>>>>>>>> to document that.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a
>>>>> user
>>>>>>>>>>>> leaves
>>>>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than an
>>>>>>>>>> intention.
>>>>>>>>>>>>> If we automatically treat it as None, the user will never
>>> realize
>>>>>> the
>>>>>>>>>>>>> mistake. Throwing an exception will let the user realize there
>>>>> is a
>>>>>>>>>>>> mistake.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>>>>> chia7712@gmail.com
>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
>>>>>> compatibility
>>>>>>>> is
>>>>>>>>>>>>>> important.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> However, according to the documentation, we don't allow an
>>> empty
>>>>>>>> value
>>>>>>>>>>>> for
>>>>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
>>>>>> exception
>>>>>>>>>> for
>>>>>>>>>>>> an
>>>>>>>>>>>>>> empty value in version 4.x? This could streamline the code
>> and
>>>>>> avoid
>>>>>>>>>> an
>>>>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regarding "The none policy will not delete or compact any
>>>>>>>> segments",
>>>>>>>>>> we
>>>>>>>>>>>>>>> should be more accurate. We won't delete segments based on
>>>>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
>> continue
>>>>> to
>>>>>>>>>> delete
>>>>>>>>>>>>>>> segments based on log.local.retention.bytes/
>> log.retention.ms.
>>>>>>>>>>>> Otherwise,
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> risk running out of local disk space when remote storage is
>>>>>>>> enabled.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io>
>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
>>> during
>>>>>>>>>>>> upgrade?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised the
>>>>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
>>> other
>>>>>>>>>> policy.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
>>>>> approach
>>>>>>>> is
>>>>>>>>>> to
>>>>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
>>>>> warning
>>>>>>>>>>>>>> message
>>>>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52
>> 寫道：
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
>>>>> cleanup.policy.
>>>>>>>> But
>>>>>>>>>>>>>> now
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> all value combinations are valid. For example, None can't
>>> be
>>>>>>>> used
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document that.
>>>>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
>>>>> config
>>>>>>>> has
>>>>>>>>>> an
>>>>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check
>>>>> does.
>>>>>>>> How
>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
>>>>>> cleanup.policy
>>>>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
>>>>>> configuration.
>>>>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
>>>>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list is
>>>>>>>> allowed.
>>>>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

hi Jiunn-Yang

The `inNonEmpty` is used to prevent users pass empty config, so should it
be non-empty too?

For example, `inNonEmpty()` should throw exception directly.

Best,
Chia-Ping

Jun Rao <ju...@confluent.io.invalid> 於 2025年5月6日 週二 上午12:28寫道：

> Hi, Jiunn-Yang,
>
> Thanks for the reply. There are still references to the deprecated method.
>
> "stop relying on the deprecated methods"
> "Finally, the deprecated method will be removed in Kafka 5.0"
>
> Thanks,
>
> Jun
>
> On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hi Jun, chia,
> >
> > Thanks for all the comments. They have all been addressed in the updated
> > KIP.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> > >
> > > hi Jiunn-Yang
> > >
> > > in the "Compatibility, Deprecation, and Migration Plan", could you add
> > > description to remind readers that "clean.policy=" should be replaced
> by
> > > "clean.policy=none" if they do want to disable delete and compact.
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> > >
> > >> Hi, Jiunn-Yang,
> > >>
> > >> It's fine not to deprecate ValidList#in for now. Could you update the
> > KIP
> > >> completely? There are still references to deprecation.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>
> > >>> Hi Jun,
> > >>>
> > >>> I don’t think we should deprecate the ValidList#in method, as there
> may
> > >> be
> > >>> future scenarios where we want
> > >>> to allow list configs to be empty. It’s useful to have both methods:
> in
> > >>> (which allows empty lists)
> > >>> and inNonEmpty (which enforces non-empty lists).
> > >>>
> > >>>> Just a minor comment. It would be useful to document that during
> > >> upgrade,
> > >>>> if cleanup.policy is empty, the broker will fail to start.
> > >>>
> > >>> I’ve updated the KIP accordingly.
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> > >>>>
> > >>>> Hi, Jiunn-Yang,
> > >>>>
> > >>>> Thanks for the reply. Sounds good.
> > >>>>
> > >>>> Just a minor comment. It would be useful to document that during
> > >> upgrade,
> > >>>> if cleanup.policy is empty, the broker will fail to start.
> > >>>>
> > >>>> Jun
> > >>>>
> > >>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>
> > >>>>> Hello Jun,
> > >>>>>
> > >>>>> Since ValidList is a public API, we need to maintain backward
> > >>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
> > >>>>> We can update the signature of isNonEmpty() to remove the boolean
> > >>>>> parameter.
> > >>>>>
> > >>>>> Best Regards,
> > >>>>> Jiunn-Yang
> > >>>>>
> > >>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> > >>>>>>
> > >>>>>> Hi, Jiunn-Yang,
> > >>>>>>
> > >>>>>> Thanks for the reply.
> > >>>>>>
> > >>>>>> Do we really need isEmptyAllowed? It's awkward since the method
> name
> > >> is
> > >>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>>
> > >>>>>> Jun
> > >>>>>>
> > >>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> Hi Jun,
> > >>>>>>>
> > >>>>>>> Thank you for pointing that out.
> > >>>>>>>
> > >>>>>>> I have revised the KIP as follows:
> > >>>>>>>
> > >>>>>>> “In this case, the change does not introduce new invalid
> > >>> configurations
> > >>>>> or
> > >>>>>>> prevent any currently valid setup.
> > >>>>>>> The main behavioral difference is that we now explicitly throw a
> > >>>>>>> ConfigException  during the storage format validation phase
> > >>>>>>> instead of relying on the current behavior.”
> > >>>>>>>
> > >>>>>>> This reflects the correct behavior.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>>>
> > >>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> > >>>>>>>>
> > >>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>
> > >>>>>>>> "The main behavioral difference introduced by this change is
> that
> > a
> > >>>>>>>> ConfigException  will now be thrown during the storage format
> > >>>>> validation
> > >>>>>>>> phase, rather than during server startup."
> > >>>>>>>>
> > >>>>>>>> It seems that currently formating the storage already fails if
> > >>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
> > empty.
> > >>> It
> > >>>>>>> just
> > >>>>>>>> gets a different exception.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Jun, chia,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for your feedback.
> > >>>>>>>>>
> > >>>>>>>>> I add a new section for this change:
> > >>>>>>>>>
> > >>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
> > >>>>> process.roles
> > >>>>>>>>> will be
> > >>>>>>>>> moved from the server startup phase to the storage format
> phase.
> > >>>>>>>>> - For cleanup.policy, setting an empty list will now be
> > considered
> > >>>>>>> invalid
> > >>>>>>>>> and will
> > >>>>>>>>> result in a ConfigException during storage format phase.
> > >>>>>>>>>
> > >>>>>>>>> Best Regards,
> > >>>>>>>>> Jiunn-Yang
> > >>>>>>>>>
> > >>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
> > >>>>>>>>>>
> > >>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>
> > >>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
> > >>>>>>>>> process.roles
> > >>>>>>>>>> configurations can't be empty right now. With this KIP, the
> user
> > >>> will
> > >>>>>>>>>> probably get a different (and more accurate) exception. It
> will
> > >> be
> > >>>>>>> useful
> > >>>>>>>>>> to document that.
> > >>>>>>>>>>
> > >>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a
> > >> user
> > >>>>>>>>> leaves
> > >>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than an
> > >>>>>>> intention.
> > >>>>>>>>>> If we automatically treat it as None, the user will never
> > realize
> > >>> the
> > >>>>>>>>>> mistake. Throwing an exception will let the user realize there
> > >> is a
> > >>>>>>>>> mistake.
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> > >> chia7712@gmail.com
> > >>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> hi Jiunn-Yang,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
> > >>> compatibility
> > >>>>> is
> > >>>>>>>>>>> important.
> > >>>>>>>>>>>
> > >>>>>>>>>>> However, according to the documentation, we don't allow an
> > empty
> > >>>>> value
> > >>>>>>>>> for
> > >>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
> > >>> exception
> > >>>>>>> for
> > >>>>>>>>> an
> > >>>>>>>>>>> empty value in version 4.x? This could streamline the code
> and
> > >>> avoid
> > >>>>>>> an
> > >>>>>>>>>>> extra deprecation cycle.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>
> > >>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Regarding "The none policy will not delete or compact any
> > >>>>> segments",
> > >>>>>>> we
> > >>>>>>>>>>>> should be more accurate. We won't delete segments based on
> > >>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should
> continue
> > >> to
> > >>>>>>> delete
> > >>>>>>>>>>>> segments based on log.local.retention.bytes/
> log.retention.ms.
> > >>>>>>>>> Otherwise,
> > >>>>>>>>>>>> we
> > >>>>>>>>>>>> risk running out of local disk space when remote storage is
> > >>>>> enabled.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io>
> > >>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Q2. What about existing empty values for
> > >>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> > during
> > >>>>>>>>> upgrade?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com>
> > >> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for review this KIP.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Q1 & Q3:
> > >>>>>>>>>>>>>> I’ve updated the method name accordingly and revised the
> > >>>>>>>>>>> cleanup.policy
> > >>>>>>>>>>>>>> documentation
> > >>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
> > other
> > >>>>>>> policy.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Q2:
> > >>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
> > >> approach
> > >>>>> is
> > >>>>>>> to
> > >>>>>>>>>>>>>> apply the none policy
> > >>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
> > >> warning
> > >>>>>>>>>>> message
> > >>>>>>>>>>>>>> will be emitted to inform users
> > >>>>>>>>>>>>>> of the upcoming change.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52
> 寫道：
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> > >> cleanup.policy.
> > >>>>> But
> > >>>>>>>>>>> now
> > >>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>> all value combinations are valid. For example, None can't
> > be
> > >>>>> used
> > >>>>>>>>>>> with
> > >>>>>>>>>>>>>>> Delete or Compact. It would be useful to document that.
> > >>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
> > >> config
> > >>>>> has
> > >>>>>>> an
> > >>>>>>>>>>>>>> empty
> > >>>>>>>>>>>>>>> list.
> > >>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check
> > >> does.
> > >>>>> How
> > >>>>>>>>>>>> about
> > >>>>>>>>>>>>>>> sth like inNonEmpty ?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com>
> > >>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> > >>> cleanup.policy
> > >>>>>>>>>>>>>> shouldn't
> > >>>>>>>>>>>>>>>> be empty
> > >>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> > >>> configuration.
> > >>>>>>>>>>>>>> Currently,
> > >>>>>>>>>>>>>>>> this setting should not be left empty.
> > >>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> > >>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list is
> > >>>>> allowed.
> > >>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply. There are still references to the deprecated method.

"stop relying on the deprecated methods"
"Finally, the deprecated method will be removed in Kafka 5.0"

Thanks,

Jun

On Sat, May 3, 2025 at 7:42 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hi Jun, chia,
>
> Thanks for all the comments. They have all been addressed in the updated
> KIP.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> >
> > hi Jiunn-Yang
> >
> > in the "Compatibility, Deprecation, and Migration Plan", could you add
> > description to remind readers that "clean.policy=" should be replaced by
> > "clean.policy=none" if they do want to disable delete and compact.
> >
> > Best,
> > Chia-Ping
> >
> > Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> >
> >> Hi, Jiunn-Yang,
> >>
> >> It's fine not to deprecate ValidList#in for now. Could you update the
> KIP
> >> completely? There are still references to deprecation.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hi Jun,
> >>>
> >>> I don’t think we should deprecate the ValidList#in method, as there may
> >> be
> >>> future scenarios where we want
> >>> to allow list configs to be empty. It’s useful to have both methods: in
> >>> (which allows empty lists)
> >>> and inNonEmpty (which enforces non-empty lists).
> >>>
> >>>> Just a minor comment. It would be useful to document that during
> >> upgrade,
> >>>> if cleanup.policy is empty, the broker will fail to start.
> >>>
> >>> I’ve updated the KIP accordingly.
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> >>>>
> >>>> Hi, Jiunn-Yang,
> >>>>
> >>>> Thanks for the reply. Sounds good.
> >>>>
> >>>> Just a minor comment. It would be useful to document that during
> >> upgrade,
> >>>> if cleanup.policy is empty, the broker will fail to start.
> >>>>
> >>>> Jun
> >>>>
> >>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>
> >>>>> Hello Jun,
> >>>>>
> >>>>> Since ValidList is a public API, we need to maintain backward
> >>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
> >>>>> We can update the signature of isNonEmpty() to remove the boolean
> >>>>> parameter.
> >>>>>
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> >>>>>>
> >>>>>> Hi, Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> Do we really need isEmptyAllowed? It's awkward since the method name
> >> is
> >>>>>> inNonEmpty. Also, it's not clear when it's set to true.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi Jun,
> >>>>>>>
> >>>>>>> Thank you for pointing that out.
> >>>>>>>
> >>>>>>> I have revised the KIP as follows:
> >>>>>>>
> >>>>>>> “In this case, the change does not introduce new invalid
> >>> configurations
> >>>>> or
> >>>>>>> prevent any currently valid setup.
> >>>>>>> The main behavioral difference is that we now explicitly throw a
> >>>>>>> ConfigException  during the storage format validation phase
> >>>>>>> instead of relying on the current behavior.”
> >>>>>>>
> >>>>>>> This reflects the correct behavior.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> >>>>>>>>
> >>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> "The main behavioral difference introduced by this change is that
> a
> >>>>>>>> ConfigException  will now be thrown during the storage format
> >>>>> validation
> >>>>>>>> phase, rather than during server startup."
> >>>>>>>>
> >>>>>>>> It seems that currently formating the storage already fails if
> >>>>>>>> group.coordinator.rebalance.protocols and process.roles  are
> empty.
> >>> It
> >>>>>>> just
> >>>>>>>> gets a different exception.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Jun, chia,
> >>>>>>>>>
> >>>>>>>>> Thanks for your feedback.
> >>>>>>>>>
> >>>>>>>>> I add a new section for this change:
> >>>>>>>>>
> >>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
> >>>>> process.roles
> >>>>>>>>> will be
> >>>>>>>>> moved from the server startup phase to the storage format phase.
> >>>>>>>>> - For cleanup.policy, setting an empty list will now be
> considered
> >>>>>>> invalid
> >>>>>>>>> and will
> >>>>>>>>> result in a ConfigException during storage format phase.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
> >>>>>>>>>>
> >>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>
> >>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
> >>>>>>>>> process.roles
> >>>>>>>>>> configurations can't be empty right now. With this KIP, the user
> >>> will
> >>>>>>>>>> probably get a different (and more accurate) exception. It will
> >> be
> >>>>>>> useful
> >>>>>>>>>> to document that.
> >>>>>>>>>>
> >>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a
> >> user
> >>>>>>>>> leaves
> >>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than an
> >>>>>>> intention.
> >>>>>>>>>> If we automatically treat it as None, the user will never
> realize
> >>> the
> >>>>>>>>>> mistake. Throwing an exception will let the user realize there
> >> is a
> >>>>>>>>> mistake.
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> >> chia7712@gmail.com
> >>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> hi Jiunn-Yang,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
> >>> compatibility
> >>>>> is
> >>>>>>>>>>> important.
> >>>>>>>>>>>
> >>>>>>>>>>> However, according to the documentation, we don't allow an
> empty
> >>>>> value
> >>>>>>>>> for
> >>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
> >>> exception
> >>>>>>> for
> >>>>>>>>> an
> >>>>>>>>>>> empty value in version 4.x? This could streamline the code and
> >>> avoid
> >>>>>>> an
> >>>>>>>>>>> extra deprecation cycle.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>
> >>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regarding "The none policy will not delete or compact any
> >>>>> segments",
> >>>>>>> we
> >>>>>>>>>>>> should be more accurate. We won't delete segments based on
> >>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should continue
> >> to
> >>>>>>> delete
> >>>>>>>>>>>> segments based on log.local.retention.bytes/log.retention.ms.
> >>>>>>>>> Otherwise,
> >>>>>>>>>>>> we
> >>>>>>>>>>>> risk running out of local disk space when remote storage is
> >>>>> enabled.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io>
> >>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Q2. What about existing empty values for
> >>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles
> during
> >>>>>>>>> upgrade?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com>
> >> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>>>>> I’ve updated the method name accordingly and revised the
> >>>>>>>>>>> cleanup.policy
> >>>>>>>>>>>>>> documentation
> >>>>>>>>>>>>>> to clarify that the none policy cannot be used with any
> other
> >>>>>>> policy.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Q2:
> >>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
> >> approach
> >>>>> is
> >>>>>>> to
> >>>>>>>>>>>>>> apply the none policy
> >>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
> >> warning
> >>>>>>>>>>> message
> >>>>>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52 寫道：
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
> >> cleanup.policy.
> >>>>> But
> >>>>>>>>>>> now
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>> all value combinations are valid. For example, None can't
> be
> >>>>> used
> >>>>>>>>>>> with
> >>>>>>>>>>>>>>> Delete or Compact. It would be useful to document that.
> >>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
> >> config
> >>>>> has
> >>>>>>> an
> >>>>>>>>>>>>>> empty
> >>>>>>>>>>>>>>> list.
> >>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check
> >> does.
> >>>>> How
> >>>>>>>>>>>> about
> >>>>>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com>
> >>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> >>> cleanup.policy
> >>>>>>>>>>>>>> shouldn't
> >>>>>>>>>>>>>>>> be empty
> >>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> >>> configuration.
> >>>>>>>>>>>>>> Currently,
> >>>>>>>>>>>>>>>> this setting should not be left empty.
> >>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
> >>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list is
> >>>>> allowed.
> >>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi Jun, chia,

Thanks for all the comments. They have all been addressed in the updated KIP.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2025年5月3日 凌晨2:03 寫道：
> 
> hi Jiunn-Yang
> 
> in the "Compatibility, Deprecation, and Migration Plan", could you add
> description to remind readers that "clean.policy=" should be replaced by
> "clean.policy=none" if they do want to disable delete and compact.
> 
> Best,
> Chia-Ping
> 
> Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：
> 
>> Hi, Jiunn-Yang,
>> 
>> It's fine not to deprecate ValidList#in for now. Could you update the KIP
>> completely? There are still references to deprecation.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hi Jun,
>>> 
>>> I don’t think we should deprecate the ValidList#in method, as there may
>> be
>>> future scenarios where we want
>>> to allow list configs to be empty. It’s useful to have both methods: in
>>> (which allows empty lists)
>>> and inNonEmpty (which enforces non-empty lists).
>>> 
>>>> Just a minor comment. It would be useful to document that during
>> upgrade,
>>>> if cleanup.policy is empty, the broker will fail to start.
>>> 
>>> I’ve updated the KIP accordingly.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
>>>> 
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Thanks for the reply. Sounds good.
>>>> 
>>>> Just a minor comment. It would be useful to document that during
>> upgrade,
>>>> if cleanup.policy is empty, the broker will fail to start.
>>>> 
>>>> Jun
>>>> 
>>>> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hello Jun,
>>>>> 
>>>>> Since ValidList is a public API, we need to maintain backward
>>>>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
>>>>> We can update the signature of isNonEmpty() to remove the boolean
>>>>> parameter.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
>>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply.
>>>>>> 
>>>>>> Do we really need isEmptyAllowed? It's awkward since the method name
>> is
>>>>>> inNonEmpty. Also, it's not clear when it's set to true.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi Jun,
>>>>>>> 
>>>>>>> Thank you for pointing that out.
>>>>>>> 
>>>>>>> I have revised the KIP as follows:
>>>>>>> 
>>>>>>> “In this case, the change does not introduce new invalid
>>> configurations
>>>>> or
>>>>>>> prevent any currently valid setup.
>>>>>>> The main behavioral difference is that we now explicitly throw a
>>>>>>> ConfigException  during the storage format validation phase
>>>>>>> instead of relying on the current behavior.”
>>>>>>> 
>>>>>>> This reflects the correct behavior.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
>>>>>>>> 
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> "The main behavioral difference introduced by this change is that a
>>>>>>>> ConfigException  will now be thrown during the storage format
>>>>> validation
>>>>>>>> phase, rather than during server startup."
>>>>>>>> 
>>>>>>>> It seems that currently formating the storage already fails if
>>>>>>>> group.coordinator.rebalance.protocols and process.roles  are empty.
>>> It
>>>>>>> just
>>>>>>>> gets a different exception.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jun, chia,
>>>>>>>>> 
>>>>>>>>> Thanks for your feedback.
>>>>>>>>> 
>>>>>>>>> I add a new section for this change:
>>>>>>>>> 
>>>>>>>>> - Validation for group.coordinator.rebalance.protocols and
>>>>> process.roles
>>>>>>>>> will be
>>>>>>>>> moved from the server startup phase to the storage format phase.
>>>>>>>>> - For cleanup.policy, setting an empty list will now be considered
>>>>>>> invalid
>>>>>>>>> and will
>>>>>>>>> result in a ConfigException during storage format phase.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
>>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply.
>>>>>>>>>> 
>>>>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
>>>>>>>>> process.roles
>>>>>>>>>> configurations can't be empty right now. With this KIP, the user
>>> will
>>>>>>>>>> probably get a different (and more accurate) exception. It will
>> be
>>>>>>> useful
>>>>>>>>>> to document that.
>>>>>>>>>> 
>>>>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a
>> user
>>>>>>>>> leaves
>>>>>>>>>> cleanup.policy empty, it's more likely to be a mistake than an
>>>>>>> intention.
>>>>>>>>>> If we automatically treat it as None, the user will never realize
>>> the
>>>>>>>>>> mistake. Throwing an exception will let the user realize there
>> is a
>>>>>>>>> mistake.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
>> chia7712@gmail.com
>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> hi Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the KIP, and I understand that maintaining
>>> compatibility
>>>>> is
>>>>>>>>>>> important.
>>>>>>>>>>> 
>>>>>>>>>>> However, according to the documentation, we don't allow an empty
>>>>> value
>>>>>>>>> for
>>>>>>>>>>> the cleanup.policy. Hence, should we consider throwing an
>>> exception
>>>>>>> for
>>>>>>>>> an
>>>>>>>>>>> empty value in version 4.x? This could streamline the code and
>>> avoid
>>>>>>> an
>>>>>>>>>>> extra deprecation cycle.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Regarding "The none policy will not delete or compact any
>>>>> segments",
>>>>>>> we
>>>>>>>>>>>> should be more accurate. We won't delete segments based on
>>>>>>>>>>>> log.retention.bytes/log.retention.ms, but we should continue
>> to
>>>>>>> delete
>>>>>>>>>>>> segments based on log.local.retention.bytes/log.retention.ms.
>>>>>>>>> Otherwise,
>>>>>>>>>>>> we
>>>>>>>>>>>> risk running out of local disk space when remote storage is
>>>>> enabled.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io>
>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles during
>>>>>>>>> upgrade?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>>>>> I’ve updated the method name accordingly and revised the
>>>>>>>>>>> cleanup.policy
>>>>>>>>>>>>>> documentation
>>>>>>>>>>>>>> to clarify that the none policy cannot be used with any other
>>>>>>> policy.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Q2:
>>>>>>>>>>>>>> For users currently using an empty cleanup.policy, the
>> approach
>>>>> is
>>>>>>> to
>>>>>>>>>>>>>> apply the none policy
>>>>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
>> warning
>>>>>>>>>>> message
>>>>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52 寫道：
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. It's fine to introduce a new value None for
>> cleanup.policy.
>>>>> But
>>>>>>>>>>> now
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> all value combinations are valid. For example, None can't be
>>>>> used
>>>>>>>>>>> with
>>>>>>>>>>>>>>> Delete or Compact. It would be useful to document that.
>>>>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
>> config
>>>>> has
>>>>>>> an
>>>>>>>>>>>>>> empty
>>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check
>> does.
>>>>> How
>>>>>>>>>>>> about
>>>>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
>>> cleanup.policy
>>>>>>>>>>>>>> shouldn't
>>>>>>>>>>>>>>>> be empty
>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
>>> configuration.
>>>>>>>>>>>>>> Currently,
>>>>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>>>>> Therefore, there are two proposed improvements:
>>>>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list is
>>>>> allowed.
>>>>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Chia-Ping Tsai <ch...@gmail.com>]

hi Jiunn-Yang

in the "Compatibility, Deprecation, and Migration Plan", could you add
description to remind readers that "clean.policy=" should be replaced by
"clean.policy=none" if they do want to disable delete and compact.

Best,
Chia-Ping

Jun Rao <ju...@confluent.io.invalid> 於 2025年5月3日 週六 上午1:55寫道：

> Hi, Jiunn-Yang,
>
> It's fine not to deprecate ValidList#in for now. Could you update the KIP
> completely? There are still references to deprecation.
>
> Thanks,
>
> Jun
>
> On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > I don’t think we should deprecate the ValidList#in method, as there may
> be
> > future scenarios where we want
> > to allow list configs to be empty. It’s useful to have both methods: in
> > (which allows empty lists)
> > and inNonEmpty (which enforces non-empty lists).
> >
> > > Just a minor comment. It would be useful to document that during
> upgrade,
> > > if cleanup.policy is empty, the broker will fail to start.
> >
> > I’ve updated the KIP accordingly.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> > >
> > > Hi, Jiunn-Yang,
> > >
> > > Thanks for the reply. Sounds good.
> > >
> > > Just a minor comment. It would be useful to document that during
> upgrade,
> > > if cleanup.policy is empty, the broker will fail to start.
> > >
> > > Jun
> > >
> > > On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >
> > >> Hello Jun,
> > >>
> > >> Since ValidList is a public API, we need to maintain backward
> > >> compatibility. Therefore, the isEmptyAllowed flag is necessary.
> > >> We can update the signature of isNonEmpty() to remove the boolean
> > >> parameter.
> > >>
> > >> Best Regards,
> > >> Jiunn-Yang
> > >>
> > >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> > >>>
> > >>> Hi, Jiunn-Yang,
> > >>>
> > >>> Thanks for the reply.
> > >>>
> > >>> Do we really need isEmptyAllowed? It's awkward since the method name
> is
> > >>> inNonEmpty. Also, it's not clear when it's set to true.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jun
> > >>>
> > >>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>
> > >>>> Hi Jun,
> > >>>>
> > >>>> Thank you for pointing that out.
> > >>>>
> > >>>> I have revised the KIP as follows:
> > >>>>
> > >>>> “In this case, the change does not introduce new invalid
> > configurations
> > >> or
> > >>>> prevent any currently valid setup.
> > >>>> The main behavioral difference is that we now explicitly throw a
> > >>>> ConfigException  during the storage format validation phase
> > >>>> instead of relying on the current behavior.”
> > >>>>
> > >>>> This reflects the correct behavior.
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> > >>>>>
> > >>>>> Hi, Jiunn-Yang,
> > >>>>>
> > >>>>> "The main behavioral difference introduced by this change is that a
> > >>>>> ConfigException  will now be thrown during the storage format
> > >> validation
> > >>>>> phase, rather than during server startup."
> > >>>>>
> > >>>>> It seems that currently formating the storage already fails if
> > >>>>> group.coordinator.rebalance.protocols and process.roles  are empty.
> > It
> > >>>> just
> > >>>>> gets a different exception.
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Jun
> > >>>>>
> > >>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>
> > >>>>>> Hi Jun, chia,
> > >>>>>>
> > >>>>>> Thanks for your feedback.
> > >>>>>>
> > >>>>>> I add a new section for this change:
> > >>>>>>
> > >>>>>> - Validation for group.coordinator.rebalance.protocols and
> > >> process.roles
> > >>>>>> will be
> > >>>>>> moved from the server startup phase to the storage format phase.
> > >>>>>> - For cleanup.policy, setting an empty list will now be considered
> > >>>> invalid
> > >>>>>> and will
> > >>>>>> result in a ConfigException during storage format phase.
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>> Jiunn-Yang
> > >>>>>>
> > >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
> > >>>>>>>
> > >>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>
> > >>>>>>> Thanks for the reply.
> > >>>>>>>
> > >>>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
> > >>>>>> process.roles
> > >>>>>>> configurations can't be empty right now. With this KIP, the user
> > will
> > >>>>>>> probably get a different (and more accurate) exception. It will
> be
> > >>>> useful
> > >>>>>>> to document that.
> > >>>>>>>
> > >>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a
> user
> > >>>>>> leaves
> > >>>>>>> cleanup.policy empty, it's more likely to be a mistake than an
> > >>>> intention.
> > >>>>>>> If we automatically treat it as None, the user will never realize
> > the
> > >>>>>>> mistake. Throwing an exception will let the user realize there
> is a
> > >>>>>> mistake.
> > >>>>>>>
> > >>>>>>> Jun
> > >>>>>>>
> > >>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <
> chia7712@gmail.com
> > >
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> hi Jiunn-Yang,
> > >>>>>>>>
> > >>>>>>>> Thanks for the KIP, and I understand that maintaining
> > compatibility
> > >> is
> > >>>>>>>> important.
> > >>>>>>>>
> > >>>>>>>> However, according to the documentation, we don't allow an empty
> > >> value
> > >>>>>> for
> > >>>>>>>> the cleanup.policy. Hence, should we consider throwing an
> > exception
> > >>>> for
> > >>>>>> an
> > >>>>>>>> empty value in version 4.x? This could streamline the code and
> > avoid
> > >>>> an
> > >>>>>>>> extra deprecation cycle.
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> Chia-Ping
> > >>>>>>>>
> > >>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
> > >>>>>>>>
> > >>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>
> > >>>>>>>>> Regarding "The none policy will not delete or compact any
> > >> segments",
> > >>>> we
> > >>>>>>>>> should be more accurate. We won't delete segments based on
> > >>>>>>>>> log.retention.bytes/log.retention.ms, but we should continue
> to
> > >>>> delete
> > >>>>>>>>> segments based on log.local.retention.bytes/log.retention.ms.
> > >>>>>> Otherwise,
> > >>>>>>>>> we
> > >>>>>>>>> risk running out of local disk space when remote storage is
> > >> enabled.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>>
> > >>>>>>>>> Jun
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io>
> > wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>
> > >>>>>>>>>> Q2. What about existing empty values for
> > >>>>>>>>>> group.coordinator.rebalance.protocols and process.roles during
> > >>>>>> upgrade?
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for review this KIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Q1 & Q3:
> > >>>>>>>>>>> I’ve updated the method name accordingly and revised the
> > >>>>>>>> cleanup.policy
> > >>>>>>>>>>> documentation
> > >>>>>>>>>>> to clarify that the none policy cannot be used with any other
> > >>>> policy.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Q2:
> > >>>>>>>>>>> For users currently using an empty cleanup.policy, the
> approach
> > >> is
> > >>>> to
> > >>>>>>>>>>> apply the none policy
> > >>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a
> warning
> > >>>>>>>> message
> > >>>>>>>>>>> will be emitted to inform users
> > >>>>>>>>>>> of the upcoming change.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best Regards,
> > >>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52 寫道：
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the KIP. A few comments.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. It's fine to introduce a new value None for
> cleanup.policy.
> > >> But
> > >>>>>>>> now
> > >>>>>>>>>>> not
> > >>>>>>>>>>>> all value combinations are valid. For example, None can't be
> > >> used
> > >>>>>>>> with
> > >>>>>>>>>>>> Delete or Compact. It would be useful to document that.
> > >>>>>>>>>>>> 2. What's the behavior during upgrade when an existing
> config
> > >> has
> > >>>> an
> > >>>>>>>>>>> empty
> > >>>>>>>>>>>> list.
> > >>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check
> does.
> > >> How
> > >>>>>>>>> about
> > >>>>>>>>>>>> sth like inNonEmpty ?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> > cleanup.policy
> > >>>>>>>>>>> shouldn't
> > >>>>>>>>>>>>> be empty
> > >>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> > configuration.
> > >>>>>>>>>>> Currently,
> > >>>>>>>>>>>>> this setting should not be left empty.
> > >>>>>>>>>>>>> Therefore, there are two proposed improvements:
> > >>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list is
> > >> allowed.
> > >>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

It's fine not to deprecate ValidList#in for now. Could you update the KIP
completely? There are still references to deprecation.

Thanks,

Jun

On Fri, May 2, 2025 at 4:59 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hi Jun,
>
> I don’t think we should deprecate the ValidList#in method, as there may be
> future scenarios where we want
> to allow list configs to be empty. It’s useful to have both methods: in
> (which allows empty lists)
> and inNonEmpty (which enforces non-empty lists).
>
> > Just a minor comment. It would be useful to document that during upgrade,
> > if cleanup.policy is empty, the broker will fail to start.
>
> I’ve updated the KIP accordingly.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the reply. Sounds good.
> >
> > Just a minor comment. It would be useful to document that during upgrade,
> > if cleanup.policy is empty, the broker will fail to start.
> >
> > Jun
> >
> > On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello Jun,
> >>
> >> Since ValidList is a public API, we need to maintain backward
> >> compatibility. Therefore, the isEmptyAllowed flag is necessary.
> >> We can update the signature of isNonEmpty() to remove the boolean
> >> parameter.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> >>>
> >>> Hi, Jiunn-Yang,
> >>>
> >>> Thanks for the reply.
> >>>
> >>> Do we really need isEmptyAllowed? It's awkward since the method name is
> >>> inNonEmpty. Also, it's not clear when it's set to true.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>
> >>>> Hi Jun,
> >>>>
> >>>> Thank you for pointing that out.
> >>>>
> >>>> I have revised the KIP as follows:
> >>>>
> >>>> “In this case, the change does not introduce new invalid
> configurations
> >> or
> >>>> prevent any currently valid setup.
> >>>> The main behavioral difference is that we now explicitly throw a
> >>>> ConfigException  during the storage format validation phase
> >>>> instead of relying on the current behavior.”
> >>>>
> >>>> This reflects the correct behavior.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> >>>>>
> >>>>> Hi, Jiunn-Yang,
> >>>>>
> >>>>> "The main behavioral difference introduced by this change is that a
> >>>>> ConfigException  will now be thrown during the storage format
> >> validation
> >>>>> phase, rather than during server startup."
> >>>>>
> >>>>> It seems that currently formating the storage already fails if
> >>>>> group.coordinator.rebalance.protocols and process.roles  are empty.
> It
> >>>> just
> >>>>> gets a different exception.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>
> >>>>>> Hi Jun, chia,
> >>>>>>
> >>>>>> Thanks for your feedback.
> >>>>>>
> >>>>>> I add a new section for this change:
> >>>>>>
> >>>>>> - Validation for group.coordinator.rebalance.protocols and
> >> process.roles
> >>>>>> will be
> >>>>>> moved from the server startup phase to the storage format phase.
> >>>>>> - For cleanup.policy, setting an empty list will now be considered
> >>>> invalid
> >>>>>> and will
> >>>>>> result in a ConfigException during storage format phase.
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
> >>>>>>>
> >>>>>>> Hi, Jiunn-Yang,
> >>>>>>>
> >>>>>>> Thanks for the reply.
> >>>>>>>
> >>>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
> >>>>>> process.roles
> >>>>>>> configurations can't be empty right now. With this KIP, the user
> will
> >>>>>>> probably get a different (and more accurate) exception. It will be
> >>>> useful
> >>>>>>> to document that.
> >>>>>>>
> >>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a user
> >>>>>> leaves
> >>>>>>> cleanup.policy empty, it's more likely to be a mistake than an
> >>>> intention.
> >>>>>>> If we automatically treat it as None, the user will never realize
> the
> >>>>>>> mistake. Throwing an exception will let the user realize there is a
> >>>>>> mistake.
> >>>>>>>
> >>>>>>> Jun
> >>>>>>>
> >>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <chia7712@gmail.com
> >
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> hi Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> Thanks for the KIP, and I understand that maintaining
> compatibility
> >> is
> >>>>>>>> important.
> >>>>>>>>
> >>>>>>>> However, according to the documentation, we don't allow an empty
> >> value
> >>>>>> for
> >>>>>>>> the cleanup.policy. Hence, should we consider throwing an
> exception
> >>>> for
> >>>>>> an
> >>>>>>>> empty value in version 4.x? This could streamline the code and
> avoid
> >>>> an
> >>>>>>>> extra deprecation cycle.
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>>
> >>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
> >>>>>>>>
> >>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>
> >>>>>>>>> Regarding "The none policy will not delete or compact any
> >> segments",
> >>>> we
> >>>>>>>>> should be more accurate. We won't delete segments based on
> >>>>>>>>> log.retention.bytes/log.retention.ms, but we should continue to
> >>>> delete
> >>>>>>>>> segments based on log.local.retention.bytes/log.retention.ms.
> >>>>>> Otherwise,
> >>>>>>>>> we
> >>>>>>>>> risk running out of local disk space when remote storage is
> >> enabled.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Jun
> >>>>>>>>>
> >>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io>
> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>
> >>>>>>>>>> Q2. What about existing empty values for
> >>>>>>>>>> group.coordinator.rebalance.protocols and process.roles during
> >>>>>> upgrade?
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>>>
> >>>>>>>>>>> Q1 & Q3:
> >>>>>>>>>>> I’ve updated the method name accordingly and revised the
> >>>>>>>> cleanup.policy
> >>>>>>>>>>> documentation
> >>>>>>>>>>> to clarify that the none policy cannot be used with any other
> >>>> policy.
> >>>>>>>>>>>
> >>>>>>>>>>> Q2:
> >>>>>>>>>>> For users currently using an empty cleanup.policy, the approach
> >> is
> >>>> to
> >>>>>>>>>>> apply the none policy
> >>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a warning
> >>>>>>>> message
> >>>>>>>>>>> will be emitted to inform users
> >>>>>>>>>>> of the upcoming change.
> >>>>>>>>>>>
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>
> >>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52 寫道：
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. It's fine to introduce a new value None for cleanup.policy.
> >> But
> >>>>>>>> now
> >>>>>>>>>>> not
> >>>>>>>>>>>> all value combinations are valid. For example, None can't be
> >> used
> >>>>>>>> with
> >>>>>>>>>>>> Delete or Compact. It would be useful to document that.
> >>>>>>>>>>>> 2. What's the behavior during upgrade when an existing config
> >> has
> >>>> an
> >>>>>>>>>>> empty
> >>>>>>>>>>>> list.
> >>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check does.
> >> How
> >>>>>>>>> about
> >>>>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would like to start a discussion on KIP-1161:
> cleanup.policy
> >>>>>>>>>>> shouldn't
> >>>>>>>>>>>>> be empty
> >>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> This proposal aims to improve the cleanup.policy
> configuration.
> >>>>>>>>>>> Currently,
> >>>>>>>>>>>>> this setting should not be left empty.
> >>>>>>>>>>>>> Therefore, there are two proposed improvements:
> >>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list is
> >> allowed.
> >>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[黃竣陽 <s7...@gmail.com>]

Hi Jun, 

I don’t think we should deprecate the ValidList#in method, as there may be future scenarios where we want 
to allow list configs to be empty. It’s useful to have both methods: in (which allows empty lists) 
and inNonEmpty (which enforces non-empty lists).

> Just a minor comment. It would be useful to document that during upgrade,
> if cleanup.policy is empty, the broker will fail to start.

I’ve updated the KIP accordingly.

Best Regards,
Jiunn-Yang

> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月2日 凌晨12:50 寫道：
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply. Sounds good.
> 
> Just a minor comment. It would be useful to document that during upgrade,
> if cleanup.policy is empty, the broker will fail to start.
> 
> Jun
> 
> On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello Jun,
>> 
>> Since ValidList is a public API, we need to maintain backward
>> compatibility. Therefore, the isEmptyAllowed flag is necessary.
>> We can update the signature of isNonEmpty() to remove the boolean
>> parameter.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
>>> 
>>> Hi, Jiunn-Yang,
>>> 
>>> Thanks for the reply.
>>> 
>>> Do we really need isEmptyAllowed? It's awkward since the method name is
>>> inNonEmpty. Also, it's not clear when it's set to true.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hi Jun,
>>>> 
>>>> Thank you for pointing that out.
>>>> 
>>>> I have revised the KIP as follows:
>>>> 
>>>> “In this case, the change does not introduce new invalid configurations
>> or
>>>> prevent any currently valid setup.
>>>> The main behavioral difference is that we now explicitly throw a
>>>> ConfigException  during the storage format validation phase
>>>> instead of relying on the current behavior.”
>>>> 
>>>> This reflects the correct behavior.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
>>>>> 
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> "The main behavioral difference introduced by this change is that a
>>>>> ConfigException  will now be thrown during the storage format
>> validation
>>>>> phase, rather than during server startup."
>>>>> 
>>>>> It seems that currently formating the storage already fails if
>>>>> group.coordinator.rebalance.protocols and process.roles  are empty. It
>>>> just
>>>>> gets a different exception.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hi Jun, chia,
>>>>>> 
>>>>>> Thanks for your feedback.
>>>>>> 
>>>>>> I add a new section for this change:
>>>>>> 
>>>>>> - Validation for group.coordinator.rebalance.protocols and
>> process.roles
>>>>>> will be
>>>>>> moved from the server startup phase to the storage format phase.
>>>>>> - For cleanup.policy, setting an empty list will now be considered
>>>> invalid
>>>>>> and will
>>>>>> result in a ConfigException during storage format phase.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
>>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the reply.
>>>>>>> 
>>>>>>> Q2. It's true that group.coordinator.rebalance.protocols and
>>>>>> process.roles
>>>>>>> configurations can't be empty right now. With this KIP, the user will
>>>>>>> probably get a different (and more accurate) exception. It will be
>>>> useful
>>>>>>> to document that.
>>>>>>> 
>>>>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a user
>>>>>> leaves
>>>>>>> cleanup.policy empty, it's more likely to be a mistake than an
>>>> intention.
>>>>>>> If we automatically treat it as None, the user will never realize the
>>>>>>> mistake. Throwing an exception will let the user realize there is a
>>>>>> mistake.
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <ch...@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> hi Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the KIP, and I understand that maintaining compatibility
>> is
>>>>>>>> important.
>>>>>>>> 
>>>>>>>> However, according to the documentation, we don't allow an empty
>> value
>>>>>> for
>>>>>>>> the cleanup.policy. Hence, should we consider throwing an exception
>>>> for
>>>>>> an
>>>>>>>> empty value in version 4.x? This could streamline the code and avoid
>>>> an
>>>>>>>> extra deprecation cycle.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>> 
>>>>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Regarding "The none policy will not delete or compact any
>> segments",
>>>> we
>>>>>>>>> should be more accurate. We won't delete segments based on
>>>>>>>>> log.retention.bytes/log.retention.ms, but we should continue to
>>>> delete
>>>>>>>>> segments based on log.local.retention.bytes/log.retention.ms.
>>>>>> Otherwise,
>>>>>>>>> we
>>>>>>>>> risk running out of local disk space when remote storage is
>> enabled.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply.
>>>>>>>>>> 
>>>>>>>>>> Q2. What about existing empty values for
>>>>>>>>>> group.coordinator.rebalance.protocols and process.roles during
>>>>>> upgrade?
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello Jun,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for review this KIP.
>>>>>>>>>>> 
>>>>>>>>>>> Q1 & Q3:
>>>>>>>>>>> I’ve updated the method name accordingly and revised the
>>>>>>>> cleanup.policy
>>>>>>>>>>> documentation
>>>>>>>>>>> to clarify that the none policy cannot be used with any other
>>>> policy.
>>>>>>>>>>> 
>>>>>>>>>>> Q2:
>>>>>>>>>>> For users currently using an empty cleanup.policy, the approach
>> is
>>>> to
>>>>>>>>>>> apply the none policy
>>>>>>>>>>> during the preProcessParsedConfig step. Additionally, a warning
>>>>>>>> message
>>>>>>>>>>> will be emitted to inform users
>>>>>>>>>>> of the upcoming change.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52 寫道：
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the KIP. A few comments.
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. It's fine to introduce a new value None for cleanup.policy.
>> But
>>>>>>>> now
>>>>>>>>>>> not
>>>>>>>>>>>> all value combinations are valid. For example, None can't be
>> used
>>>>>>>> with
>>>>>>>>>>>> Delete or Compact. It would be useful to document that.
>>>>>>>>>>>> 2. What's the behavior during upgrade when an existing config
>> has
>>>> an
>>>>>>>>>>> empty
>>>>>>>>>>>> list.
>>>>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check does.
>> How
>>>>>>>>> about
>>>>>>>>>>>> sth like inNonEmpty ?
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would like to start a discussion on KIP-1161: cleanup.policy
>>>>>>>>>>> shouldn't
>>>>>>>>>>>>> be empty
>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> This proposal aims to improve the cleanup.policy configuration.
>>>>>>>>>>> Currently,
>>>>>>>>>>>>> this setting should not be left empty.
>>>>>>>>>>>>> Therefore, there are two proposed improvements:
>>>>>>>>>>>>> 1. Update ValidList to validate whether an empty list is
>> allowed.
>>>>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 

Re: [DISCUSS] KIP-1161: cleanup.policy shouldn't be empty

[Jun Rao <ju...@confluent.io.INVALID>]

Hi, Jiunn-Yang,

Thanks for the reply. Sounds good.

Just a minor comment. It would be useful to document that during upgrade,
if cleanup.policy is empty, the broker will fail to start.

Jun

On Thu, May 1, 2025 at 8:40 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> Since ValidList is a public API, we need to maintain backward
> compatibility. Therefore, the isEmptyAllowed flag is necessary.
> We can update the signature of isNonEmpty() to remove the boolean
> parameter.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao <ju...@confluent.io.INVALID> 於 2025年5月1日 凌晨4:17 寫道：
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the reply.
> >
> > Do we really need isEmptyAllowed? It's awkward since the method name is
> > inNonEmpty. Also, it's not clear when it's set to true.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Apr 25, 2025 at 6:26 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hi Jun,
> >>
> >> Thank you for pointing that out.
> >>
> >> I have revised the KIP as follows:
> >>
> >> “In this case, the change does not introduce new invalid configurations
> or
> >> prevent any currently valid setup.
> >> The main behavioral difference is that we now explicitly throw a
> >> ConfigException  during the storage format validation phase
> >> instead of relying on the current behavior.”
> >>
> >> This reflects the correct behavior.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月25日 凌晨1:17 寫道：
> >>>
> >>> Hi, Jiunn-Yang,
> >>>
> >>> "The main behavioral difference introduced by this change is that a
> >>> ConfigException  will now be thrown during the storage format
> validation
> >>> phase, rather than during server startup."
> >>>
> >>> It seems that currently formating the storage already fails if
> >>> group.coordinator.rebalance.protocols and process.roles  are empty. It
> >> just
> >>> gets a different exception.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Thu, Apr 24, 2025 at 5:32 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>
> >>>> Hi Jun, chia,
> >>>>
> >>>> Thanks for your feedback.
> >>>>
> >>>> I add a new section for this change:
> >>>>
> >>>> - Validation for group.coordinator.rebalance.protocols and
> process.roles
> >>>> will be
> >>>> moved from the server startup phase to the storage format phase.
> >>>> - For cleanup.policy, setting an empty list will now be considered
> >> invalid
> >>>> and will
> >>>> result in a ConfigException during storage format phase.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月24日 凌晨4:44 寫道：
> >>>>>
> >>>>> Hi, Jiunn-Yang,
> >>>>>
> >>>>> Thanks for the reply.
> >>>>>
> >>>>> Q2. It's true that group.coordinator.rebalance.protocols and
> >>>> process.roles
> >>>>> configurations can't be empty right now. With this KIP, the user will
> >>>>> probably get a different (and more accurate) exception. It will be
> >> useful
> >>>>> to document that.
> >>>>>
> >>>>> Regarding cleanup.policy, I kind of agree with Chia-Ping. If a user
> >>>> leaves
> >>>>> cleanup.policy empty, it's more likely to be a mistake than an
> >> intention.
> >>>>> If we automatically treat it as None, the user will never realize the
> >>>>> mistake. Throwing an exception will let the user realize there is a
> >>>> mistake.
> >>>>>
> >>>>> Jun
> >>>>>
> >>>>> On Wed, Apr 23, 2025 at 8:26 AM Chia-Ping Tsai <ch...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> hi Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the KIP, and I understand that maintaining compatibility
> is
> >>>>>> important.
> >>>>>>
> >>>>>> However, according to the documentation, we don't allow an empty
> value
> >>>> for
> >>>>>> the cleanup.policy. Hence, should we consider throwing an exception
> >> for
> >>>> an
> >>>>>> empty value in version 4.x? This could streamline the code and avoid
> >> an
> >>>>>> extra deprecation cycle.
> >>>>>>
> >>>>>> Best,
> >>>>>> Chia-Ping
> >>>>>>
> >>>>>> Jun Rao <ju...@confluent.io.invalid> 於 2025年4月23日 週三 上午6:33寫道：
> >>>>>>
> >>>>>>> Hi, Jiunn-Yang,
> >>>>>>>
> >>>>>>> Regarding "The none policy will not delete or compact any
> segments",
> >> we
> >>>>>>> should be more accurate. We won't delete segments based on
> >>>>>>> log.retention.bytes/log.retention.ms, but we should continue to
> >> delete
> >>>>>>> segments based on log.local.retention.bytes/log.retention.ms.
> >>>> Otherwise,
> >>>>>>> we
> >>>>>>> risk running out of local disk space when remote storage is
> enabled.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Jun
> >>>>>>>
> >>>>>>> On Tue, Apr 22, 2025 at 9:45 AM Jun Rao <ju...@confluent.io> wrote:
> >>>>>>>
> >>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> Thanks for the reply.
> >>>>>>>>
> >>>>>>>> Q2. What about existing empty values for
> >>>>>>>> group.coordinator.rebalance.protocols and process.roles during
> >>>> upgrade?
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Tue, Apr 22, 2025 at 7:29 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hello Jun,
> >>>>>>>>>
> >>>>>>>>> Thanks for review this KIP.
> >>>>>>>>>
> >>>>>>>>> Q1 & Q3:
> >>>>>>>>> I’ve updated the method name accordingly and revised the
> >>>>>> cleanup.policy
> >>>>>>>>> documentation
> >>>>>>>>> to clarify that the none policy cannot be used with any other
> >> policy.
> >>>>>>>>>
> >>>>>>>>> Q2:
> >>>>>>>>> For users currently using an empty cleanup.policy, the approach
> is
> >> to
> >>>>>>>>> apply the none policy
> >>>>>>>>> during the preProcessParsedConfig step. Additionally, a warning
> >>>>>> message
> >>>>>>>>> will be emitted to inform users
> >>>>>>>>> of the upcoming change.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>> Jun Rao <ju...@confluent.io.INVALID> 於 2025年4月22日 凌晨4:52 寫道：
> >>>>>>>>>>
> >>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the KIP. A few comments.
> >>>>>>>>>>
> >>>>>>>>>> 1. It's fine to introduce a new value None for cleanup.policy.
> But
> >>>>>> now
> >>>>>>>>> not
> >>>>>>>>>> all value combinations are valid. For example, None can't be
> used
> >>>>>> with
> >>>>>>>>>> Delete or Compact. It would be useful to document that.
> >>>>>>>>>> 2. What's the behavior during upgrade when an existing config
> has
> >> an
> >>>>>>>>> empty
> >>>>>>>>>> list.
> >>>>>>>>>> 3. inWithEmptyCheck: It's not clear what the empty check does.
> How
> >>>>>>> about
> >>>>>>>>>> sth like inNonEmpty ?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Apr 15, 2025 at 8:25 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to start a discussion on KIP-1161: cleanup.policy
> >>>>>>>>> shouldn't
> >>>>>>>>>>> be empty
> >>>>>>>>>>> <https://cwiki.apache.org/confluence/x/HArXF>
> >>>>>>>>>>>
> >>>>>>>>>>> This proposal aims to improve the cleanup.policy configuration.
> >>>>>>>>> Currently,
> >>>>>>>>>>> this setting should not be left empty.
> >>>>>>>>>>> Therefore, there are two proposed improvements:
> >>>>>>>>>>> 1. Update ValidList to validate whether an empty list is
> allowed.
> >>>>>>>>>>> 2. Introduce a new 'none' value for cleanup.policy.
> >>>>>>>>>>>
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>