You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by 黃竣陽 <s7...@gmail.com> on 2026/05/01 14:37:35 UTC

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Hello all,

Thanks for the feedback.

DJ01/DJ02:

MetadataResponse bumps from v13 to v14. The PartitionMetadata struct gains a new 
field PartitionAgeMs (int64, default -1), computed server-side by the broker as 
broker_current_time - partition_creation_time.

Also add the consumer heartbeat flow. when MembershipManager detects a newly assigned 
partition, it explicitly invalidates the metadata for the affected topic and forces a fresh MetadataRequest
before making the offset reset decision, even if the topic ID is already in the cache.

MB0:

The consumer learns the broker's maximum supported MetadataResponse version via the
ApiVersions negotiation at connection time. If the negotiated version is unsupported, the consumer 
knows the broker does not support PartitionAgeMs at all and can throw an UnsupportedVersionException 
immediately, rather than silently falling back to latest and risking data loss without any operator-visible signal.

MB1/MB2/MB3:

I have addressed these changes in the KIP.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> 
> hi David
> 
> I agree with the direction of moving the 'age' resolution from the Heartbeat API to the Metadata API to keep the control plane clean. The main trade-off, as we noted before, is introducing inter-broker clock skew. The Group Coordinator approach provided a single source of truth for time.
> 
> However, realistically, this time skew should be negligible. Given that the max.age threshold will likely be configured in minutes or hours, a typical NTP skew (in milliseconds) between brokers won't impact the fallback decision.
> 
> Best,
> Chia-Ping
> 
>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
>> 
>> Hi all,
>> 
>> Thanks for the KIP!
>> 
>> Sorry, I haven't really followed the previous conversation but I took a
>> quick look at this one.
>> 
>> DJ01: I don't clearly understand the flow with the ConsumerGroupHeartbeat
>> API after reading the KIP. There is a new boolean; the KIP states that
>> partition ages are returned only when this boolean is set. Implicitly, this
>> means that when the consumer receives a new partition, it will issue a new
>> HB request with the boolean set to receive the ages. Is my understanding
>> correct? We should perhaps clarify the flow and also explain how it fits
>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
>> DJ02: It my understanding is correct, I wonder if
>> the ConsumerGroupHeartbeat API is the right place for this given that a new
>> round trip is done anyway. Alternatively, it could simply include the
>> metadata. Generally, we should be rather cautious about not overloading
>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is a
>> control plane API for assigning or revoking partitions. The fact that we
>> don't want to add it to the corresponding Streams API also suggests
>> something is not quite right. What would we do if we want to support
>> Streams in the future?
>> 
>> Best,
>> David
>> 
>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
>>> dev@kafka.apache.org> wrote:
>>> 
>>> Hi Jiunn,
>>> 
>>> Thank you for this great kip. Good to know about the gap.
>>> 
>>> mb-0 - why a new v2 version bump for RequestPartitionAges field. Can a
>>> tagged field (for ex: on response, PartitionAges on TopicPartitions) be
>>> used here and avoid version bump?
>>> 
>>> mb-1 - For the new config, is there a recommended value or a ConfigDef
>>> validator? Probably it should based on the metadata.max.age.ms ? Sizing
>>> instructions can be part of javadocs I guess.
>>> 
>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it be better
>>> to add this new config auto.offset.reset.latest.max.age to the
>>> StreamsConfig block list (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) for a
>>> clear warning, incase users configure it? This is the most familiar
>>> consumer config and users might easily mistakenly configure it. Or may be
>>> it's not worth it to add.
>>> 
>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest" reads as
>>> if the config were being changed per-partition which isn't supported. May
>>> be rephrasing to something like "consumer resolves the initial position to
>>> start offset for that partition" as if earliest was applied to that
>>> partition only and auto.offset.reset config is unchanged.
>>> 
>>> Thanks,
>>> Murali
>>> 
>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>> Hi chia,
>>>> 
>>>> I have updated the KIP to include this change.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
>>>>> 
>>>>> hi Jiunn-Yang
>>>>> 
>>>>> chia_0: Should we expose the partition creation time via the Admin API?
>>>> I assume it would be valuable for users to diagnose and troubleshoot the
>>>> behavior of auto.offset.reset.latest.max.age
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>> Hello everyone,
>>>>>> 
>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data Loss
>>> on
>>>> Partition Expansion for Latest Policy
>>>>>> <
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>> 
>>>>>> 
>>>>>> This proposal aims to introduces auto.offset.reset.latest.max.age, a
>>>> consumer config that lets the
>>>>>> latest reset policy distinguish newly expanded (hot) partitions from
>>>> long-existing (cold) ones. Partitions
>>>>>> younger than the configured threshold automatically fall back to
>>>> earliest, preventing silent data loss
>>>>>> during topic expansion without forcing a full historical reprocess.
>>>>>> 
>>>>>> Best regards,
>>>>>> Jiunn-Yang
>>>> 
>>>> 
>>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Muralidhar Basani via dev <de...@kafka.apache.org>.
Thanks Jiunn for the responses.
I don't have any further questions.

Regards,
Murali

On Thu, May 7, 2026 at 2:07 PM 黃竣陽 <s7...@gmail.com> wrote:

> Hi all,
>
> I’d like to manually bump this thread.
>
> Best Regards,
> Jiunn-Yang
>
> > 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >
> > Hello all,
> >
> > Thanks for the feedback.
> >
> > DJ01/DJ02:
> >
> > MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
> gains a new
> > field PartitionAgeMs (int64, default -1), computed server-side by the
> broker as
> > broker_current_time - partition_creation_time.
> >
> > Also add the consumer heartbeat flow. when MembershipManager detects a
> newly assigned
> > partition, it explicitly invalidates the metadata for the affected topic
> and forces a fresh MetadataRequest
> > before making the offset reset decision, even if the topic ID is already
> in the cache.
> >
> > MB0:
> >
> > The consumer learns the broker's maximum supported MetadataResponse
> version via the
> > ApiVersions negotiation at connection time. If the negotiated version is
> unsupported, the consumer
> > knows the broker does not support PartitionAgeMs at all and can throw an
> UnsupportedVersionException
> > immediately, rather than silently falling back to latest and risking
> data loss without any operator-visible signal.
> >
> > MB1/MB2/MB3:
> >
> > I have addressed these changes in the KIP.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> >> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> >>
> >> hi David
> >>
> >> I agree with the direction of moving the 'age' resolution from the
> Heartbeat API to the Metadata API to keep the control plane clean. The main
> trade-off, as we noted before, is introducing inter-broker clock skew. The
> Group Coordinator approach provided a single source of truth for time.
> >>
> >> However, realistically, this time skew should be negligible. Given that
> the max.age threshold will likely be configured in minutes or hours, a
> typical NTP skew (in milliseconds) between brokers won't impact the
> fallback decision.
> >>
> >> Best,
> >> Chia-Ping
> >>
> >>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
> >>>
> >>> Hi all,
> >>>
> >>> Thanks for the KIP!
> >>>
> >>> Sorry, I haven't really followed the previous conversation but I took a
> >>> quick look at this one.
> >>>
> >>> DJ01: I don't clearly understand the flow with the
> ConsumerGroupHeartbeat
> >>> API after reading the KIP. There is a new boolean; the KIP states that
> >>> partition ages are returned only when this boolean is set. Implicitly,
> this
> >>> means that when the consumer receives a new partition, it will issue a
> new
> >>> HB request with the boolean set to receive the ages. Is my
> understanding
> >>> correct? We should perhaps clarify the flow and also explain how it
> fits
> >>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
> >>> DJ02: It my understanding is correct, I wonder if
> >>> the ConsumerGroupHeartbeat API is the right place for this given that
> a new
> >>> round trip is done anyway. Alternatively, it could simply include the
> >>> metadata. Generally, we should be rather cautious about not overloading
> >>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is a
> >>> control plane API for assigning or revoking partitions. The fact that
> we
> >>> don't want to add it to the corresponding Streams API also suggests
> >>> something is not quite right. What would we do if we want to support
> >>> Streams in the future?
> >>>
> >>> Best,
> >>> David
> >>>
> >>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
> >>>> dev@kafka.apache.org> wrote:
> >>>>
> >>>> Hi Jiunn,
> >>>>
> >>>> Thank you for this great kip. Good to know about the gap.
> >>>>
> >>>> mb-0 - why a new v2 version bump for RequestPartitionAges field. Can a
> >>>> tagged field (for ex: on response, PartitionAges on TopicPartitions)
> be
> >>>> used here and avoid version bump?
> >>>>
> >>>> mb-1 - For the new config, is there a recommended value or a ConfigDef
> >>>> validator? Probably it should based on the metadata.max.age.ms ?
> Sizing
> >>>> instructions can be part of javadocs I guess.
> >>>>
> >>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it be
> better
> >>>> to add this new config auto.offset.reset.latest.max.age to the
> >>>> StreamsConfig block list (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> for a
> >>>> clear warning, incase users configure it? This is the most familiar
> >>>> consumer config and users might easily mistakenly configure it. Or
> may be
> >>>> it's not worth it to add.
> >>>>
> >>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
> reads as
> >>>> if the config were being changed per-partition which isn't supported.
> May
> >>>> be rephrasing to something like "consumer resolves the initial
> position to
> >>>> start offset for that partition" as if earliest was applied to that
> >>>> partition only and auto.offset.reset config is unchanged.
> >>>>
> >>>> Thanks,
> >>>> Murali
> >>>>
> >>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>
> >>>>> Hi chia,
> >>>>>
> >>>>> I have updated the KIP to include this change.
> >>>>>
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
> >>>>>>
> >>>>>> hi Jiunn-Yang
> >>>>>>
> >>>>>> chia_0: Should we expose the partition creation time via the Admin
> API?
> >>>>> I assume it would be valuable for users to diagnose and troubleshoot
> the
> >>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>
> >>>>>> Best,
> >>>>>> Chia-Ping
> >>>>>>
> >>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>> Hello everyone,
> >>>>>>>
> >>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
> Loss
> >>>> on
> >>>>> Partition Expansion for Latest Policy
> >>>>>>> <
> >>>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>
> >>>>>>>
> >>>>>>> This proposal aims to introduces auto.offset.reset.latest.max.age,
> a
> >>>>> consumer config that lets the
> >>>>>>> latest reset policy distinguish newly expanded (hot) partitions
> from
> >>>>> long-existing (cold) ones. Partitions
> >>>>>>> younger than the configured threshold automatically fall back to
> >>>>> earliest, preventing silent data loss
> >>>>>>> during topic expansion without forcing a full historical reprocess.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> Jiunn-Yang
> >>>>>
> >>>>>
> >>>>
> >
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
> I am not sure that we need an out-of-the-box solution. The proposal in the
> KIP is also not out-of-the-box and requires users to opt-in.

It was, until we decided to apply it to all policies. It would have been a great out-of-the-box solution if it were only for the latest policy, where data loss is a real issue. However, applying it to all policies would cause strange behavior in others. That is why we didn't set a default value, meaning it can no longer be an out-of-the-box solution.

Best,
Chia-Ping

On 2026/07/06 21:49:23 Jun Rao via dev wrote:
> Hi, Chia-Ping,
> 
> I am not sure that we need an out-of-the-box solution. The proposal in the
> KIP is also not out-of-the-box and requires users to opt-in.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> 
> > hi Jun
> >
> > Yes, updating the docs requires minimal changes to the codebase, but it is
> > not an out-of-the-box solution. From my perspective, this issue can be
> > addressed by setting the console consumer's policy back to latest.
> >
> > Let's see what Jiunn thinks
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/07/06 15:59:37 Jun Rao via dev wrote:
> > > Hi, Chia-Ping,
> > >
> > > One potential downside of changing the default is that some existing
> > tools
> > > (e.g. console consumer) may depend on this behavior. For example, a user
> > > testing the system may expect to see only new messages and could be
> > > surprised by the change. Another option is to leave the default
> > unchanged,
> > > but document clearly how to prevent losing messages on new partitions
> > > through by_duration. The users can then make the corresponding config
> > > changes if needed.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Thanks for the feedback. I agree with your conclusion. It would be
> > better
> > > > to "make haste" with a new KIP to address this. We can focus the
> > discussion
> > > > on the following points:
> > > >
> > > > 1) Should we change the default reset policy to by_duration for all
> > > > consumers?
> > > > 2) What is the optimal duration to prevent data loss while avoiding
> > large
> > > > backlogs? Note that the classic consumer has a 3-second heartbeat
> > interval,
> > > > whereas the async consumer uses 5 seconds.
> > > > 3) What are the potential side effects of replacing latest with
> > > > by_duration as the default behavior?
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> > > > > Hi, Chia-Ping,
> > > > >
> > > > > I agree that using by_duration with a value of 5s to 10s addresses
> > most
> > > > > common issues with expanded partitions, given the default
> > > > > heartbeat.interval.ms is 3 seconds. We can consider setting that as
> > the
> > > > new
> > > > > default. The current proposal may address some other rarer cases,
> > but it
> > > > > feels a bit like over engineering.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org>
> > > > wrote:
> > > > >
> > > > > > - Short durations fail during extended downtimes, leading to the
> > exact
> > > > > > silent data
> > > > > > loss this KIP aims to prevent.
> > > > > >
> > > > > > Losing the records produced to expanded partitions during a
> > downtime is
> > > > > > actually the expected behavior for the latest policy. This is
> > exactly
> > > > why,
> > > > > > in the current KIP proposal, we prefer by_duration over earliest
> > for
> > > > the
> > > > > > expanded partitions—we want to avoid forcing users to consume a
> > massive
> > > > > > accumulated backlog just because a partition is technically "new".
> > > > > >
> > > > > > To quote my previous comment from May 15:
> > > > > > "...if a consumer is offline for a while and a new partition is
> > created
> > > > > > during that downtime, the user might actually want to skip to
> > latest
> > > > when
> > > > > > resuming, rather than reading from earliest just because the
> > partition
> > > > is
> > > > > > technically 'new' to the group."
> > > > > >
> > > > > > On 2026/06/30 12:12:47 黃竣陽 wrote:
> > > > > > > Hello all,
> > > > > > >
> > > > > > > The metadata blindness and consumer downtime windows are
> > inherently
> > > > > > > unpredictable, making a fixed by_duration approach unviable. No
> > > > static
> > > > > > > duration can accommodate both extremes:
> > > > > > >
> > > > > > > - Short durations fail during extended downtimes, leading to the
> > > > exact
> > > > > > silent data
> > > > > > > loss this KIP aims to prevent.
> > > > > > > - Long durations (calibrated for worst-case scenarios) cause
> > routine
> > > > > > restarts to
> > > > > > > trigger unnecessary mass replays, undermining the latest offset
> > reset
> > > > > > policy.
> > > > > > >
> > > > > > > This is why the proposed solution uses creation time. It
> > provides a
> > > > > > discrete,
> > > > > > > deterministic signal—verifying whether a partition is actually
> > newer
> > > > > > than the group,
> > > > > > > rather than relying on an arbitrary time window that has no
> > > > correlation
> > > > > > with the consumer’s
> > > > > > > actual downtime.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Jiunn-Yang
> > > > > > >
> > > > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'd like to revisit Jun previous comment about using
> > > > by_duration=5s. I
> > > > > > realize now that having a small backlog is actually an acceptable
> > > > > > trade-off, especially since the latest policy inherently brings a
> > tiny
> > > > > > backlog to starting consumers anyway due to the time gap between
> > > > fetching
> > > > > > the offset and actually consuming.
> > > > > > > >
> > > > > > > > If we bring the simple by_duration=5s solution back to the
> > KIP, I'd
> > > > > > like to discuss a further step: Should we just change the default
> > > > policy
> > > > > > from latest to by_duration=5s specifically for the new consumer and
> > > > share
> > > > > > consumer?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Chia-Ping
> > > > > > > >
> > > > > > > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > > > > > > >> hi Jun
> > > > > > > >>
> > > > > > > >> The most important part of this story is how users should
> > expect
> > > > the
> > > > > > data
> > > > > > > >> they can see when using the latest or by_duration policy with
> > > > expanded
> > > > > > > >> partitions.
> > > > > > > >>
> > > > > > > >> Yes, the by_duration policy can minimize data loss, but it is
> > > > > > > >> non-deterministic, which means users will either read too many
> > > > > > historical
> > > > > > > >> records from existing partitions or lose some records from
> > > > expanded
> > > > > > > >> partitions.
> > > > > > > >>
> > > > > > > >> Also, I agree that auto.offset.reset.max.age.ms is a bit
> > hard to
> > > > > > > >> understand, and that is why I preferred having a whole new
> > policy
> > > > > > based
> > > > > > > >> entirely on group creation time (KIP-1282)
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Chia-Ping
> > > > > > > >>
> > > > > > > >> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > 上午1:08寫道:
> > > > > > > >>
> > > > > > > >>> Hi, Chia-Ping and Jiunn-Yang,
> > > > > > > >>>
> > > > > > > >>> Thanks for the reply. I am still trying to understand the
> > value
> > > > of
> > > > > > the new
> > > > > > > >>> configs with the KIP.
> > > > > > > >>>
> > > > > > > >>> The motivation of the KIP is that a user doesn't want to
> > miss the
> > > > > > data if
> > > > > > > >>> the backlog is small. The backlog of the existing partition
> > is
> > > > easy
> > > > > > to
> > > > > > > >>> understand because it relates to retention time. The backlog
> > for
> > > > the
> > > > > > new
> > > > > > > >>> partition is a bit subtle to understand since it depends on
> > the
> > > > > > metadata
> > > > > > > >>> refresh delay. To set auto.offset.reset.max.age.ms, the user
> > > > needs
> > > > > > to
> > > > > > > >>> understand the metadata refresh delay on the consumer side
> > and
> > > > use
> > > > > > it to
> > > > > > > >>> set the config.
> > > > > > > >>>
> > > > > > > >>> Now, let's consider the alternative: setting the same value
> > for
> > > > the
> > > > > > > >>> existing by_duration policy. The KIP lists three issues with
> > this
> > > > > > approach.
> > > > > > > >>> 1. It computes the seek target client-side as now() -
> > duration,
> > > > which
> > > > > > > >>> introduces clock skew across consumers and forces operators
> > to
> > > > choose
> > > > > > > >>> overly large durations, causing unnecessary reprocessing.
> > > > > > > >>> 2. The target timestamp is recomputed on each retry, so
> > failed
> > > > > > > >>> ListOffsetsRequest retries can shift the target forward and
> > > > > > potentially
> > > > > > > >>> miss records produced between attempts.
> > > > > > > >>> 3. It applies uniformly to all partitions without committed
> > > > offsets,
> > > > > > and
> > > > > > > >>> cannot distinguish newly expanded partitions from
> > long-existing
> > > > > > partitions
> > > > > > > >>> newly assigned to the group, leading to unnecessary replay.
> > > > > > > >>>
> > > > > > > >>> Issues 1 and 2 are uncommon and can be mitigated by adding a
> > bit
> > > > > > buffer to
> > > > > > > >>> the metadata refresh delay. We could also consider improving
> > the
> > > > > > > >>> implementation. For issue 3, the metadata refresh delay is
> > > > typically
> > > > > > low
> > > > > > > >>> (in the order of minutes with the classic consumer and tens
> > of
> > > > > > seconds with
> > > > > > > >>> the new consumer). If a user is ok with reading that much
> > backlog
> > > > > > for new
> > > > > > > >>> partitions, it seems they will be ok doing the same for
> > existing
> > > > > > > >>> partitions.
> > > > > > > >>>
> > > > > > > >>> So, instead of introducing a new config, could we just reuse
> > the
> > > > > > existing
> > > > > > > >>> config with better documentation and/or implementation?
> > > > > > > >>>
> > > > > > > >>> Jun
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > > > > > >>>
> > > > > > > >>>> Hello Jun,
> > > > > > > >>>>
> > > > > > > >>>> You're right that group creation time is the more intuitive
> > > > answer
> > > > > > at
> > > > > > > >>>> first glance,
> > > > > > > >>>> the KIP's own motivation talks about partitions that
> > "predate
> > > > the
> > > > > > group"
> > > > > > > >>>> vs partitions
> > > > > > > >>>> "created during group runtime," which directly points to a
> > > > > > > >>> group-lifecycle
> > > > > > > >>>> classifier.
> > > > > > > >>>> I'd like to walk through why we landed on partition age,
> > and the
> > > > > > > >>>> trade-offs we considered.
> > > > > > > >>>>
> > > > > > > >>>> We evaluated three candidate signals:
> > > > > > > >>>>
> > > > > > > >>>> 1. `by_duration:5secs`
> > > > > > > >>>>
> > > > > > > >>>> This covers the metadata blindness window, but has issues
> > the
> > > > KIP
> > > > > > > >>>> currently documents
> > > > > > > >>>> under "Why not use `by_duration`?":
> > > > > > > >>>>
> > > > > > > >>>> - Client-side `now() - duration` introduces clock skew
> > across
> > > > > > consumers.
> > > > > > > >>>> - `ListOffsets` retries shift the target forward,
> > potentially
> > > > > > missing
> > > > > > > >>>> records produced between
> > > > > > > >>>> attempts.
> > > > > > > >>>> - It applies uniformly to all partitions without committed
> > > > offsets,
> > > > > > > >>>> including pre-existing partitions
> > > > > > > >>>> newly assigned to the group, causing unnecessary replay.
> > > > > > > >>>>
> > > > > > > >>>> 2. Group creation time as classifier
> > > > > > > >>>>
> > > > > > > >>>> This works cleanly when the consumer is actively running.
> > Our
> > > > > > concern
> > > > > > > >>>> is the idle / late-rejoin case:
> > > > > > > >>>>
> > > > > > > >>>> T=0:         Group created.
> > > > > > > >>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > > > > > >>>> T=50:        Partition added during the idle window.
> > > > > > > >>>> T=100:       Consumer resumes.
> > > > > > > >>>>
> > > > > > > >>>> Under group creation time, the new partition is classified
> > as
> > > > new
> > > > > > > >>>> (`50 > 0`) and reset to `earliest`, replaying everything
> > from
> > > > T=50.
> > > > > > > >>>> But during `[T=1, T=100]`, base partitions also accumulated
> > data
> > > > > > that
> > > > > > > >>>> the consumer accepts as lost — that is precisely the
> > contract of
> > > > > > > >>>> `auto.offset.reset=latest`. There is no principled reason to
> > > > treat
> > > > > > > >>>> the new partition differently; both contain backlog
> > accumulated
> > > > > > during
> > > > > > > >>>> the same idle window.
> > > > > > > >>>>
> > > > > > > >>>> This aligns with the "backlog is backlog” principle you
> > raised
> > > > in
> > > > > > > >>>> the KIP-1282 thread: a `latest` user has tolerated some
> > backlog
> > > > on
> > > > > > > >>>> every other partition during the same idle period; forcing
> > > > 0-backlog
> > > > > > > >>>> tolerance only on new partitions would be inconsistent with
> > that
> > > > > > > >>>> tolerance.
> > > > > > > >>>>
> > > > > > > >>>> 3. Partition age vs threshold
> > > > > > > >>>>
> > > > > > > >>>> Partition age corresponds to the actual silent data loss
> > window,
> > > > > > > >>>> the gap between partition creation and the consumer’s
> > metadata
> > > > > > > >>>> refresh. Within this window, data loss is genuinely silent:
> > the
> > > > > > > >>>> consumer had no opportunity to know about the partition.
> > Outside
> > > > > > this
> > > > > > > >>>> window, missing data reflects either:
> > > > > > > >>>>
> > > > > > > >>>> - (a) the user’s tolerated cost of running with idle
> > consumers,
> > > > or
> > > > > > > >>>> - (b) an operational issue to surface via monitoring, not
> > via
> > > > reset
> > > > > > > >>> policy.
> > > > > > > >>>>
> > > > > > > >>>> We did not choose partition age because it is more elegant
> > than
> > > > > > group
> > > > > > > >>>> creation time — we chose it because its failure mode
> > (requires a
> > > > > > > >>>> threshold) is
> > > > > > > >>>> less invasive than the failure mode of group creation time
> > > > > > (overrides
> > > > > > > >>>> user-stated
> > > > > > > >>>> `latest` intent during idle periods).
> > > > > > > >>>>
> > > > > > > >>>> Best Regards,
> > > > > > > >>>> Jiunn-Yang
> > > > > > > >>>>
> > > > > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> > 寫道:
> > > > > > > >>>>>
> > > > > > > >>>>> Hi Jun,
> > > > > > > >>>>>
> > > > > > > >>>>> Relying on both creation times will create an inconsistent
> > > > > > scenario. A
> > > > > > > >>>>> consumer that lost all offsets due to a long sleep will
> > seek
> > > > to the
> > > > > > > >>>>> beginning for the partitions created later than the group.
> > > > > > > >>>>>
> > > > > > > >>>>> That is why we initially proposed KIP-1282 to fix the
> > > > inconsistency
> > > > > > > >>>> using a
> > > > > > > >>>>> whole new policy. Since KIP-1282 couldn't reach a
> > consensus,
> > > > > > KIP-1327
> > > > > > > >>>> goes
> > > > > > > >>>>> back to using flexible configurations to prevent users from
> > > > falling
> > > > > > > >>> into
> > > > > > > >>>>> that pitfall.
> > > > > > > >>>>>
> > > > > > > >>>>> Best, Chia-Ping
> > > > > > > >>>>>
> > > > > > > >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > > > 上午6:49寫道:
> > > > > > > >>>>>
> > > > > > > >>>>>> Hi, Jiunn-Yang,
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks for the reply and sorry for the late reply.
> > > > > > > >>>>>>
> > > > > > > >>>>>> JR1. The design of auto.offset.reset.max.age.ms still
> > feels
> > > > > > weird to
> > > > > > > >>>> me.
> > > > > > > >>>>>> It
> > > > > > > >>>>>> categorizes partitions as new or existing based on the
> > > > partition
> > > > > > > >>>> creation
> > > > > > > >>>>>> time. Intuitively, the categorization should be based on
> > the
> > > > group
> > > > > > > >>>> creation
> > > > > > > >>>>>> time: all partitions existing when the group is created
> > are
> > > > > > existing
> > > > > > > >>> and
> > > > > > > >>>>>> all partitions created after the group creation are new
> > > > > > partitions.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Jun
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > > > wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Hi all,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Manually bumping this thread. If there is no further
> > > > > > > >>>>>>> discussion, I will close the vote.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Best Regards,
> > > > > > > >>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Hello Jian,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thanks for your feedback,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Agreed, partition expansion is a common operational
> > task,
> > > > not an
> > > > > > > >>> edge
> > > > > > > >>>>>>>> case. I've updated the Motivation section accordingly.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Best Regards,
> > > > > > > >>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Hi Jiunn-Yang:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Thanks for the KIP. I think it would be useful to
> > clarify
> > > > that
> > > > > > this
> > > > > > > >>>>>> is a
> > > > > > > >>>>>>>>> common scenario rather than an edge case, which further
> > > > > > > >>> demonstrates
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>> need for this optimization. For example:
> > > > > > > >>>>>>>>> A partition expansion is a common operational task in
> > > > Kafka: To
> > > > > > > >>>>>> balance
> > > > > > > >>>>>>>>> resource utilization and cost, topics are typically
> > created
> > > > > > with a
> > > > > > > >>>>>>> moderate
> > > > > > > >>>>>>>>> default partition count. However, as traffic grows over
> > > > time,
> > > > > > it is
> > > > > > > >>>>>>> often
> > > > > > > >>>>>>>>> necessary to increase the number of partitions to
> > > > accommodate
> > > > > > the
> > > > > > > >>>>>> higher
> > > > > > > >>>>>>>>> workload.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Regards
> > > > > > > >>>>>>>>> Jian
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> Hello chia,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> > 晚上8:29
> > > > 寫道:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Hi Jiunn-Yang,
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> > when
> > > > > > > >>> describing
> > > > > > > >>>>>>>>>>> partitions in the KIP? I understand you are using
> > them to
> > > > > > > >>> describe
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>>>> "freshness" or the users' need for the records, but
> > > > applying
> > > > > > > >>> these
> > > > > > > >>>>>>> terms
> > > > > > > >>>>>>>>>> to
> > > > > > > >>>>>>>>>>> the partition itself feels a bit unnatural.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> After all, in this scenario, users don't really care
> > > > whether
> > > > > > a
> > > > > > > >>>>>>> partition
> > > > > > > >>>>>>>>>> is
> > > > > > > >>>>>>>>>>> newly expanded or not. Their only expectation is that
> > > > they
> > > > > > won't
> > > > > > > >>>>>>> silently
> > > > > > > >>>>>>>>>>> lose any live records produced to the topic during
> > their
> > > > > > active
> > > > > > > >>>>>>>>>> consumption.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Best, Chia-Ping
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Hello Jun,
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > > > motivation
> > > > > > > >>>> section.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > 2026年5月30日
> > > > > > 凌晨1:12 寫道:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > > > > > motivation for
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>>> KIP.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> The KIP says "The core insight is that not all
> > > > partitions
> > > > > > > >>> without
> > > > > > > >>>>>> a
> > > > > > > >>>>>>>>>>>>> committed offset are the same. A newly expanded
> > > > partition
> > > > > > (hot)
> > > > > > > >>>> is
> > > > > > > >>>>>>>>>>>>> fundamentally different from a partition the
> > consumer
> > > > has
> > > > > > never
> > > > > > > >>>>>> seen
> > > > > > > >>>>>>>>>>>>> because it predates the group (cold)." Why is the
> > hot
> > > > > > partition
> > > > > > > >>>>>>>>>>>>> fundamentally different from the cold?
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> > also
> > > > > > > >>>> insufficient
> > > > > > > >>>>>>>>>>>> because:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> - The calculated seek time (now() - duration)
> > varies
> > > > across
> > > > > > > >>> nodes
> > > > > > > >>>>>>> due
> > > > > > > >>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> > large
> > > > > > > >>> duration,
> > > > > > > >>>>>>>>>>>> causing
> > > > > > > >>>>>>>>>>>>> unnecessary reprocessing.
> > > > > > > >>>>>>>>>>>>> - On network errors, the client recalculates the
> > seek
> > > > time
> > > > > > on
> > > > > > > >>>>>> retry,
> > > > > > > >>>>>>>>>>>>> shifting the target timestamp forward and risking
> > data
> > > > > > loss."
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> However, both of these situations are rare. If
> > these
> > > > issues
> > > > > > > >>>>>> persist,
> > > > > > > >>>>>>>>>> more
> > > > > > > >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > situations
> > > > > > don't
> > > > > > > >>>>>> need a
> > > > > > > >>>>>>>>>>>> common
> > > > > > > >>>>>>>>>>>>> solution. If users care about those rare
> > situations,
> > > > they
> > > > > > can
> > > > > > > >>>>>>> implement
> > > > > > > >>>>>>>>>>>>> customized logic using
> > > > > > > >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > > s7133700@gmail.com>
> > > > > > > >>> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Hello chia,
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Thanks for the feedback,
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > > should
> > > > > > always
> > > > > > > >>>> be
> > > > > > > >>>>>>>>>>>> greater
> > > > > > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > > >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > >   any
> > > > > > > >>>>>>> field
> > > > > > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> The earliest point at which we can detect the
> > version
> > > > > > mismatch
> > > > > > > >>>> is
> > > > > > > >>>>>>>>>> during
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>> first metadata fetch after assignment, which
> > occurs
> > > > inside
> > > > > > > >>>>>> poll().
> > > > > > > >>>>>>>>>>>>>> Therefore, the
> > > > > > > >>>>>>>>>>>>>> user would encounter an
> > UnsupportedVersionException
> > > > from
> > > > > > > >>> poll().
> > > > > > > >>>>>>> I’ll
> > > > > > > >>>>>>>>>>>>>> clarify this in the KIP.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2026年5月17日
> > > > > > 下午4:50 寫道:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> hi Jiunn
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> > this
> > > > > > > >>> partition
> > > > > > > >>>>>> in
> > > > > > > >>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> > as
> > > > > > > >>>>>>>>>> broker_current_time
> > > > > > > >>>>>>>>>>>> -
> > > > > > > >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > > > does not
> > > > > > > >>>>>> support
> > > > > > > >>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > > should
> > > > > > always
> > > > > > > >>>> be
> > > > > > > >>>>>>>>>>>> greater
> > > > > > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > >   any
> > > > > > > >>>>>>> field
> > > > > > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> > when
> > > > > > calling
> > > > > > > >>>>>>>>>> `poll()`?
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > > > > >>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> > based on
> > > > the
> > > > > > > >>>>>>> discussion
> > > > > > > >>>>>>>>>>>>>>>> feedback.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > reset
> > > > > > > >>> behavior
> > > > > > > >>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to
> > all
> > > > > > > >>>>>>> auto.offset.reset
> > > > > > > >>>>>>>>>>>>>> values
> > > > > > > >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > > > > > >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> > resets to
> > > > > > > >>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > > > > > > >>>>>>>>>>>>>>>> config setting
> > > > > > > >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > > > > > >>> auto.offset.reset
> > > > > > > >>>>>>>>>> policy
> > > > > > > >>>>>>>>>>>>>> continues
> > > > > > > >>>>>>>>>>>>>>>> to apply unchanged.
> > > > > > > >>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > represented
> > > > by a
> > > > > > > >>>> separate
> > > > > > > >>>>>>>>>>>>>> internal config
> > > > > > > >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently
> > fixed
> > > > to
> > > > > > > >>>>>> earliest).
> > > > > > > >>>>>>>>>> This
> > > > > > > >>>>>>>>>>>>>> decoupled design makes
> > > > > > > >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > > > public
> > > > > > > >>>>>>> user-facing
> > > > > > > >>>>>>>>>>>>>> configuration in a future KIP.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2026年5月16日
> > > > > > 清晨7:46
> > > > > > > >>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> hi Jun
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> > is
> > > > listed
> > > > > > > >>>> below:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
> > > > default
> > > > > > > >>> value
> > > > > > > >>>>>> of
> > > > > > > >>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > > > by_duration and
> > > > > > > >>>>>> latest,
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>> does not change the logic of
> > > > auto.offset.reset=earliest.
> > > > > > > >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> > > > > > internal
> > > > > > > >>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > > .partitions=earliest
> > > > > > > >>>> already
> > > > > > > >>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> > cases
> > > > of
> > > > > > other
> > > > > > > >>>>>>> values
> > > > > > > >>>>>>>>>>>> in a
> > > > > > > >>>>>>>>>>>>>> separate KIP.
> > > > > > > >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > .partitions
> > > > and
> > > > > > > >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be
> > applied
> > > > to
> > > > > > all
> > > > > > > >>> for
> > > > > > > >>>>>>>>>>>>>> consistency.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> WDYT?
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > > > > > >>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> > "When a
> > > > > > Kafka
> > > > > > > >>>>>> topic
> > > > > > > >>>>>>> is
> > > > > > > >>>>>>>>>>>>>> expanded
> > > > > > > >>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> > latest
> > > > auto
> > > > > > > >>> offset
> > > > > > > >>>>>>> reset
> > > > > > > >>>>>>>>>>>>>> policy
> > > > > > > >>>>>>>>>>>>>>>>>> will silently miss all records produced to
> > those
> > > > > > > >>> partitions
> > > > > > > >>>>>>> before
> > > > > > > >>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > > > record
> > > > > > loss
> > > > > > > >>>>>> issue
> > > > > > > >>>>>>>>>> could
> > > > > > > >>>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>>>>> happen, right?
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > > > .partitions
> > > > > > will
> > > > > > > >>>> take
> > > > > > > >>>>>>> the
> > > > > > > >>>>>>>>>>>> same
> > > > > > > >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> > set
> > > > it
> > > > > > > >>>>>>> by_duration if
> > > > > > > >>>>>>>>>>>>>> needed.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> > Tsai <
> > > > > > > >>>>>>>>>> chia7712@apache.org
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > > > something
> > > > > > > >>>>>>> important
> > > > > > > >>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>> your
> > > > > > > >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> > to
> > > > > > clarify
> > > > > > > >>> with
> > > > > > > >>>>>> a
> > > > > > > >>>>>>> few
> > > > > > > >>>>>>>>>>>>>> questions:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> > this
> > > > > > logic to
> > > > > > > >>>>>>> other
> > > > > > > >>>>>>>>>>>> reset
> > > > > > > >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > earliest
> > > > or
> > > > > > > >>>>>> by_duration
> > > > > > > >>>>>>>>>>>> don't
> > > > > > > >>>>>>>>>>>>>> seem
> > > > > > > >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> > issue
> > > > when a
> > > > > > > >>>>>>> partition
> > > > > > > >>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>> expanded.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > > > configure for
> > > > > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they
> > set
> > > > it to
> > > > > > > >>>>>> earliest
> > > > > > > >>>>>>> or
> > > > > > > >>>>>>>>>>>>>> latest,
> > > > > > > >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases.
> > For
> > > > > > example,
> > > > > > > >>>>>> if a
> > > > > > > >>>>>>>>>>>>>> consumer is
> > > > > > > >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > > > created
> > > > > > during
> > > > > > > >>>>>> that
> > > > > > > >>>>>>>>>>>>>> downtime,
> > > > > > > >>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> > latest
> > > > when
> > > > > > > >>>>>> resuming,
> > > > > > > >>>>>>>>>>>> rather
> > > > > > > >>>>>>>>>>>>>> than
> > > > > > > >>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > partition
> > > > is
> > > > > > > >>>>>>> technically
> > > > > > > >>>>>>>>>>>>>> "new" to
> > > > > > > >>>>>>>>>>>>>>>>>>> the group.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing
> > a
> > > > > > max.age
> > > > > > > >>>>>>> threshold.
> > > > > > > >>>>>>>>>>>> It
> > > > > > > >>>>>>>>>>>>>> gives
> > > > > > > >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > > > genuinely
> > > > > > > >>>> "hot/new"
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>>>>>>> what is
> > > > > > > >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> > bit
> > > > > > weird. It
> > > > > > > >>>>>> only
> > > > > > > >>>>>>>>>>>>>> applies when
> > > > > > > >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> > seems
> > > > that
> > > > > > the
> > > > > > > >>>>>>>>>> motivation
> > > > > > > >>>>>>>>>>>>>>>>>>> equally
> > > > > > > >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> > other
> > > > > > values
> > > > > > > >>> like
> > > > > > > >>>>>>>>>>>>>> by_duration.
> > > > > > > >>>>>>>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> > > > way to
> > > > > > > >>>> control
> > > > > > > >>>>>>>>>> newly
> > > > > > > >>>>>>>>>>>>>> created
> > > > > > > >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > group
> > > > > > starts.
> > > > > > > >>>>>>> Have we
> > > > > > > >>>>>>>>>>>>>>>>>>> considered
> > > > > > > >>>>>>>>>>>>>>>>>>>> adding a new config like
> > auto.offset.reset.new
> > > > > > > >>>> .partitions?
> > > > > > > >>>>>>> If
> > > > > > > >>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > > > defaults
> > > > > > to
> > > > > > > >>> the
> > > > > > > >>>>>>>>>> policy
> > > > > > > >>>>>>>>>>>>>> used
> > > > > > > >>>>>>>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > > > > > explicitly to
> > > > > > > >>>>>>>>>> customize
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > > > > s7133700@gmail.com>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> > 晚上10:37
> > > > 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> > The
> > > > > > > >>>>>>> PartitionMetadata
> > > > > > > >>>>>>>>>>>>>> struct
> > > > > > > >>>>>>>>>>>>>>>>>>>>> gains a new
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > > > computed
> > > > > > > >>>>>>> server-side
> > > > > > > >>>>>>>>>>>> by
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> broker as
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > partition_creation_time.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > > > > > >>>>>>> MembershipManager
> > > > > > > >>>>>>>>>>>>>> detects
> > > > > > > >>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > > > metadata
> > > > > > for
> > > > > > > >>>> the
> > > > > > > >>>>>>>>>>>> affected
> > > > > > > >>>>>>>>>>>>>>>>>>> topic
> > > > > > > >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> > even
> > > > if
> > > > > > the
> > > > > > > >>>>>> topic
> > > > > > > >>>>>>> ID
> > > > > > > >>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>> already
> > > > > > > >>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > > > supported
> > > > > > > >>>>>>>>>>>>>> MetadataResponse
> > > > > > > >>>>>>>>>>>>>>>>>>>>> version via the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> > time.
> > > > If the
> > > > > > > >>>>>>> negotiated
> > > > > > > >>>>>>>>>>>>>>>>>>> version is
> > > > > > > >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > > > PartitionAgeMs
> > > > > > at
> > > > > > > >>> all
> > > > > > > >>>>>> and
> > > > > > > >>>>>>>>>> can
> > > > > > > >>>>>>>>>>>>>>>>>>> throw an
> > > > > > > >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > > > back to
> > > > > > > >>> latest
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>>>>>>>>> risking
> > > > > > > >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> > signal.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > > 2026年4月29日
> > > > > > > >>>> 下午4:04
> > > > > > > >>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > > > 'age'
> > > > > > > >>>>>> resolution
> > > > > > > >>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> > the
> > > > > > control
> > > > > > > >>>>>> plane
> > > > > > > >>>>>>>>>>>> clean.
> > > > > > > >>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>>>> main
> > > > > > > >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > introducing
> > > > > > > >>>> inter-broker
> > > > > > > >>>>>>>>>> clock
> > > > > > > >>>>>>>>>>>>>> skew.
> > > > > > > >>>>>>>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> > single
> > > > > > source of
> > > > > > > >>>>>> truth
> > > > > > > >>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>> time.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > > > should be
> > > > > > > >>>>>>> negligible.
> > > > > > > >>>>>>>>>>>>>> Given
> > > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > > > configured in
> > > > > > > >>>> minutes
> > > > > > > >>>>>>> or
> > > > > > > >>>>>>>>>>>>>> hours, a
> > > > > > > >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > > > brokers
> > > > > > > >>> won't
> > > > > > > >>>>>>> impact
> > > > > > > >>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > dev@kafka.apache.org>
> > > > 於
> > > > > > > >>>>>> 2026年4月29日
> > > > > > > >>>>>>>>>>>> 下午3:29
> > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > > > previous
> > > > > > > >>>>>>> conversation
> > > > > > > >>>>>>>>>>>> but I
> > > > > > > >>>>>>>>>>>>>>>>>>> took a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> > flow
> > > > with
> > > > > > the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> > new
> > > > > > boolean;
> > > > > > > >>> the
> > > > > > > >>>>>>> KIP
> > > > > > > >>>>>>>>>>>>>> states
> > > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> > this
> > > > > > boolean
> > > > > > > >>> is
> > > > > > > >>>>>>> set.
> > > > > > > >>>>>>>>>>>>>>>>>>> Implicitly,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a
> > new
> > > > > > > >>> partition,
> > > > > > > >>>>>> it
> > > > > > > >>>>>>>>>> will
> > > > > > > >>>>>>>>>>>>>>>>>>> issue a
> > > > > > > >>>>>>>>>>>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> > receive
> > > > the
> > > > > > ages.
> > > > > > > >>>> Is
> > > > > > > >>>>>>> my
> > > > > > > >>>>>>>>>>>>>>>>>>>>> understanding
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> > flow
> > > > and
> > > > > > also
> > > > > > > >>>>>>> explain
> > > > > > > >>>>>>>>>>>>>> how it
> > > > > > > >>>>>>>>>>>>>>>>>>>>> fits
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > offsets,
> > > > fetch
> > > > > > > >>>>>> offsets,
> > > > > > > >>>>>>>>>>>> etc.).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > > > wonder
> > > > > > if
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> > right
> > > > > > place
> > > > > > > >>> for
> > > > > > > >>>>>>> this
> > > > > > > >>>>>>>>>>>> given
> > > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>> a new
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > Alternatively, it
> > > > > > could
> > > > > > > >>>>>> simply
> > > > > > > >>>>>>>>>>>>>> include
> > > > > > > >>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > > > > > cautious
> > > > > > > >>>> about
> > > > > > > >>>>>>> not
> > > > > > > >>>>>>>>>>>>>>>>>>> overloading
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > > > unrelated
> > > > > > > >>>> concepts.
> > > > > > > >>>>>>> The
> > > > > > > >>>>>>>>>>>> API
> > > > > > > >>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> > revoking
> > > > > > > >>>> partitions.
> > > > > > > >>>>>>> The
> > > > > > > >>>>>>>>>>>> fact
> > > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>> we
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > corresponding
> > > > > > Streams
> > > > > > > >>> API
> > > > > > > >>>>>>> also
> > > > > > > >>>>>>>>>>>>>> suggests
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> > would we
> > > > do
> > > > > > if we
> > > > > > > >>>>>>> want to
> > > > > > > >>>>>>>>>>>>>>>>>>> support
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> David
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > Muralidhar
> > > > > > Basani
> > > > > > > >>>> via
> > > > > > > >>>>>>> dev
> > > > > > > >>>>>>>>>> <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> > know
> > > > > > about
> > > > > > > >>> the
> > > > > > > >>>>>>> gap.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > > > > > >>>>>>> RequestPartitionAges
> > > > > > > >>>>>>>>>>>>>> field.
> > > > > > > >>>>>>>>>>>>>>>>>>> Can a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > > > > > PartitionAges on
> > > > > > > >>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > > > > >>>>>>>>>>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > > > > > recommended
> > > > > > > >>>>>> value
> > > > > > > >>>>>>> or
> > > > > > > >>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>> ConfigDef
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on
> > the
> > > > > > > >>>>>>>>>>>> metadata.max.age.ms
> > > > > > > >>>>>>>>>>>>>> ?
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Sizing
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > > > guess.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes
> > to
> > > > Kafka
> > > > > > > >>>>>> Streams,
> > > > > > > >>>>>>>>>>>> would
> > > > > > > >>>>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>>>>>> better
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > > > > >>>>>> auto.offset.reset.latest.max.age
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > > > > >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > > > > >>>>>>>>>>>>>>>>>>>>> for a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> > it?
> > > > This
> > > > > > is
> > > > > > > >>> the
> > > > > > > >>>>>>> most
> > > > > > > >>>>>>>>>>>>>>>>>>> familiar
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > > > > > mistakenly
> > > > > > > >>>>>>> configure
> > > > > > > >>>>>>>>>>>>>> it. Or
> > > > > > > >>>>>>>>>>>>>>>>>>>>> may be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > consumer
> > > > falls
> > > > > > > >>> back
> > > > > > > >>>>>> to
> > > > > > > >>>>>>>>>>>>>> earliest"
> > > > > > > >>>>>>>>>>>>>>>>>>>>> reads as
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > > > per-partition
> > > > > > > >>> which
> > > > > > > >>>>>>> isn't
> > > > > > > >>>>>>>>>>>>>>>>>>> supported.
> > > > > > > >>>>>>>>>>>>>>>>>>>>> May
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > "consumer
> > > > > > resolves
> > > > > > > >>>> the
> > > > > > > >>>>>>>>>>>> initial
> > > > > > > >>>>>>>>>>>>>>>>>>>>> position to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > > > > > earliest was
> > > > > > > >>>>>>> applied
> > > > > > > >>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > > > config is
> > > > > > > >>>>>>> unchanged.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > > > > > >>>>>>> s7133700@gmail.com>
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> > > > change.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > 於
> > > > > > > >>> 2026年4月28日
> > > > > > > >>>>>>> 晚上8:03
> > > > > > > >>>>>>>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > partition
> > > > > > creation
> > > > > > > >>>> time
> > > > > > > >>>>>>> via
> > > > > > > >>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>> Admin
> > > > > > > >>>>>>>>>>>>>>>>>>>>> API?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> > users to
> > > > > > > >>> diagnose
> > > > > > > >>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>>>> troubleshoot
> > > > > > > >>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > > auto.offset.reset.latest.max.age
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> > on
> > > > > > KIP-1327
> > > > > > > >>>>>>> Prevent
> > > > > > > >>>>>>>>>> Hot
> > > > > > > >>>>>>>>>>>>>> Data
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Loss
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > >
> > > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > > > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> > newly
> > > > > > expanded
> > > > > > > >>>>>> (hot)
> > > > > > > >>>>>>>>>>>>>> partitions
> > > > > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > threshold
> > > > > > > >>>> automatically
> > > > > > > >>>>>>> fall
> > > > > > > >>>>>>>>>>>>>> back
> > > > > > > >>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> > forcing a
> > > > > > full
> > > > > > > >>>>>>> historical
> > > > > > > >>>>>>>>>>>>>>>>>>> reprocess.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Andrew

Yes, we are on the same page :)

Best,
Chia-Ping

> Andrew Schofield <sc...@apache.org> 於 2026年7月8日 晚上10:47 寫道:
> 
> Hi,
> I'd like to confirm what is happening with this KIP for AK 4.4. I think the answer is that we've decided to improve the docs (I see a PR) and leave it at that for now. Do I understand correctly?
> 
> Thanks,
> Andrew
> 
>> On 2026/07/08 01:32:41 Chia-Ping Tsai wrote:
>> hi all,
>> 
>> While I agree we shouldn't surprise the majority of users, relying solely on documentation means some users will still fall into this trap. Realizing this limitation only when data is silently lost during a partition expansion is a truly frustrating experience.
>> 
>> Since we've reached the KIP freeze for 4.4.0, I recommend we focus on updating the docs first.
>> 
>> We can put this KIP on hold for now and revisit it after Community Over Code. IIRC, Jiunn-Yang is going to share this specific issue at the event with other Kafka users and providers. Hopefully, we can gather some fresh perspectives and inspiration from the community there.
>> 
>> Best,
>> Chia-Ping
>> 
>>> On 2026/07/07 21:38:22 Jun Rao via dev wrote:
>>> Hi, Jiunn-Yang,
>>> 
>>> Thanks for the reply.
>>> 
>>> Changing the default value only makes sense if it benefits most users.
>>> Most users
>>> probably never increase the number of partitions. Changing the default
>>> config value gives them no benefit and only introduces a probably
>>> surprising behavior
>>> change.
>>> 
>>> Jun
>>> 
>>>> On Tue, Jul 7, 2026 at 1:22 PM Chia-Ping Tsai <ch...@apache.org> wrote:
>>> 
>>>> hi Jiunn
>>>> 
>>>> +1 to Approach A :)
>>>> 
>>>> Best,
>>>> Chia-Ping
>>>> 
>>>> On 2026/07/07 11:55:14 黃竣陽 wrote:
>>>>> Hi Jun, Chia-Ping,
>>>>> 
>>>>> I'd like to first explain why I believe changing the default is
>>>> worthwhile.
>>>>> 
>>>>> Documentation can only help users who already know about the problem.
>>>>> Because the failure mode is silent, most users discover it only after
>>>> data has
>>>>> already been missed. A safer default such as by_duration:PT5S protects
>>>> users
>>>>> automatically, without requiring prior awareness of the issue.
>>>>> 
>>>>> Given that, the question is not whether the issue exists, but how best
>>>> to introduce
>>>>> a safer default.
>>>>> 
>>>>> Approach A — Follow a deprecation cycle and land the change in 5.0
>>>>> - Keep the current default (`latest`) in upcoming 4.x releases, but emit
>>>> a warning that
>>>>> the default will change to `by_duration:PT5S` in 5.0 and explain how
>>>> users can retain
>>>>> the current behavior explicitly.
>>>>> - Change the default to `by_duration:PT5S` in 5.0.
>>>>> - Update shell tools (for example, the console consumer) to explicitly
>>>> use `latest`, preserving
>>>>> the expected "show only new messages" behavior.
>>>>> 
>>>>> Approach B — Change the default in a 4.4 minor release
>>>>> - Change the default to `by_duration:PT5S` in a 4.x minor release.
>>>>> - Leave shell tools unchanged.
>>>>> - Emit a warning explaining the change and how users can explicitly
>>>> restore `latest`.
>>>>> 
>>>>> I favor Approach B:
>>>>> 
>>>>> Silent data loss during partition expansion is a correctness issue, not
>>>> a usability issue.
>>>>> Delaying the safer default until 5.0 leaves users relying on the
>>>> implicit default exposed unnecessarily.
>>>>> Users who need latest can simply pin it explicitly.
>>>>> I don't think shell tools need special handling. by_duration:PT5S only
>>>> replays a few seconds of data,
>>>>> which is negligible in practice and preferable to silently missing
>>>> records. Keeping the behavior consistent
>>>>> is simpler.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年7月7日 清晨5:49 寫道:
>>>>>> 
>>>>>> Hi, Chia-Ping,
>>>>>> 
>>>>>> I am not sure that we need an out-of-the-box solution. The proposal in
>>>> the
>>>>>> KIP is also not out-of-the-box and requires users to opt-in.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <ch...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> hi Jun
>>>>>>> 
>>>>>>> Yes, updating the docs requires minimal changes to the codebase, but
>>>> it is
>>>>>>> not an out-of-the-box solution. From my perspective, this issue can be
>>>>>>> addressed by setting the console consumer's policy back to latest.
>>>>>>> 
>>>>>>> Let's see what Jiunn thinks
>>>>>>> 
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>> On 2026/07/06 15:59:37 Jun Rao via dev wrote:
>>>>>>>> Hi, Chia-Ping,
>>>>>>>> 
>>>>>>>> One potential downside of changing the default is that some existing
>>>>>>> tools
>>>>>>>> (e.g. console consumer) may depend on this behavior. For example, a
>>>> user
>>>>>>>> testing the system may expect to see only new messages and could be
>>>>>>>> surprised by the change. Another option is to leave the default
>>>>>>> unchanged,
>>>>>>>> but document clearly how to prevent losing messages on new partitions
>>>>>>>> through by_duration. The users can then make the corresponding config
>>>>>>>> changes if needed.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Jun,
>>>>>>>>> 
>>>>>>>>> Thanks for the feedback. I agree with your conclusion. It would be
>>>>>>> better
>>>>>>>>> to "make haste" with a new KIP to address this. We can focus the
>>>>>>> discussion
>>>>>>>>> on the following points:
>>>>>>>>> 
>>>>>>>>> 1) Should we change the default reset policy to by_duration for all
>>>>>>>>> consumers?
>>>>>>>>> 2) What is the optimal duration to prevent data loss while avoiding
>>>>>>> large
>>>>>>>>> backlogs? Note that the classic consumer has a 3-second heartbeat
>>>>>>> interval,
>>>>>>>>> whereas the async consumer uses 5 seconds.
>>>>>>>>> 3) What are the potential side effects of replacing latest with
>>>>>>>>> by_duration as the default behavior?
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote:
>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>> 
>>>>>>>>>> I agree that using by_duration with a value of 5s to 10s addresses
>>>>>>> most
>>>>>>>>>> common issues with expanded partitions, given the default
>>>>>>>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that
>>>> as
>>>>>>> the
>>>>>>>>> new
>>>>>>>>>> default. The current proposal may address some other rarer cases,
>>>>>>> but it
>>>>>>>>>> feels a bit like over engineering.
>>>>>>>>>> 
>>>>>>>>>> Thanks,
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <
>>>> chia7712@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> - Short durations fail during extended downtimes, leading to the
>>>>>>> exact
>>>>>>>>>>> silent data
>>>>>>>>>>> loss this KIP aims to prevent.
>>>>>>>>>>> 
>>>>>>>>>>> Losing the records produced to expanded partitions during a
>>>>>>> downtime is
>>>>>>>>>>> actually the expected behavior for the latest policy. This is
>>>>>>> exactly
>>>>>>>>> why,
>>>>>>>>>>> in the current KIP proposal, we prefer by_duration over earliest
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>> expanded partitions—we want to avoid forcing users to consume a
>>>>>>> massive
>>>>>>>>>>> accumulated backlog just because a partition is technically "new".
>>>>>>>>>>> 
>>>>>>>>>>> To quote my previous comment from May 15:
>>>>>>>>>>> "...if a consumer is offline for a while and a new partition is
>>>>>>> created
>>>>>>>>>>> during that downtime, the user might actually want to skip to
>>>>>>> latest
>>>>>>>>> when
>>>>>>>>>>> resuming, rather than reading from earliest just because the
>>>>>>> partition
>>>>>>>>> is
>>>>>>>>>>> technically 'new' to the group."
>>>>>>>>>>> 
>>>>>>>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote:
>>>>>>>>>>>> Hello all,
>>>>>>>>>>>> 
>>>>>>>>>>>> The metadata blindness and consumer downtime windows are
>>>>>>> inherently
>>>>>>>>>>>> unpredictable, making a fixed by_duration approach unviable. No
>>>>>>>>> static
>>>>>>>>>>>> duration can accommodate both extremes:
>>>>>>>>>>>> 
>>>>>>>>>>>> - Short durations fail during extended downtimes, leading to the
>>>>>>>>> exact
>>>>>>>>>>> silent data
>>>>>>>>>>>> loss this KIP aims to prevent.
>>>>>>>>>>>> - Long durations (calibrated for worst-case scenarios) cause
>>>>>>> routine
>>>>>>>>>>> restarts to
>>>>>>>>>>>> trigger unnecessary mass replays, undermining the latest offset
>>>>>>> reset
>>>>>>>>>>> policy.
>>>>>>>>>>>> 
>>>>>>>>>>>> This is why the proposed solution uses creation time. It
>>>>>>> provides a
>>>>>>>>>>> discrete,
>>>>>>>>>>>> deterministic signal—verifying whether a partition is actually
>>>>>>> newer
>>>>>>>>>>> than the group,
>>>>>>>>>>>> rather than relying on an arbitrary time window that has no
>>>>>>>>> correlation
>>>>>>>>>>> with the consumer’s
>>>>>>>>>>>> actual downtime.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I'd like to revisit Jun previous comment about using
>>>>>>>>> by_duration=5s. I
>>>>>>>>>>> realize now that having a small backlog is actually an acceptable
>>>>>>>>>>> trade-off, especially since the latest policy inherently brings a
>>>>>>> tiny
>>>>>>>>>>> backlog to starting consumers anyway due to the time gap between
>>>>>>>>> fetching
>>>>>>>>>>> the offset and actually consuming.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If we bring the simple by_duration=5s solution back to the
>>>>>>> KIP, I'd
>>>>>>>>>>> like to discuss a further step: Should we just change the default
>>>>>>>>> policy
>>>>>>>>>>> from latest to by_duration=5s specifically for the new consumer
>>>> and
>>>>>>>>> share
>>>>>>>>>>> consumer?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The most important part of this story is how users should
>>>>>>> expect
>>>>>>>>> the
>>>>>>>>>>> data
>>>>>>>>>>>>>> they can see when using the latest or by_duration policy with
>>>>>>>>> expanded
>>>>>>>>>>>>>> partitions.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
>>>>>>>>>>>>>> non-deterministic, which means users will either read too many
>>>>>>>>>>> historical
>>>>>>>>>>>>>> records from existing partitions or lose some records from
>>>>>>>>> expanded
>>>>>>>>>>>>>> partitions.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit
>>>>>>> hard to
>>>>>>>>>>>>>> understand, and that is why I preferred having a whole new
>>>>>>> policy
>>>>>>>>>>> based
>>>>>>>>>>>>>> entirely on group creation time (KIP-1282)
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
>>>>>>> 上午1:08寫道:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand the
>>>>>>> value
>>>>>>>>> of
>>>>>>>>>>> the new
>>>>>>>>>>>>>>> configs with the KIP.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want to
>>>>>>> miss the
>>>>>>>>>>> data if
>>>>>>>>>>>>>>> the backlog is small. The backlog of the existing partition
>>>>>>> is
>>>>>>>>> easy
>>>>>>>>>>> to
>>>>>>>>>>>>>>> understand because it relates to retention time. The backlog
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>> new
>>>>>>>>>>>>>>> partition is a bit subtle to understand since it depends on
>>>>>>> the
>>>>>>>>>>> metadata
>>>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user
>>>>>>>>> needs
>>>>>>>>>>> to
>>>>>>>>>>>>>>> understand the metadata refresh delay on the consumer side
>>>>>>> and
>>>>>>>>> use
>>>>>>>>>>> it to
>>>>>>>>>>>>>>> set the config.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Now, let's consider the alternative: setting the same value
>>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>>>>> existing by_duration policy. The KIP lists three issues with
>>>>>>> this
>>>>>>>>>>> approach.
>>>>>>>>>>>>>>> 1. It computes the seek target client-side as now() -
>>>>>>> duration,
>>>>>>>>> which
>>>>>>>>>>>>>>> introduces clock skew across consumers and forces operators
>>>>>>> to
>>>>>>>>> choose
>>>>>>>>>>>>>>> overly large durations, causing unnecessary reprocessing.
>>>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
>>>>>>> failed
>>>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
>>>>>>>>>>> potentially
>>>>>>>>>>>>>>> miss records produced between attempts.
>>>>>>>>>>>>>>> 3. It applies uniformly to all partitions without committed
>>>>>>>>> offsets,
>>>>>>>>>>> and
>>>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
>>>>>>> long-existing
>>>>>>>>>>> partitions
>>>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a
>>>>>>> bit
>>>>>>>>>>> buffer to
>>>>>>>>>>>>>>> the metadata refresh delay. We could also consider improving
>>>>>>> the
>>>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
>>>>>>>>> typically
>>>>>>>>>>> low
>>>>>>>>>>>>>>> (in the order of minutes with the classic consumer and tens
>>>>>>> of
>>>>>>>>>>> seconds with
>>>>>>>>>>>>>>> the new consumer). If a user is ok with reading that much
>>>>>>> backlog
>>>>>>>>>>> for new
>>>>>>>>>>>>>>> partitions, it seems they will be ok doing the same for
>>>>>>> existing
>>>>>>>>>>>>>>> partitions.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> So, instead of introducing a new config, could we just reuse
>>>>>>> the
>>>>>>>>>>> existing
>>>>>>>>>>>>>>> config with better documentation and/or implementation?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> You're right that group creation time is the more intuitive
>>>>>>>>> answer
>>>>>>>>>>> at
>>>>>>>>>>>>>>>> first glance,
>>>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that
>>>>>>> "predate
>>>>>>>>> the
>>>>>>>>>>> group"
>>>>>>>>>>>>>>>> vs partitions
>>>>>>>>>>>>>>>> "created during group runtime," which directly points to a
>>>>>>>>>>>>>>> group-lifecycle
>>>>>>>>>>>>>>>> classifier.
>>>>>>>>>>>>>>>> I'd like to walk through why we landed on partition age,
>>>>>>> and the
>>>>>>>>>>>>>>>> trade-offs we considered.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> We evaluated three candidate signals:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 1. `by_duration:5secs`
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This covers the metadata blindness window, but has issues
>>>>>>> the
>>>>>>>>> KIP
>>>>>>>>>>>>>>>> currently documents
>>>>>>>>>>>>>>>> under "Why not use `by_duration`?":
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
>>>>>>> across
>>>>>>>>>>> consumers.
>>>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
>>>>>>> potentially
>>>>>>>>>>> missing
>>>>>>>>>>>>>>>> records produced between
>>>>>>>>>>>>>>>> attempts.
>>>>>>>>>>>>>>>> - It applies uniformly to all partitions without committed
>>>>>>>>> offsets,
>>>>>>>>>>>>>>>> including pre-existing partitions
>>>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 2. Group creation time as classifier
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This works cleanly when the consumer is actively running.
>>>>>>> Our
>>>>>>>>>>> concern
>>>>>>>>>>>>>>>> is the idle / late-rejoin case:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> T=0:         Group created.
>>>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>>>>>>>>>>>>>> T=50:        Partition added during the idle window.
>>>>>>>>>>>>>>>> T=100:       Consumer resumes.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Under group creation time, the new partition is classified
>>>>>>> as
>>>>>>>>> new
>>>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
>>>>>>> from
>>>>>>>>> T=50.
>>>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
>>>>>>> data
>>>>>>>>>>> that
>>>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the
>>>>>>> contract of
>>>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
>>>>>>>>> treat
>>>>>>>>>>>>>>>> the new partition differently; both contain backlog
>>>>>>> accumulated
>>>>>>>>>>> during
>>>>>>>>>>>>>>>> the same idle window.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle you
>>>>>>> raised
>>>>>>>>> in
>>>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
>>>>>>> backlog
>>>>>>>>> on
>>>>>>>>>>>>>>>> every other partition during the same idle period; forcing
>>>>>>>>> 0-backlog
>>>>>>>>>>>>>>>> tolerance only on new partitions would be inconsistent with
>>>>>>> that
>>>>>>>>>>>>>>>> tolerance.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 3. Partition age vs threshold
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Partition age corresponds to the actual silent data loss
>>>>>>> window,
>>>>>>>>>>>>>>>> the gap between partition creation and the consumer’s
>>>>>>> metadata
>>>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely silent:
>>>>>>> the
>>>>>>>>>>>>>>>> consumer had no opportunity to know about the partition.
>>>>>>> Outside
>>>>>>>>>>> this
>>>>>>>>>>>>>>>> window, missing data reflects either:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
>>>>>>> consumers,
>>>>>>>>> or
>>>>>>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
>>>>>>> via
>>>>>>>>> reset
>>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> We did not choose partition age because it is more elegant
>>>>>>> than
>>>>>>>>>>> group
>>>>>>>>>>>>>>>> creation time — we chose it because its failure mode
>>>>>>> (requires a
>>>>>>>>>>>>>>>> threshold) is
>>>>>>>>>>>>>>>> less invasive than the failure mode of group creation time
>>>>>>>>>>> (overrides
>>>>>>>>>>>>>>>> user-stated
>>>>>>>>>>>>>>>> `latest` intent during idle periods).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
>>>>>>> 寫道:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Relying on both creation times will create an inconsistent
>>>>>>>>>>> scenario. A
>>>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
>>>>>>> seek
>>>>>>>>> to the
>>>>>>>>>>>>>>>>> beginning for the partitions created later than the group.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
>>>>>>>>> inconsistency
>>>>>>>>>>>>>>>> using a
>>>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
>>>>>>> consensus,
>>>>>>>>>>> KIP-1327
>>>>>>>>>>>>>>>> goes
>>>>>>>>>>>>>>>>> back to using flexible configurations to prevent users from
>>>>>>>>> falling
>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>> that pitfall.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
>>>>>>>>> 上午6:49寫道:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still
>>>>>>> feels
>>>>>>>>>>> weird to
>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on the
>>>>>>>>> partition
>>>>>>>>>>>>>>>> creation
>>>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be based on
>>>>>>> the
>>>>>>>>> group
>>>>>>>>>>>>>>>> creation
>>>>>>>>>>>>>>>>>> time: all partitions existing when the group is created
>>>>>>> are
>>>>>>>>>>> existing
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> all partitions created after the group creation are new
>>>>>>>>>>> partitions.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no further
>>>>>>>>>>>>>>>>>>> discussion, I will close the vote.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hello Jian,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for your feedback,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
>>>>>>> task,
>>>>>>>>> not an
>>>>>>>>>>>>>>> edge
>>>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
>>>>>>> clarify
>>>>>>>>> that
>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
>>>>>>>>>>>>>>> demonstrates
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
>>>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational task in
>>>>>>>>> Kafka: To
>>>>>>>>>>>>>>>>>> balance
>>>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
>>>>>>> created
>>>>>>>>>>> with a
>>>>>>>>>>>>>>>>>>> moderate
>>>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
>>>>>>>>> time,
>>>>>>>>>>> it is
>>>>>>>>>>>>>>>>>>> often
>>>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions to
>>>>>>>>> accommodate
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> higher
>>>>>>>>>>>>>>>>>>>>> workload.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>> Jian
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
>>>>>>> 晚上8:29
>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
>>>>>>> when
>>>>>>>>>>>>>>> describing
>>>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
>>>>>>> them to
>>>>>>>>>>>>>>> describe
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
>>>>>>>>> applying
>>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>>> terms
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
>>>>>>>>> whether
>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
>>>>>>>>> they
>>>>>>>>>>> won't
>>>>>>>>>>>>>>>>>>> silently
>>>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
>>>>>>> their
>>>>>>>>>>> active
>>>>>>>>>>>>>>>>>>>>>> consumption.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
>>>>>>>>> motivation
>>>>>>>>>>>>>>>> section.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
>>>>>>> 2026年5月30日
>>>>>>>>>>> 凌晨1:12 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
>>>>>>>>>>> motivation for
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
>>>>>>>>> partitions
>>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
>>>>>>>>> partition
>>>>>>>>>>> (hot)
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
>>>>>>> consumer
>>>>>>>>> has
>>>>>>>>>>> never
>>>>>>>>>>>>>>>>>> seen
>>>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
>>>>>>> hot
>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
>>>>>>> also
>>>>>>>>>>>>>>>> insufficient
>>>>>>>>>>>>>>>>>>>>>>>> because:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
>>>>>>> varies
>>>>>>>>> across
>>>>>>>>>>>>>>> nodes
>>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
>>>>>>> large
>>>>>>>>>>>>>>> duration,
>>>>>>>>>>>>>>>>>>>>>>>> causing
>>>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
>>>>>>> seek
>>>>>>>>> time
>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> retry,
>>>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
>>>>>>> data
>>>>>>>>>>> loss."
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
>>>>>>> these
>>>>>>>>> issues
>>>>>>>>>>>>>>>>>> persist,
>>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
>>>>>>> situations
>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>> need a
>>>>>>>>>>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
>>>>>>> situations,
>>>>>>>>> they
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>>>>>>>>>> customized logic using
>>>>>>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
>>>>>>>>> should
>>>>>>>>>>> always
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
>>>>>>> version
>>>>>>>>>>> mismatch
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
>>>>>>> occurs
>>>>>>>>> inside
>>>>>>>>>>>>>>>>>> poll().
>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
>>>>>>> UnsupportedVersionException
>>>>>>>>> from
>>>>>>>>>>>>>>> poll().
>>>>>>>>>>>>>>>>>>> I’ll
>>>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>>>>>> 2026年5月17日
>>>>>>>>>>> 下午4:50 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
>>>>>>> this
>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>> broker_current_time
>>>>>>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
>>>>>>>>> does not
>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
>>>>>>>>> should
>>>>>>>>>>> always
>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>>>>>>>>> any
>>>>>>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
>>>>>>> when
>>>>>>>>>>> calling
>>>>>>>>>>>>>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
>>>>>>> based on
>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
>>>>>>> reset
>>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to
>>>>>>> all
>>>>>>>>>>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
>>>>>>> resets to
>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>>>>>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
>>>>>>> represented
>>>>>>>>> by a
>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently
>>>>>>> fixed
>>>>>>>>> to
>>>>>>>>>>>>>>>>>> earliest).
>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
>>>>>>>>> public
>>>>>>>>>>>>>>>>>>> user-facing
>>>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>>>>>> 2026年5月16日
>>>>>>>>>>> 清晨7:46
>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
>>>>>>> is
>>>>>>>>> listed
>>>>>>>>>>>>>>>> below:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
>>>>>>>>> default
>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
>>>>>>>>> by_duration and
>>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
>>>>>>>>> auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
>>>>>>>>> .partitions=earliest
>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
>>>>>>> cases
>>>>>>>>> of
>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>>>>>>> .partitions
>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be
>>>>>>> applied
>>>>>>>>> to
>>>>>>>>>>> all
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
>>>>>>> "When a
>>>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
>>>>>>> latest
>>>>>>>>> auto
>>>>>>>>>>>>>>> offset
>>>>>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
>>>>>>> those
>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
>>>>>>>>> record
>>>>>>>>>>> loss
>>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>>>>>>>>> .partitions
>>>>>>>>>>> will
>>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
>>>>>>> set
>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> by_duration if
>>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
>>>>>>> Tsai <
>>>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
>>>>>>>>> something
>>>>>>>>>>>>>>>>>>> important
>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
>>>>>>> to
>>>>>>>>>>> clarify
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
>>>>>>> this
>>>>>>>>>>> logic to
>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
>>>>>>> earliest
>>>>>>>>> or
>>>>>>>>>>>>>>>>>> by_duration
>>>>>>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
>>>>>>> issue
>>>>>>>>> when a
>>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
>>>>>>>>> configure for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they
>>>>>>> set
>>>>>>>>> it to
>>>>>>>>>>>>>>>>>> earliest
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases.
>>>>>>> For
>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>>> if a
>>>>>>>>>>>>>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
>>>>>>>>> created
>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
>>>>>>> latest
>>>>>>>>> when
>>>>>>>>>>>>>>>>>> resuming,
>>>>>>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
>>>>>>> partition
>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> technically
>>>>>>>>>>>>>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing
>>>>>>> a
>>>>>>>>>>> max.age
>>>>>>>>>>>>>>>>>>> threshold.
>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
>>>>>>>>> genuinely
>>>>>>>>>>>>>>>> "hot/new"
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
>>>>>>> bit
>>>>>>>>>>> weird. It
>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
>>>>>>> seems
>>>>>>>>> that
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
>>>>>>> other
>>>>>>>>>>> values
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
>>>>>>>>> way to
>>>>>>>>>>>>>>>> control
>>>>>>>>>>>>>>>>>>>>>> newly
>>>>>>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
>>>>>>> group
>>>>>>>>>>> starts.
>>>>>>>>>>>>>>>>>>> Have we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
>>>>>>> auto.offset.reset.new
>>>>>>>>>>>>>>>> .partitions?
>>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
>>>>>>>>> defaults
>>>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
>>>>>>>>>>> explicitly to
>>>>>>>>>>>>>>>>>>>>>> customize
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
>>>>>>> 晚上10:37
>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
>>>>>>> The
>>>>>>>>>>>>>>>>>>> PartitionMetadata
>>>>>>>>>>>>>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
>>>>>>>>> computed
>>>>>>>>>>>>>>>>>>> server-side
>>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
>>>>>>> partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>>>>>>>>>>>>>> MembershipManager
>>>>>>>>>>>>>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
>>>>>>>>> metadata
>>>>>>>>>>> for
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
>>>>>>> even
>>>>>>>>> if
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>> ID
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
>>>>>>>>> supported
>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
>>>>>>> time.
>>>>>>>>> If the
>>>>>>>>>>>>>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
>>>>>>>>> PartitionAgeMs
>>>>>>>>>>> at
>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
>>>>>>>>> back to
>>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
>>>>>>> signal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>>>>>>>>> 2026年4月29日
>>>>>>>>>>>>>>>> 下午4:04
>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
>>>>>>>>> 'age'
>>>>>>>>>>>>>>>>>> resolution
>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
>>>>>>> the
>>>>>>>>>>> control
>>>>>>>>>>>>>>>>>> plane
>>>>>>>>>>>>>>>>>>>>>>>> clean.
>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
>>>>>>> introducing
>>>>>>>>>>>>>>>> inter-broker
>>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
>>>>>>> single
>>>>>>>>>>> source of
>>>>>>>>>>>>>>>>>> truth
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
>>>>>>>>> should be
>>>>>>>>>>>>>>>>>>> negligible.
>>>>>>>>>>>>>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
>>>>>>>>> configured in
>>>>>>>>>>>>>>>> minutes
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
>>>>>>>>> brokers
>>>>>>>>>>>>>>> won't
>>>>>>>>>>>>>>>>>>> impact
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
>>>>>>> dev@kafka.apache.org>
>>>>>>>>> 於
>>>>>>>>>>>>>>>>>> 2026年4月29日
>>>>>>>>>>>>>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
>>>>>>>>> previous
>>>>>>>>>>>>>>>>>>> conversation
>>>>>>>>>>>>>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
>>>>>>> flow
>>>>>>>>> with
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
>>>>>>> new
>>>>>>>>>>> boolean;
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
>>>>>>> this
>>>>>>>>>>> boolean
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a
>>>>>>> new
>>>>>>>>>>>>>>> partition,
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
>>>>>>> receive
>>>>>>>>> the
>>>>>>>>>>> ages.
>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
>>>>>>> flow
>>>>>>>>> and
>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>> explain
>>>>>>>>>>>>>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
>>>>>>> offsets,
>>>>>>>>> fetch
>>>>>>>>>>>>>>>>>> offsets,
>>>>>>>>>>>>>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
>>>>>>>>> wonder
>>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
>>>>>>> right
>>>>>>>>>>> place
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
>>>>>>> Alternatively, it
>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
>>>>>>>>>>> cautious
>>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
>>>>>>>>> unrelated
>>>>>>>>>>>>>>>> concepts.
>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
>>>>>>> revoking
>>>>>>>>>>>>>>>> partitions.
>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
>>>>>>> corresponding
>>>>>>>>>>> Streams
>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
>>>>>>> would we
>>>>>>>>> do
>>>>>>>>>>> if we
>>>>>>>>>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
>>>>>>> Muralidhar
>>>>>>>>>>> Basani
>>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>> dev
>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
>>>>>>> know
>>>>>>>>>>> about
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>>>>>>>>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
>>>>>>>>>>> PartitionAges on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
>>>>>>>>>>> recommended
>>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on
>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
>>>>>>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
>>>>>>>>> guess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes
>>>>>>> to
>>>>>>>>> Kafka
>>>>>>>>>>>>>>>>>> Streams,
>>>>>>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
>>>>>>> it?
>>>>>>>>> This
>>>>>>>>>>> is
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
>>>>>>>>>>> mistakenly
>>>>>>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
>>>>>>> consumer
>>>>>>>>> falls
>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
>>>>>>>>> per-partition
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
>>>>>>> "consumer
>>>>>>>>>>> resolves
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
>>>>>>>>>>> earliest was
>>>>>>>>>>>>>>>>>>> applied
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
>>>>>>>>> config is
>>>>>>>>>>>>>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
>>>>>>>>> change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
>>>>>>> 於
>>>>>>>>>>>>>>> 2026年4月28日
>>>>>>>>>>>>>>>>>>> 晚上8:03
>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
>>>>>>> partition
>>>>>>>>>>> creation
>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
>>>>>>> users to
>>>>>>>>>>>>>>> diagnose
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
>>>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
>>>>>>> on
>>>>>>>>>>> KIP-1327
>>>>>>>>>>>>>>>>>>> Prevent
>>>>>>>>>>>>>>>>>>>>>> Hot
>>>>>>>>>>>>>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
>>>>>>> newly
>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>> (hot)
>>>>>>>>>>>>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
>>>>>>> threshold
>>>>>>>>>>>>>>>> automatically
>>>>>>>>>>>>>>>>>>> fall
>>>>>>>>>>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
>>>>>>> forcing a
>>>>>>>>>>> full
>>>>>>>>>>>>>>>>>>> historical
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Andrew Schofield <sc...@apache.org>.
Hi,
I'd like to confirm what is happening with this KIP for AK 4.4. I think the answer is that we've decided to improve the docs (I see a PR) and leave it at that for now. Do I understand correctly?

Thanks,
Andrew

On 2026/07/08 01:32:41 Chia-Ping Tsai wrote:
> hi all,
> 
> While I agree we shouldn't surprise the majority of users, relying solely on documentation means some users will still fall into this trap. Realizing this limitation only when data is silently lost during a partition expansion is a truly frustrating experience.
> 
> Since we've reached the KIP freeze for 4.4.0, I recommend we focus on updating the docs first.
> 
> We can put this KIP on hold for now and revisit it after Community Over Code. IIRC, Jiunn-Yang is going to share this specific issue at the event with other Kafka users and providers. Hopefully, we can gather some fresh perspectives and inspiration from the community there.
> 
> Best,
> Chia-Ping
> 
> On 2026/07/07 21:38:22 Jun Rao via dev wrote:
> > Hi, Jiunn-Yang,
> > 
> > Thanks for the reply.
> > 
> > Changing the default value only makes sense if it benefits most users.
> > Most users
> > probably never increase the number of partitions. Changing the default
> > config value gives them no benefit and only introduces a probably
> > surprising behavior
> > change.
> > 
> > Jun
> > 
> > On Tue, Jul 7, 2026 at 1:22 PM Chia-Ping Tsai <ch...@apache.org> wrote:
> > 
> > > hi Jiunn
> > >
> > > +1 to Approach A :)
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2026/07/07 11:55:14 黃竣陽 wrote:
> > > > Hi Jun, Chia-Ping,
> > > >
> > > > I'd like to first explain why I believe changing the default is
> > > worthwhile.
> > > >
> > > > Documentation can only help users who already know about the problem.
> > > > Because the failure mode is silent, most users discover it only after
> > > data has
> > > > already been missed. A safer default such as by_duration:PT5S protects
> > > users
> > > > automatically, without requiring prior awareness of the issue.
> > > >
> > > > Given that, the question is not whether the issue exists, but how best
> > > to introduce
> > > > a safer default.
> > > >
> > > > Approach A — Follow a deprecation cycle and land the change in 5.0
> > > > - Keep the current default (`latest`) in upcoming 4.x releases, but emit
> > > a warning that
> > > > the default will change to `by_duration:PT5S` in 5.0 and explain how
> > > users can retain
> > > > the current behavior explicitly.
> > > > - Change the default to `by_duration:PT5S` in 5.0.
> > > > - Update shell tools (for example, the console consumer) to explicitly
> > > use `latest`, preserving
> > > > the expected "show only new messages" behavior.
> > > >
> > > > Approach B — Change the default in a 4.4 minor release
> > > > - Change the default to `by_duration:PT5S` in a 4.x minor release.
> > > > - Leave shell tools unchanged.
> > > > - Emit a warning explaining the change and how users can explicitly
> > > restore `latest`.
> > > >
> > > > I favor Approach B:
> > > >
> > > > Silent data loss during partition expansion is a correctness issue, not
> > > a usability issue.
> > > > Delaying the safer default until 5.0 leaves users relying on the
> > > implicit default exposed unnecessarily.
> > > > Users who need latest can simply pin it explicitly.
> > > > I don't think shell tools need special handling. by_duration:PT5S only
> > > replays a few seconds of data,
> > > > which is negligible in practice and preferable to silently missing
> > > records. Keeping the behavior consistent
> > > > is simpler.
> > > >
> > > > Best Regards,
> > > > Jiunn-Yang
> > > >
> > > > > Jun Rao via dev <de...@kafka.apache.org> 於 2026年7月7日 清晨5:49 寫道:
> > > > >
> > > > > Hi, Chia-Ping,
> > > > >
> > > > > I am not sure that we need an out-of-the-box solution. The proposal in
> > > the
> > > > > KIP is also not out-of-the-box and requires users to opt-in.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > > > >
> > > > >> hi Jun
> > > > >>
> > > > >> Yes, updating the docs requires minimal changes to the codebase, but
> > > it is
> > > > >> not an out-of-the-box solution. From my perspective, this issue can be
> > > > >> addressed by setting the console consumer's policy back to latest.
> > > > >>
> > > > >> Let's see what Jiunn thinks
> > > > >>
> > > > >> Best,
> > > > >> Chia-Ping
> > > > >>
> > > > >> On 2026/07/06 15:59:37 Jun Rao via dev wrote:
> > > > >>> Hi, Chia-Ping,
> > > > >>>
> > > > >>> One potential downside of changing the default is that some existing
> > > > >> tools
> > > > >>> (e.g. console consumer) may depend on this behavior. For example, a
> > > user
> > > > >>> testing the system may expect to see only new messages and could be
> > > > >>> surprised by the change. Another option is to leave the default
> > > > >> unchanged,
> > > > >>> but document clearly how to prevent losing messages on new partitions
> > > > >>> through by_duration. The users can then make the corresponding config
> > > > >>> changes if needed.
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org>
> > > > >> wrote:
> > > > >>>
> > > > >>>> Hi Jun,
> > > > >>>>
> > > > >>>> Thanks for the feedback. I agree with your conclusion. It would be
> > > > >> better
> > > > >>>> to "make haste" with a new KIP to address this. We can focus the
> > > > >> discussion
> > > > >>>> on the following points:
> > > > >>>>
> > > > >>>> 1) Should we change the default reset policy to by_duration for all
> > > > >>>> consumers?
> > > > >>>> 2) What is the optimal duration to prevent data loss while avoiding
> > > > >> large
> > > > >>>> backlogs? Note that the classic consumer has a 3-second heartbeat
> > > > >> interval,
> > > > >>>> whereas the async consumer uses 5 seconds.
> > > > >>>> 3) What are the potential side effects of replacing latest with
> > > > >>>> by_duration as the default behavior?
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Chia-Ping
> > > > >>>>
> > > > >>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> > > > >>>>> Hi, Chia-Ping,
> > > > >>>>>
> > > > >>>>> I agree that using by_duration with a value of 5s to 10s addresses
> > > > >> most
> > > > >>>>> common issues with expanded partitions, given the default
> > > > >>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that
> > > as
> > > > >> the
> > > > >>>> new
> > > > >>>>> default. The current proposal may address some other rarer cases,
> > > > >> but it
> > > > >>>>> feels a bit like over engineering.
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>>
> > > > >>>>> Jun
> > > > >>>>>
> > > > >>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <
> > > chia7712@apache.org>
> > > > >>>> wrote:
> > > > >>>>>
> > > > >>>>>> - Short durations fail during extended downtimes, leading to the
> > > > >> exact
> > > > >>>>>> silent data
> > > > >>>>>> loss this KIP aims to prevent.
> > > > >>>>>>
> > > > >>>>>> Losing the records produced to expanded partitions during a
> > > > >> downtime is
> > > > >>>>>> actually the expected behavior for the latest policy. This is
> > > > >> exactly
> > > > >>>> why,
> > > > >>>>>> in the current KIP proposal, we prefer by_duration over earliest
> > > > >> for
> > > > >>>> the
> > > > >>>>>> expanded partitions—we want to avoid forcing users to consume a
> > > > >> massive
> > > > >>>>>> accumulated backlog just because a partition is technically "new".
> > > > >>>>>>
> > > > >>>>>> To quote my previous comment from May 15:
> > > > >>>>>> "...if a consumer is offline for a while and a new partition is
> > > > >> created
> > > > >>>>>> during that downtime, the user might actually want to skip to
> > > > >> latest
> > > > >>>> when
> > > > >>>>>> resuming, rather than reading from earliest just because the
> > > > >> partition
> > > > >>>> is
> > > > >>>>>> technically 'new' to the group."
> > > > >>>>>>
> > > > >>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote:
> > > > >>>>>>> Hello all,
> > > > >>>>>>>
> > > > >>>>>>> The metadata blindness and consumer downtime windows are
> > > > >> inherently
> > > > >>>>>>> unpredictable, making a fixed by_duration approach unviable. No
> > > > >>>> static
> > > > >>>>>>> duration can accommodate both extremes:
> > > > >>>>>>>
> > > > >>>>>>> - Short durations fail during extended downtimes, leading to the
> > > > >>>> exact
> > > > >>>>>> silent data
> > > > >>>>>>> loss this KIP aims to prevent.
> > > > >>>>>>> - Long durations (calibrated for worst-case scenarios) cause
> > > > >> routine
> > > > >>>>>> restarts to
> > > > >>>>>>> trigger unnecessary mass replays, undermining the latest offset
> > > > >> reset
> > > > >>>>>> policy.
> > > > >>>>>>>
> > > > >>>>>>> This is why the proposed solution uses creation time. It
> > > > >> provides a
> > > > >>>>>> discrete,
> > > > >>>>>>> deterministic signal—verifying whether a partition is actually
> > > > >> newer
> > > > >>>>>> than the group,
> > > > >>>>>>> rather than relying on an arbitrary time window that has no
> > > > >>>> correlation
> > > > >>>>>> with the consumer’s
> > > > >>>>>>> actual downtime.
> > > > >>>>>>>
> > > > >>>>>>> Best Regards,
> > > > >>>>>>> Jiunn-Yang
> > > > >>>>>>>
> > > > >>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > > > >>>>>>>>
> > > > >>>>>>>> Hi all,
> > > > >>>>>>>>
> > > > >>>>>>>> I'd like to revisit Jun previous comment about using
> > > > >>>> by_duration=5s. I
> > > > >>>>>> realize now that having a small backlog is actually an acceptable
> > > > >>>>>> trade-off, especially since the latest policy inherently brings a
> > > > >> tiny
> > > > >>>>>> backlog to starting consumers anyway due to the time gap between
> > > > >>>> fetching
> > > > >>>>>> the offset and actually consuming.
> > > > >>>>>>>>
> > > > >>>>>>>> If we bring the simple by_duration=5s solution back to the
> > > > >> KIP, I'd
> > > > >>>>>> like to discuss a further step: Should we just change the default
> > > > >>>> policy
> > > > >>>>>> from latest to by_duration=5s specifically for the new consumer
> > > and
> > > > >>>> share
> > > > >>>>>> consumer?
> > > > >>>>>>>>
> > > > >>>>>>>> Best,
> > > > >>>>>>>> Chia-Ping
> > > > >>>>>>>>
> > > > >>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > > > >>>>>>>>> hi Jun
> > > > >>>>>>>>>
> > > > >>>>>>>>> The most important part of this story is how users should
> > > > >> expect
> > > > >>>> the
> > > > >>>>>> data
> > > > >>>>>>>>> they can see when using the latest or by_duration policy with
> > > > >>>> expanded
> > > > >>>>>>>>> partitions.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > > > >>>>>>>>> non-deterministic, which means users will either read too many
> > > > >>>>>> historical
> > > > >>>>>>>>> records from existing partitions or lose some records from
> > > > >>>> expanded
> > > > >>>>>>>>> partitions.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit
> > > > >> hard to
> > > > >>>>>>>>> understand, and that is why I preferred having a whole new
> > > > >> policy
> > > > >>>>>> based
> > > > >>>>>>>>> entirely on group creation time (KIP-1282)
> > > > >>>>>>>>>
> > > > >>>>>>>>> Best,
> > > > >>>>>>>>> Chia-Ping
> > > > >>>>>>>>>
> > > > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > > > >> 上午1:08寫道:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks for the reply. I am still trying to understand the
> > > > >> value
> > > > >>>> of
> > > > >>>>>> the new
> > > > >>>>>>>>>> configs with the KIP.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> The motivation of the KIP is that a user doesn't want to
> > > > >> miss the
> > > > >>>>>> data if
> > > > >>>>>>>>>> the backlog is small. The backlog of the existing partition
> > > > >> is
> > > > >>>> easy
> > > > >>>>>> to
> > > > >>>>>>>>>> understand because it relates to retention time. The backlog
> > > > >> for
> > > > >>>> the
> > > > >>>>>> new
> > > > >>>>>>>>>> partition is a bit subtle to understand since it depends on
> > > > >> the
> > > > >>>>>> metadata
> > > > >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user
> > > > >>>> needs
> > > > >>>>>> to
> > > > >>>>>>>>>> understand the metadata refresh delay on the consumer side
> > > > >> and
> > > > >>>> use
> > > > >>>>>> it to
> > > > >>>>>>>>>> set the config.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Now, let's consider the alternative: setting the same value
> > > > >> for
> > > > >>>> the
> > > > >>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> > > > >> this
> > > > >>>>>> approach.
> > > > >>>>>>>>>> 1. It computes the seek target client-side as now() -
> > > > >> duration,
> > > > >>>> which
> > > > >>>>>>>>>> introduces clock skew across consumers and forces operators
> > > > >> to
> > > > >>>> choose
> > > > >>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> > > > >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
> > > > >> failed
> > > > >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> > > > >>>>>> potentially
> > > > >>>>>>>>>> miss records produced between attempts.
> > > > >>>>>>>>>> 3. It applies uniformly to all partitions without committed
> > > > >>>> offsets,
> > > > >>>>>> and
> > > > >>>>>>>>>> cannot distinguish newly expanded partitions from
> > > > >> long-existing
> > > > >>>>>> partitions
> > > > >>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a
> > > > >> bit
> > > > >>>>>> buffer to
> > > > >>>>>>>>>> the metadata refresh delay. We could also consider improving
> > > > >> the
> > > > >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> > > > >>>> typically
> > > > >>>>>> low
> > > > >>>>>>>>>> (in the order of minutes with the classic consumer and tens
> > > > >> of
> > > > >>>>>> seconds with
> > > > >>>>>>>>>> the new consumer). If a user is ok with reading that much
> > > > >> backlog
> > > > >>>>>> for new
> > > > >>>>>>>>>> partitions, it seems they will be ok doing the same for
> > > > >> existing
> > > > >>>>>>>>>> partitions.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> So, instead of introducing a new config, could we just reuse
> > > > >> the
> > > > >>>>>> existing
> > > > >>>>>>>>>> config with better documentation and/or implementation?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Jun
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > > > >> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Hello Jun,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> You're right that group creation time is the more intuitive
> > > > >>>> answer
> > > > >>>>>> at
> > > > >>>>>>>>>>> first glance,
> > > > >>>>>>>>>>> the KIP's own motivation talks about partitions that
> > > > >> "predate
> > > > >>>> the
> > > > >>>>>> group"
> > > > >>>>>>>>>>> vs partitions
> > > > >>>>>>>>>>> "created during group runtime," which directly points to a
> > > > >>>>>>>>>> group-lifecycle
> > > > >>>>>>>>>>> classifier.
> > > > >>>>>>>>>>> I'd like to walk through why we landed on partition age,
> > > > >> and the
> > > > >>>>>>>>>>> trade-offs we considered.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> We evaluated three candidate signals:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> 1. `by_duration:5secs`
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> This covers the metadata blindness window, but has issues
> > > > >> the
> > > > >>>> KIP
> > > > >>>>>>>>>>> currently documents
> > > > >>>>>>>>>>> under "Why not use `by_duration`?":
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
> > > > >> across
> > > > >>>>>> consumers.
> > > > >>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > > >> potentially
> > > > >>>>>> missing
> > > > >>>>>>>>>>> records produced between
> > > > >>>>>>>>>>> attempts.
> > > > >>>>>>>>>>> - It applies uniformly to all partitions without committed
> > > > >>>> offsets,
> > > > >>>>>>>>>>> including pre-existing partitions
> > > > >>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> 2. Group creation time as classifier
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> This works cleanly when the consumer is actively running.
> > > > >> Our
> > > > >>>>>> concern
> > > > >>>>>>>>>>> is the idle / late-rejoin case:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> T=0:         Group created.
> > > > >>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > > >>>>>>>>>>> T=50:        Partition added during the idle window.
> > > > >>>>>>>>>>> T=100:       Consumer resumes.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Under group creation time, the new partition is classified
> > > > >> as
> > > > >>>> new
> > > > >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
> > > > >> from
> > > > >>>> T=50.
> > > > >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> > > > >> data
> > > > >>>>>> that
> > > > >>>>>>>>>>> the consumer accepts as lost — that is precisely the
> > > > >> contract of
> > > > >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
> > > > >>>> treat
> > > > >>>>>>>>>>> the new partition differently; both contain backlog
> > > > >> accumulated
> > > > >>>>>> during
> > > > >>>>>>>>>>> the same idle window.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> This aligns with the "backlog is backlog” principle you
> > > > >> raised
> > > > >>>> in
> > > > >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
> > > > >> backlog
> > > > >>>> on
> > > > >>>>>>>>>>> every other partition during the same idle period; forcing
> > > > >>>> 0-backlog
> > > > >>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> > > > >> that
> > > > >>>>>>>>>>> tolerance.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> 3. Partition age vs threshold
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Partition age corresponds to the actual silent data loss
> > > > >> window,
> > > > >>>>>>>>>>> the gap between partition creation and the consumer’s
> > > > >> metadata
> > > > >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent:
> > > > >> the
> > > > >>>>>>>>>>> consumer had no opportunity to know about the partition.
> > > > >> Outside
> > > > >>>>>> this
> > > > >>>>>>>>>>> window, missing data reflects either:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> > > > >> consumers,
> > > > >>>> or
> > > > >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
> > > > >> via
> > > > >>>> reset
> > > > >>>>>>>>>> policy.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> We did not choose partition age because it is more elegant
> > > > >> than
> > > > >>>>>> group
> > > > >>>>>>>>>>> creation time — we chose it because its failure mode
> > > > >> (requires a
> > > > >>>>>>>>>>> threshold) is
> > > > >>>>>>>>>>> less invasive than the failure mode of group creation time
> > > > >>>>>> (overrides
> > > > >>>>>>>>>>> user-stated
> > > > >>>>>>>>>>> `latest` intent during idle periods).
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> > > > >> 寫道:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi Jun,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Relying on both creation times will create an inconsistent
> > > > >>>>>> scenario. A
> > > > >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
> > > > >> seek
> > > > >>>> to the
> > > > >>>>>>>>>>>> beginning for the partitions created later than the group.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> > > > >>>> inconsistency
> > > > >>>>>>>>>>> using a
> > > > >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> > > > >> consensus,
> > > > >>>>>> KIP-1327
> > > > >>>>>>>>>>> goes
> > > > >>>>>>>>>>>> back to using flexible configurations to prevent users from
> > > > >>>> falling
> > > > >>>>>>>>>> into
> > > > >>>>>>>>>>>> that pitfall.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best, Chia-Ping
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > > > >>>> 上午6:49寫道:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still
> > > > >> feels
> > > > >>>>>> weird to
> > > > >>>>>>>>>>> me.
> > > > >>>>>>>>>>>>> It
> > > > >>>>>>>>>>>>> categorizes partitions as new or existing based on the
> > > > >>>> partition
> > > > >>>>>>>>>>> creation
> > > > >>>>>>>>>>>>> time. Intuitively, the categorization should be based on
> > > > >> the
> > > > >>>> group
> > > > >>>>>>>>>>> creation
> > > > >>>>>>>>>>>>> time: all partitions existing when the group is created
> > > > >> are
> > > > >>>>>> existing
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>>>>> all partitions created after the group creation are new
> > > > >>>>>> partitions.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > > > >>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> > > > >>>>>>>>>>>>>> discussion, I will close the vote.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Hello Jian,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Thanks for your feedback,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
> > > > >> task,
> > > > >>>> not an
> > > > >>>>>>>>>> edge
> > > > >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
> > > > >> clarify
> > > > >>>> that
> > > > >>>>>> this
> > > > >>>>>>>>>>>>> is a
> > > > >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
> > > > >>>>>>>>>> demonstrates
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>> need for this optimization. For example:
> > > > >>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> > > > >>>> Kafka: To
> > > > >>>>>>>>>>>>> balance
> > > > >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> > > > >> created
> > > > >>>>>> with a
> > > > >>>>>>>>>>>>>> moderate
> > > > >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
> > > > >>>> time,
> > > > >>>>>> it is
> > > > >>>>>>>>>>>>>> often
> > > > >>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> > > > >>>> accommodate
> > > > >>>>>> the
> > > > >>>>>>>>>>>>> higher
> > > > >>>>>>>>>>>>>>>> workload.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Regards
> > > > >>>>>>>>>>>>>>>> Jian
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hello chia,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> > > > >> 晚上8:29
> > > > >>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> > > > >> when
> > > > >>>>>>>>>> describing
> > > > >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
> > > > >> them to
> > > > >>>>>>>>>> describe
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> > > > >>>> applying
> > > > >>>>>>>>>> these
> > > > >>>>>>>>>>>>>> terms
> > > > >>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> > > > >>>> whether
> > > > >>>>>> a
> > > > >>>>>>>>>>>>>> partition
> > > > >>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
> > > > >>>> they
> > > > >>>>>> won't
> > > > >>>>>>>>>>>>>> silently
> > > > >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
> > > > >> their
> > > > >>>>>> active
> > > > >>>>>>>>>>>>>>>>> consumption.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Hello Jun,
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > > > >>>> motivation
> > > > >>>>>>>>>>> section.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > > >> 2026年5月30日
> > > > >>>>>> 凌晨1:12 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > > > >>>>>> motivation for
> > > > >>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> KIP.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> > > > >>>> partitions
> > > > >>>>>>>>>> without
> > > > >>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> > > > >>>> partition
> > > > >>>>>> (hot)
> > > > >>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> > > > >> consumer
> > > > >>>> has
> > > > >>>>>> never
> > > > >>>>>>>>>>>>> seen
> > > > >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
> > > > >> hot
> > > > >>>>>> partition
> > > > >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> > > > >> also
> > > > >>>>>>>>>>> insufficient
> > > > >>>>>>>>>>>>>>>>>>> because:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
> > > > >> varies
> > > > >>>> across
> > > > >>>>>>>>>> nodes
> > > > >>>>>>>>>>>>>> due
> > > > >>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> > > > >> large
> > > > >>>>>>>>>> duration,
> > > > >>>>>>>>>>>>>>>>>>> causing
> > > > >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > > >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
> > > > >> seek
> > > > >>>> time
> > > > >>>>>> on
> > > > >>>>>>>>>>>>> retry,
> > > > >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
> > > > >> data
> > > > >>>>>> loss."
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
> > > > >> these
> > > > >>>> issues
> > > > >>>>>>>>>>>>> persist,
> > > > >>>>>>>>>>>>>>>>> more
> > > > >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > > > >> situations
> > > > >>>>>> don't
> > > > >>>>>>>>>>>>> need a
> > > > >>>>>>>>>>>>>>>>>>> common
> > > > >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > > >> situations,
> > > > >>>> they
> > > > >>>>>> can
> > > > >>>>>>>>>>>>>> implement
> > > > >>>>>>>>>>>>>>>>>>>> customized logic using
> > > > >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > > >>>> s7133700@gmail.com>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > > >>>> should
> > > > >>>>>> always
> > > > >>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>> greater
> > > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > > >>>>  any
> > > > >>>>>>>>>>>>>> field
> > > > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
> > > > >> version
> > > > >>>>>> mismatch
> > > > >>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>> during
> > > > >>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
> > > > >> occurs
> > > > >>>> inside
> > > > >>>>>>>>>>>>> poll().
> > > > >>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > > >>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > > >> UnsupportedVersionException
> > > > >>>> from
> > > > >>>>>>>>>> poll().
> > > > >>>>>>>>>>>>>> I’ll
> > > > >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > >> 2026年5月17日
> > > > >>>>>> 下午4:50 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> > > > >> this
> > > > >>>>>>>>>> partition
> > > > >>>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> > > > >> as
> > > > >>>>>>>>>>>>>>>>> broker_current_time
> > > > >>>>>>>>>>>>>>>>>>> -
> > > > >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > > > >>>> does not
> > > > >>>>>>>>>>>>> support
> > > > >>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > > >>>> should
> > > > >>>>>> always
> > > > >>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>> greater
> > > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > > >>>>  any
> > > > >>>>>>>>>>>>>> field
> > > > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> > > > >> when
> > > > >>>>>> calling
> > > > >>>>>>>>>>>>>>>>> `poll()`?
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> > > > >> based on
> > > > >>>> the
> > > > >>>>>>>>>>>>>> discussion
> > > > >>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > > > >> reset
> > > > >>>>>>>>>> behavior
> > > > >>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to
> > > > >> all
> > > > >>>>>>>>>>>>>> auto.offset.reset
> > > > >>>>>>>>>>>>>>>>>>>>> values
> > > > >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > > >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> > > > >> resets to
> > > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > > > >>>>>>>>>>>>>>>>>>>>>>> config setting
> > > > >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > > >>>>>>>>>> auto.offset.reset
> > > > >>>>>>>>>>>>>>>>> policy
> > > > >>>>>>>>>>>>>>>>>>>>> continues
> > > > >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > > >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > > >> represented
> > > > >>>> by a
> > > > >>>>>>>>>>> separate
> > > > >>>>>>>>>>>>>>>>>>>>> internal config
> > > > >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently
> > > > >> fixed
> > > > >>>> to
> > > > >>>>>>>>>>>>> earliest).
> > > > >>>>>>>>>>>>>>>>> This
> > > > >>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > > >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > > > >>>> public
> > > > >>>>>>>>>>>>>> user-facing
> > > > >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > >> 2026年5月16日
> > > > >>>>>> 清晨7:46
> > > > >>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> > > > >> is
> > > > >>>> listed
> > > > >>>>>>>>>>> below:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
> > > > >>>> default
> > > > >>>>>>>>>> value
> > > > >>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > > > >>>> by_duration and
> > > > >>>>>>>>>>>>> latest,
> > > > >>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > > >>>> auto.offset.reset=earliest.
> > > > >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> > > > >>>>>> internal
> > > > >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > > >>>> .partitions=earliest
> > > > >>>>>>>>>>> already
> > > > >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> > > > >> cases
> > > > >>>> of
> > > > >>>>>> other
> > > > >>>>>>>>>>>>>> values
> > > > >>>>>>>>>>>>>>>>>>> in a
> > > > >>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > > > >> .partitions
> > > > >>>> and
> > > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be
> > > > >> applied
> > > > >>>> to
> > > > >>>>>> all
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>>> consistency.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> > > > >> "When a
> > > > >>>>>> Kafka
> > > > >>>>>>>>>>>>> topic
> > > > >>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>> expanded
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> > > > >> latest
> > > > >>>> auto
> > > > >>>>>>>>>> offset
> > > > >>>>>>>>>>>>>> reset
> > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
> > > > >> those
> > > > >>>>>>>>>> partitions
> > > > >>>>>>>>>>>>>> before
> > > > >>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > > > >>>> record
> > > > >>>>>> loss
> > > > >>>>>>>>>>>>> issue
> > > > >>>>>>>>>>>>>>>>> could
> > > > >>>>>>>>>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > > > >>>> .partitions
> > > > >>>>>> will
> > > > >>>>>>>>>>> take
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>> same
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> > > > >> set
> > > > >>>> it
> > > > >>>>>>>>>>>>>> by_duration if
> > > > >>>>>>>>>>>>>>>>>>>>> needed.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> > > > >> Tsai <
> > > > >>>>>>>>>>>>>>>>> chia7712@apache.org
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > > > >>>> something
> > > > >>>>>>>>>>>>>> important
> > > > >>>>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>> your
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> > > > >> to
> > > > >>>>>> clarify
> > > > >>>>>>>>>> with
> > > > >>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>> few
> > > > >>>>>>>>>>>>>>>>>>>>> questions:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> > > > >> this
> > > > >>>>>> logic to
> > > > >>>>>>>>>>>>>> other
> > > > >>>>>>>>>>>>>>>>>>> reset
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > > > >> earliest
> > > > >>>> or
> > > > >>>>>>>>>>>>> by_duration
> > > > >>>>>>>>>>>>>>>>>>> don't
> > > > >>>>>>>>>>>>>>>>>>>>> seem
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> > > > >> issue
> > > > >>>> when a
> > > > >>>>>>>>>>>>>> partition
> > > > >>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>> expanded.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > > > >>>> configure for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they
> > > > >> set
> > > > >>>> it to
> > > > >>>>>>>>>>>>> earliest
> > > > >>>>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>>>>>> latest,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases.
> > > > >> For
> > > > >>>>>> example,
> > > > >>>>>>>>>>>>> if a
> > > > >>>>>>>>>>>>>>>>>>>>> consumer is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > > > >>>> created
> > > > >>>>>> during
> > > > >>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>> downtime,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> > > > >> latest
> > > > >>>> when
> > > > >>>>>>>>>>>>> resuming,
> > > > >>>>>>>>>>>>>>>>>>> rather
> > > > >>>>>>>>>>>>>>>>>>>>> than
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > > > >> partition
> > > > >>>> is
> > > > >>>>>>>>>>>>>> technically
> > > > >>>>>>>>>>>>>>>>>>>>> "new" to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing
> > > > >> a
> > > > >>>>>> max.age
> > > > >>>>>>>>>>>>>> threshold.
> > > > >>>>>>>>>>>>>>>>>>> It
> > > > >>>>>>>>>>>>>>>>>>>>> gives
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > > > >>>> genuinely
> > > > >>>>>>>>>>> "hot/new"
> > > > >>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>> what is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> > > > >> bit
> > > > >>>>>> weird. It
> > > > >>>>>>>>>>>>> only
> > > > >>>>>>>>>>>>>>>>>>>>> applies when
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> > > > >> seems
> > > > >>>> that
> > > > >>>>>> the
> > > > >>>>>>>>>>>>>>>>> motivation
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> > > > >> other
> > > > >>>>>> values
> > > > >>>>>>>>>> like
> > > > >>>>>>>>>>>>>>>>>>>>> by_duration.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> > > > >>>> way to
> > > > >>>>>>>>>>> control
> > > > >>>>>>>>>>>>>>>>> newly
> > > > >>>>>>>>>>>>>>>>>>>>> created
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > > > >> group
> > > > >>>>>> starts.
> > > > >>>>>>>>>>>>>> Have we
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > > >> auto.offset.reset.new
> > > > >>>>>>>>>>> .partitions?
> > > > >>>>>>>>>>>>>> If
> > > > >>>>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > > > >>>> defaults
> > > > >>>>>> to
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> policy
> > > > >>>>>>>>>>>>>>>>>>>>> used
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > > > >>>>>> explicitly to
> > > > >>>>>>>>>>>>>>>>> customize
> > > > >>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > > >>>>>> s7133700@gmail.com>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> > > > >> 晚上10:37
> > > > >>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> > > > >> The
> > > > >>>>>>>>>>>>>> PartitionMetadata
> > > > >>>>>>>>>>>>>>>>>>>>> struct
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > > > >>>> computed
> > > > >>>>>>>>>>>>>> server-side
> > > > >>>>>>>>>>>>>>>>>>> by
> > > > >>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > > >> partition_creation_time.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > > >>>>>>>>>>>>>> MembershipManager
> > > > >>>>>>>>>>>>>>>>>>>>> detects
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > > > >>>> metadata
> > > > >>>>>> for
> > > > >>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>> affected
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> > > > >> even
> > > > >>>> if
> > > > >>>>>> the
> > > > >>>>>>>>>>>>> topic
> > > > >>>>>>>>>>>>>> ID
> > > > >>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > > > >>>> supported
> > > > >>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> > > > >> time.
> > > > >>>> If the
> > > > >>>>>>>>>>>>>> negotiated
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > > > >>>> PartitionAgeMs
> > > > >>>>>> at
> > > > >>>>>>>>>> all
> > > > >>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > > > >>>> back to
> > > > >>>>>>>>>> latest
> > > > >>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>> risking
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> > > > >> signal.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > > >>>> 2026年4月29日
> > > > >>>>>>>>>>> 下午4:04
> > > > >>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > > > >>>> 'age'
> > > > >>>>>>>>>>>>> resolution
> > > > >>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> > > > >> the
> > > > >>>>>> control
> > > > >>>>>>>>>>>>> plane
> > > > >>>>>>>>>>>>>>>>>>> clean.
> > > > >>>>>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > > > >> introducing
> > > > >>>>>>>>>>> inter-broker
> > > > >>>>>>>>>>>>>>>>> clock
> > > > >>>>>>>>>>>>>>>>>>>>> skew.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> > > > >> single
> > > > >>>>>> source of
> > > > >>>>>>>>>>>>> truth
> > > > >>>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>>> time.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > > > >>>> should be
> > > > >>>>>>>>>>>>>> negligible.
> > > > >>>>>>>>>>>>>>>>>>>>> Given
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > > > >>>> configured in
> > > > >>>>>>>>>>> minutes
> > > > >>>>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>>>>>> hours, a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > > > >>>> brokers
> > > > >>>>>>>>>> won't
> > > > >>>>>>>>>>>>>> impact
> > > > >>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > > >> dev@kafka.apache.org>
> > > > >>>> 於
> > > > >>>>>>>>>>>>> 2026年4月29日
> > > > >>>>>>>>>>>>>>>>>>> 下午3:29
> > > > >>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > > > >>>> previous
> > > > >>>>>>>>>>>>>> conversation
> > > > >>>>>>>>>>>>>>>>>>> but I
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> > > > >> flow
> > > > >>>> with
> > > > >>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> > > > >> new
> > > > >>>>>> boolean;
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>> KIP
> > > > >>>>>>>>>>>>>>>>>>>>> states
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> > > > >> this
> > > > >>>>>> boolean
> > > > >>>>>>>>>> is
> > > > >>>>>>>>>>>>>> set.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a
> > > > >> new
> > > > >>>>>>>>>> partition,
> > > > >>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> > > > >> receive
> > > > >>>> the
> > > > >>>>>> ages.
> > > > >>>>>>>>>>> Is
> > > > >>>>>>>>>>>>>> my
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> > > > >> flow
> > > > >>>> and
> > > > >>>>>> also
> > > > >>>>>>>>>>>>>> explain
> > > > >>>>>>>>>>>>>>>>>>>>> how it
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > > > >> offsets,
> > > > >>>> fetch
> > > > >>>>>>>>>>>>> offsets,
> > > > >>>>>>>>>>>>>>>>>>> etc.).
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > > > >>>> wonder
> > > > >>>>>> if
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> > > > >> right
> > > > >>>>>> place
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>> given
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > > >> Alternatively, it
> > > > >>>>>> could
> > > > >>>>>>>>>>>>> simply
> > > > >>>>>>>>>>>>>>>>>>>>> include
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > > > >>>>>> cautious
> > > > >>>>>>>>>>> about
> > > > >>>>>>>>>>>>>> not
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > > > >>>> unrelated
> > > > >>>>>>>>>>> concepts.
> > > > >>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>> API
> > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> > > > >> revoking
> > > > >>>>>>>>>>> partitions.
> > > > >>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>> fact
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > > >> corresponding
> > > > >>>>>> Streams
> > > > >>>>>>>>>> API
> > > > >>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>>>>>>> suggests
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> > > > >> would we
> > > > >>>> do
> > > > >>>>>> if we
> > > > >>>>>>>>>>>>>> want to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > > > >> Muralidhar
> > > > >>>>>> Basani
> > > > >>>>>>>>>>> via
> > > > >>>>>>>>>>>>>> dev
> > > > >>>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> > > > >> know
> > > > >>>>>> about
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>> gap.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > > >>>>>>>>>>>>>> RequestPartitionAges
> > > > >>>>>>>>>>>>>>>>>>>>> field.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > > > >>>>>> PartitionAges on
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > > > >>>>>> recommended
> > > > >>>>>>>>>>>>> value
> > > > >>>>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on
> > > > >> the
> > > > >>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > > >>>>>>>>>>>>>>>>>>>>> ?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > > > >>>> guess.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes
> > > > >> to
> > > > >>>> Kafka
> > > > >>>>>>>>>>>>> Streams,
> > > > >>>>>>>>>>>>>>>>>>> would
> > > > >>>>>>>>>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > >>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > > >>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> > > > >> it?
> > > > >>>> This
> > > > >>>>>> is
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>> most
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > > > >>>>>> mistakenly
> > > > >>>>>>>>>>>>>> configure
> > > > >>>>>>>>>>>>>>>>>>>>> it. Or
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > > > >> consumer
> > > > >>>> falls
> > > > >>>>>>>>>> back
> > > > >>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>> earliest"
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > > > >>>> per-partition
> > > > >>>>>>>>>> which
> > > > >>>>>>>>>>>>>> isn't
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > > > >> "consumer
> > > > >>>>>> resolves
> > > > >>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>> initial
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > > > >>>>>> earliest was
> > > > >>>>>>>>>>>>>> applied
> > > > >>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > > > >>>> config is
> > > > >>>>>>>>>>>>>> unchanged.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > > >>>>>>>>>>>>>> s7133700@gmail.com>
> > > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> > > > >>>> change.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > > > >> 於
> > > > >>>>>>>>>> 2026年4月28日
> > > > >>>>>>>>>>>>>> 晚上8:03
> > > > >>>>>>>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > > > >> partition
> > > > >>>>>> creation
> > > > >>>>>>>>>>> time
> > > > >>>>>>>>>>>>>> via
> > > > >>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> > > > >> users to
> > > > >>>>>>>>>> diagnose
> > > > >>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > > >>>> auto.offset.reset.latest.max.age
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> > > > >> on
> > > > >>>>>> KIP-1327
> > > > >>>>>>>>>>>>>> Prevent
> > > > >>>>>>>>>>>>>>>>> Hot
> > > > >>>>>>>>>>>>>>>>>>>>> Data
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>
> > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> > > > >> newly
> > > > >>>>>> expanded
> > > > >>>>>>>>>>>>> (hot)
> > > > >>>>>>>>>>>>>>>>>>>>> partitions
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > > > >> threshold
> > > > >>>>>>>>>>> automatically
> > > > >>>>>>>>>>>>>> fall
> > > > >>>>>>>>>>>>>>>>>>>>> back
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> > > > >> forcing a
> > > > >>>>>> full
> > > > >>>>>>>>>>>>>> historical
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> > 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
hi all,

While I agree we shouldn't surprise the majority of users, relying solely on documentation means some users will still fall into this trap. Realizing this limitation only when data is silently lost during a partition expansion is a truly frustrating experience.

Since we've reached the KIP freeze for 4.4.0, I recommend we focus on updating the docs first.

We can put this KIP on hold for now and revisit it after Community Over Code. IIRC, Jiunn-Yang is going to share this specific issue at the event with other Kafka users and providers. Hopefully, we can gather some fresh perspectives and inspiration from the community there.

Best,
Chia-Ping

On 2026/07/07 21:38:22 Jun Rao via dev wrote:
> Hi, Jiunn-Yang,
> 
> Thanks for the reply.
> 
> Changing the default value only makes sense if it benefits most users.
> Most users
> probably never increase the number of partitions. Changing the default
> config value gives them no benefit and only introduces a probably
> surprising behavior
> change.
> 
> Jun
> 
> On Tue, Jul 7, 2026 at 1:22 PM Chia-Ping Tsai <ch...@apache.org> wrote:
> 
> > hi Jiunn
> >
> > +1 to Approach A :)
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/07/07 11:55:14 黃竣陽 wrote:
> > > Hi Jun, Chia-Ping,
> > >
> > > I'd like to first explain why I believe changing the default is
> > worthwhile.
> > >
> > > Documentation can only help users who already know about the problem.
> > > Because the failure mode is silent, most users discover it only after
> > data has
> > > already been missed. A safer default such as by_duration:PT5S protects
> > users
> > > automatically, without requiring prior awareness of the issue.
> > >
> > > Given that, the question is not whether the issue exists, but how best
> > to introduce
> > > a safer default.
> > >
> > > Approach A — Follow a deprecation cycle and land the change in 5.0
> > > - Keep the current default (`latest`) in upcoming 4.x releases, but emit
> > a warning that
> > > the default will change to `by_duration:PT5S` in 5.0 and explain how
> > users can retain
> > > the current behavior explicitly.
> > > - Change the default to `by_duration:PT5S` in 5.0.
> > > - Update shell tools (for example, the console consumer) to explicitly
> > use `latest`, preserving
> > > the expected "show only new messages" behavior.
> > >
> > > Approach B — Change the default in a 4.4 minor release
> > > - Change the default to `by_duration:PT5S` in a 4.x minor release.
> > > - Leave shell tools unchanged.
> > > - Emit a warning explaining the change and how users can explicitly
> > restore `latest`.
> > >
> > > I favor Approach B:
> > >
> > > Silent data loss during partition expansion is a correctness issue, not
> > a usability issue.
> > > Delaying the safer default until 5.0 leaves users relying on the
> > implicit default exposed unnecessarily.
> > > Users who need latest can simply pin it explicitly.
> > > I don't think shell tools need special handling. by_duration:PT5S only
> > replays a few seconds of data,
> > > which is negligible in practice and preferable to silently missing
> > records. Keeping the behavior consistent
> > > is simpler.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > > > Jun Rao via dev <de...@kafka.apache.org> 於 2026年7月7日 清晨5:49 寫道:
> > > >
> > > > Hi, Chia-Ping,
> > > >
> > > > I am not sure that we need an out-of-the-box solution. The proposal in
> > the
> > > > KIP is also not out-of-the-box and requires users to opt-in.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > > >
> > > >> hi Jun
> > > >>
> > > >> Yes, updating the docs requires minimal changes to the codebase, but
> > it is
> > > >> not an out-of-the-box solution. From my perspective, this issue can be
> > > >> addressed by setting the console consumer's policy back to latest.
> > > >>
> > > >> Let's see what Jiunn thinks
> > > >>
> > > >> Best,
> > > >> Chia-Ping
> > > >>
> > > >> On 2026/07/06 15:59:37 Jun Rao via dev wrote:
> > > >>> Hi, Chia-Ping,
> > > >>>
> > > >>> One potential downside of changing the default is that some existing
> > > >> tools
> > > >>> (e.g. console consumer) may depend on this behavior. For example, a
> > user
> > > >>> testing the system may expect to see only new messages and could be
> > > >>> surprised by the change. Another option is to leave the default
> > > >> unchanged,
> > > >>> but document clearly how to prevent losing messages on new partitions
> > > >>> through by_duration. The users can then make the corresponding config
> > > >>> changes if needed.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org>
> > > >> wrote:
> > > >>>
> > > >>>> Hi Jun,
> > > >>>>
> > > >>>> Thanks for the feedback. I agree with your conclusion. It would be
> > > >> better
> > > >>>> to "make haste" with a new KIP to address this. We can focus the
> > > >> discussion
> > > >>>> on the following points:
> > > >>>>
> > > >>>> 1) Should we change the default reset policy to by_duration for all
> > > >>>> consumers?
> > > >>>> 2) What is the optimal duration to prevent data loss while avoiding
> > > >> large
> > > >>>> backlogs? Note that the classic consumer has a 3-second heartbeat
> > > >> interval,
> > > >>>> whereas the async consumer uses 5 seconds.
> > > >>>> 3) What are the potential side effects of replacing latest with
> > > >>>> by_duration as the default behavior?
> > > >>>>
> > > >>>> Best,
> > > >>>> Chia-Ping
> > > >>>>
> > > >>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> > > >>>>> Hi, Chia-Ping,
> > > >>>>>
> > > >>>>> I agree that using by_duration with a value of 5s to 10s addresses
> > > >> most
> > > >>>>> common issues with expanded partitions, given the default
> > > >>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that
> > as
> > > >> the
> > > >>>> new
> > > >>>>> default. The current proposal may address some other rarer cases,
> > > >> but it
> > > >>>>> feels a bit like over engineering.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>>
> > > >>>>> Jun
> > > >>>>>
> > > >>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <
> > chia7712@apache.org>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>>> - Short durations fail during extended downtimes, leading to the
> > > >> exact
> > > >>>>>> silent data
> > > >>>>>> loss this KIP aims to prevent.
> > > >>>>>>
> > > >>>>>> Losing the records produced to expanded partitions during a
> > > >> downtime is
> > > >>>>>> actually the expected behavior for the latest policy. This is
> > > >> exactly
> > > >>>> why,
> > > >>>>>> in the current KIP proposal, we prefer by_duration over earliest
> > > >> for
> > > >>>> the
> > > >>>>>> expanded partitions—we want to avoid forcing users to consume a
> > > >> massive
> > > >>>>>> accumulated backlog just because a partition is technically "new".
> > > >>>>>>
> > > >>>>>> To quote my previous comment from May 15:
> > > >>>>>> "...if a consumer is offline for a while and a new partition is
> > > >> created
> > > >>>>>> during that downtime, the user might actually want to skip to
> > > >> latest
> > > >>>> when
> > > >>>>>> resuming, rather than reading from earliest just because the
> > > >> partition
> > > >>>> is
> > > >>>>>> technically 'new' to the group."
> > > >>>>>>
> > > >>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote:
> > > >>>>>>> Hello all,
> > > >>>>>>>
> > > >>>>>>> The metadata blindness and consumer downtime windows are
> > > >> inherently
> > > >>>>>>> unpredictable, making a fixed by_duration approach unviable. No
> > > >>>> static
> > > >>>>>>> duration can accommodate both extremes:
> > > >>>>>>>
> > > >>>>>>> - Short durations fail during extended downtimes, leading to the
> > > >>>> exact
> > > >>>>>> silent data
> > > >>>>>>> loss this KIP aims to prevent.
> > > >>>>>>> - Long durations (calibrated for worst-case scenarios) cause
> > > >> routine
> > > >>>>>> restarts to
> > > >>>>>>> trigger unnecessary mass replays, undermining the latest offset
> > > >> reset
> > > >>>>>> policy.
> > > >>>>>>>
> > > >>>>>>> This is why the proposed solution uses creation time. It
> > > >> provides a
> > > >>>>>> discrete,
> > > >>>>>>> deterministic signal—verifying whether a partition is actually
> > > >> newer
> > > >>>>>> than the group,
> > > >>>>>>> rather than relying on an arbitrary time window that has no
> > > >>>> correlation
> > > >>>>>> with the consumer’s
> > > >>>>>>> actual downtime.
> > > >>>>>>>
> > > >>>>>>> Best Regards,
> > > >>>>>>> Jiunn-Yang
> > > >>>>>>>
> > > >>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > > >>>>>>>>
> > > >>>>>>>> Hi all,
> > > >>>>>>>>
> > > >>>>>>>> I'd like to revisit Jun previous comment about using
> > > >>>> by_duration=5s. I
> > > >>>>>> realize now that having a small backlog is actually an acceptable
> > > >>>>>> trade-off, especially since the latest policy inherently brings a
> > > >> tiny
> > > >>>>>> backlog to starting consumers anyway due to the time gap between
> > > >>>> fetching
> > > >>>>>> the offset and actually consuming.
> > > >>>>>>>>
> > > >>>>>>>> If we bring the simple by_duration=5s solution back to the
> > > >> KIP, I'd
> > > >>>>>> like to discuss a further step: Should we just change the default
> > > >>>> policy
> > > >>>>>> from latest to by_duration=5s specifically for the new consumer
> > and
> > > >>>> share
> > > >>>>>> consumer?
> > > >>>>>>>>
> > > >>>>>>>> Best,
> > > >>>>>>>> Chia-Ping
> > > >>>>>>>>
> > > >>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > > >>>>>>>>> hi Jun
> > > >>>>>>>>>
> > > >>>>>>>>> The most important part of this story is how users should
> > > >> expect
> > > >>>> the
> > > >>>>>> data
> > > >>>>>>>>> they can see when using the latest or by_duration policy with
> > > >>>> expanded
> > > >>>>>>>>> partitions.
> > > >>>>>>>>>
> > > >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > > >>>>>>>>> non-deterministic, which means users will either read too many
> > > >>>>>> historical
> > > >>>>>>>>> records from existing partitions or lose some records from
> > > >>>> expanded
> > > >>>>>>>>> partitions.
> > > >>>>>>>>>
> > > >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit
> > > >> hard to
> > > >>>>>>>>> understand, and that is why I preferred having a whole new
> > > >> policy
> > > >>>>>> based
> > > >>>>>>>>> entirely on group creation time (KIP-1282)
> > > >>>>>>>>>
> > > >>>>>>>>> Best,
> > > >>>>>>>>> Chia-Ping
> > > >>>>>>>>>
> > > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > > >> 上午1:08寫道:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for the reply. I am still trying to understand the
> > > >> value
> > > >>>> of
> > > >>>>>> the new
> > > >>>>>>>>>> configs with the KIP.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The motivation of the KIP is that a user doesn't want to
> > > >> miss the
> > > >>>>>> data if
> > > >>>>>>>>>> the backlog is small. The backlog of the existing partition
> > > >> is
> > > >>>> easy
> > > >>>>>> to
> > > >>>>>>>>>> understand because it relates to retention time. The backlog
> > > >> for
> > > >>>> the
> > > >>>>>> new
> > > >>>>>>>>>> partition is a bit subtle to understand since it depends on
> > > >> the
> > > >>>>>> metadata
> > > >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user
> > > >>>> needs
> > > >>>>>> to
> > > >>>>>>>>>> understand the metadata refresh delay on the consumer side
> > > >> and
> > > >>>> use
> > > >>>>>> it to
> > > >>>>>>>>>> set the config.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Now, let's consider the alternative: setting the same value
> > > >> for
> > > >>>> the
> > > >>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> > > >> this
> > > >>>>>> approach.
> > > >>>>>>>>>> 1. It computes the seek target client-side as now() -
> > > >> duration,
> > > >>>> which
> > > >>>>>>>>>> introduces clock skew across consumers and forces operators
> > > >> to
> > > >>>> choose
> > > >>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> > > >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
> > > >> failed
> > > >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> > > >>>>>> potentially
> > > >>>>>>>>>> miss records produced between attempts.
> > > >>>>>>>>>> 3. It applies uniformly to all partitions without committed
> > > >>>> offsets,
> > > >>>>>> and
> > > >>>>>>>>>> cannot distinguish newly expanded partitions from
> > > >> long-existing
> > > >>>>>> partitions
> > > >>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a
> > > >> bit
> > > >>>>>> buffer to
> > > >>>>>>>>>> the metadata refresh delay. We could also consider improving
> > > >> the
> > > >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> > > >>>> typically
> > > >>>>>> low
> > > >>>>>>>>>> (in the order of minutes with the classic consumer and tens
> > > >> of
> > > >>>>>> seconds with
> > > >>>>>>>>>> the new consumer). If a user is ok with reading that much
> > > >> backlog
> > > >>>>>> for new
> > > >>>>>>>>>> partitions, it seems they will be ok doing the same for
> > > >> existing
> > > >>>>>>>>>> partitions.
> > > >>>>>>>>>>
> > > >>>>>>>>>> So, instead of introducing a new config, could we just reuse
> > > >> the
> > > >>>>>> existing
> > > >>>>>>>>>> config with better documentation and/or implementation?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Jun
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > > >> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> You're right that group creation time is the more intuitive
> > > >>>> answer
> > > >>>>>> at
> > > >>>>>>>>>>> first glance,
> > > >>>>>>>>>>> the KIP's own motivation talks about partitions that
> > > >> "predate
> > > >>>> the
> > > >>>>>> group"
> > > >>>>>>>>>>> vs partitions
> > > >>>>>>>>>>> "created during group runtime," which directly points to a
> > > >>>>>>>>>> group-lifecycle
> > > >>>>>>>>>>> classifier.
> > > >>>>>>>>>>> I'd like to walk through why we landed on partition age,
> > > >> and the
> > > >>>>>>>>>>> trade-offs we considered.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> We evaluated three candidate signals:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 1. `by_duration:5secs`
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This covers the metadata blindness window, but has issues
> > > >> the
> > > >>>> KIP
> > > >>>>>>>>>>> currently documents
> > > >>>>>>>>>>> under "Why not use `by_duration`?":
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
> > > >> across
> > > >>>>>> consumers.
> > > >>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > >> potentially
> > > >>>>>> missing
> > > >>>>>>>>>>> records produced between
> > > >>>>>>>>>>> attempts.
> > > >>>>>>>>>>> - It applies uniformly to all partitions without committed
> > > >>>> offsets,
> > > >>>>>>>>>>> including pre-existing partitions
> > > >>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 2. Group creation time as classifier
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This works cleanly when the consumer is actively running.
> > > >> Our
> > > >>>>>> concern
> > > >>>>>>>>>>> is the idle / late-rejoin case:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> T=0:         Group created.
> > > >>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > >>>>>>>>>>> T=50:        Partition added during the idle window.
> > > >>>>>>>>>>> T=100:       Consumer resumes.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Under group creation time, the new partition is classified
> > > >> as
> > > >>>> new
> > > >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
> > > >> from
> > > >>>> T=50.
> > > >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> > > >> data
> > > >>>>>> that
> > > >>>>>>>>>>> the consumer accepts as lost — that is precisely the
> > > >> contract of
> > > >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
> > > >>>> treat
> > > >>>>>>>>>>> the new partition differently; both contain backlog
> > > >> accumulated
> > > >>>>>> during
> > > >>>>>>>>>>> the same idle window.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This aligns with the "backlog is backlog” principle you
> > > >> raised
> > > >>>> in
> > > >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
> > > >> backlog
> > > >>>> on
> > > >>>>>>>>>>> every other partition during the same idle period; forcing
> > > >>>> 0-backlog
> > > >>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> > > >> that
> > > >>>>>>>>>>> tolerance.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 3. Partition age vs threshold
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Partition age corresponds to the actual silent data loss
> > > >> window,
> > > >>>>>>>>>>> the gap between partition creation and the consumer’s
> > > >> metadata
> > > >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent:
> > > >> the
> > > >>>>>>>>>>> consumer had no opportunity to know about the partition.
> > > >> Outside
> > > >>>>>> this
> > > >>>>>>>>>>> window, missing data reflects either:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> > > >> consumers,
> > > >>>> or
> > > >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
> > > >> via
> > > >>>> reset
> > > >>>>>>>>>> policy.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> We did not choose partition age because it is more elegant
> > > >> than
> > > >>>>>> group
> > > >>>>>>>>>>> creation time — we chose it because its failure mode
> > > >> (requires a
> > > >>>>>>>>>>> threshold) is
> > > >>>>>>>>>>> less invasive than the failure mode of group creation time
> > > >>>>>> (overrides
> > > >>>>>>>>>>> user-stated
> > > >>>>>>>>>>> `latest` intent during idle periods).
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> > > >> 寫道:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Hi Jun,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Relying on both creation times will create an inconsistent
> > > >>>>>> scenario. A
> > > >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
> > > >> seek
> > > >>>> to the
> > > >>>>>>>>>>>> beginning for the partitions created later than the group.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> > > >>>> inconsistency
> > > >>>>>>>>>>> using a
> > > >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> > > >> consensus,
> > > >>>>>> KIP-1327
> > > >>>>>>>>>>> goes
> > > >>>>>>>>>>>> back to using flexible configurations to prevent users from
> > > >>>> falling
> > > >>>>>>>>>> into
> > > >>>>>>>>>>>> that pitfall.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best, Chia-Ping
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > > >>>> 上午6:49寫道:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still
> > > >> feels
> > > >>>>>> weird to
> > > >>>>>>>>>>> me.
> > > >>>>>>>>>>>>> It
> > > >>>>>>>>>>>>> categorizes partitions as new or existing based on the
> > > >>>> partition
> > > >>>>>>>>>>> creation
> > > >>>>>>>>>>>>> time. Intuitively, the categorization should be based on
> > > >> the
> > > >>>> group
> > > >>>>>>>>>>> creation
> > > >>>>>>>>>>>>> time: all partitions existing when the group is created
> > > >> are
> > > >>>>>> existing
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>> all partitions created after the group creation are new
> > > >>>>>> partitions.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > > >>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> > > >>>>>>>>>>>>>> discussion, I will close the vote.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hello Jian,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for your feedback,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
> > > >> task,
> > > >>>> not an
> > > >>>>>>>>>> edge
> > > >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
> > > >> clarify
> > > >>>> that
> > > >>>>>> this
> > > >>>>>>>>>>>>> is a
> > > >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
> > > >>>>>>>>>> demonstrates
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> need for this optimization. For example:
> > > >>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> > > >>>> Kafka: To
> > > >>>>>>>>>>>>> balance
> > > >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> > > >> created
> > > >>>>>> with a
> > > >>>>>>>>>>>>>> moderate
> > > >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
> > > >>>> time,
> > > >>>>>> it is
> > > >>>>>>>>>>>>>> often
> > > >>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> > > >>>> accommodate
> > > >>>>>> the
> > > >>>>>>>>>>>>> higher
> > > >>>>>>>>>>>>>>>> workload.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Regards
> > > >>>>>>>>>>>>>>>> Jian
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hello chia,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> > > >> 晚上8:29
> > > >>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> > > >> when
> > > >>>>>>>>>> describing
> > > >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
> > > >> them to
> > > >>>>>>>>>> describe
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> > > >>>> applying
> > > >>>>>>>>>> these
> > > >>>>>>>>>>>>>> terms
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> > > >>>> whether
> > > >>>>>> a
> > > >>>>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
> > > >>>> they
> > > >>>>>> won't
> > > >>>>>>>>>>>>>> silently
> > > >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
> > > >> their
> > > >>>>>> active
> > > >>>>>>>>>>>>>>>>> consumption.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > > >>>> motivation
> > > >>>>>>>>>>> section.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > >> 2026年5月30日
> > > >>>>>> 凌晨1:12 寫道:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > > >>>>>> motivation for
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> KIP.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> > > >>>> partitions
> > > >>>>>>>>>> without
> > > >>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> > > >>>> partition
> > > >>>>>> (hot)
> > > >>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> > > >> consumer
> > > >>>> has
> > > >>>>>> never
> > > >>>>>>>>>>>>> seen
> > > >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
> > > >> hot
> > > >>>>>> partition
> > > >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> > > >> also
> > > >>>>>>>>>>> insufficient
> > > >>>>>>>>>>>>>>>>>>> because:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
> > > >> varies
> > > >>>> across
> > > >>>>>>>>>> nodes
> > > >>>>>>>>>>>>>> due
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> > > >> large
> > > >>>>>>>>>> duration,
> > > >>>>>>>>>>>>>>>>>>> causing
> > > >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
> > > >> seek
> > > >>>> time
> > > >>>>>> on
> > > >>>>>>>>>>>>> retry,
> > > >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
> > > >> data
> > > >>>>>> loss."
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
> > > >> these
> > > >>>> issues
> > > >>>>>>>>>>>>> persist,
> > > >>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > > >> situations
> > > >>>>>> don't
> > > >>>>>>>>>>>>> need a
> > > >>>>>>>>>>>>>>>>>>> common
> > > >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > >> situations,
> > > >>>> they
> > > >>>>>> can
> > > >>>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>>>>>>>> customized logic using
> > > >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > >>>> s7133700@gmail.com>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > >>>> should
> > > >>>>>> always
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>> greater
> > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > >>>>  any
> > > >>>>>>>>>>>>>> field
> > > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
> > > >> version
> > > >>>>>> mismatch
> > > >>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>> during
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
> > > >> occurs
> > > >>>> inside
> > > >>>>>>>>>>>>> poll().
> > > >>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > >>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > >> UnsupportedVersionException
> > > >>>> from
> > > >>>>>>>>>> poll().
> > > >>>>>>>>>>>>>> I’ll
> > > >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > >> 2026年5月17日
> > > >>>>>> 下午4:50 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> > > >> this
> > > >>>>>>>>>> partition
> > > >>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> > > >> as
> > > >>>>>>>>>>>>>>>>> broker_current_time
> > > >>>>>>>>>>>>>>>>>>> -
> > > >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > > >>>> does not
> > > >>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > >>>> should
> > > >>>>>> always
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>> greater
> > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > >>>>  any
> > > >>>>>>>>>>>>>> field
> > > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> > > >> when
> > > >>>>>> calling
> > > >>>>>>>>>>>>>>>>> `poll()`?
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> > > >> based on
> > > >>>> the
> > > >>>>>>>>>>>>>> discussion
> > > >>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > > >> reset
> > > >>>>>>>>>> behavior
> > > >>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to
> > > >> all
> > > >>>>>>>>>>>>>> auto.offset.reset
> > > >>>>>>>>>>>>>>>>>>>>> values
> > > >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> > > >> resets to
> > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > > >>>>>>>>>>>>>>>>>>>>>>> config setting
> > > >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > >>>>>>>>>> auto.offset.reset
> > > >>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>> continues
> > > >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > >> represented
> > > >>>> by a
> > > >>>>>>>>>>> separate
> > > >>>>>>>>>>>>>>>>>>>>> internal config
> > > >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently
> > > >> fixed
> > > >>>> to
> > > >>>>>>>>>>>>> earliest).
> > > >>>>>>>>>>>>>>>>> This
> > > >>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > > >>>> public
> > > >>>>>>>>>>>>>> user-facing
> > > >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > >> 2026年5月16日
> > > >>>>>> 清晨7:46
> > > >>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> > > >> is
> > > >>>> listed
> > > >>>>>>>>>>> below:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
> > > >>>> default
> > > >>>>>>>>>> value
> > > >>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > > >>>> by_duration and
> > > >>>>>>>>>>>>> latest,
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > >>>> auto.offset.reset=earliest.
> > > >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> > > >>>>>> internal
> > > >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > >>>> .partitions=earliest
> > > >>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> > > >> cases
> > > >>>> of
> > > >>>>>> other
> > > >>>>>>>>>>>>>> values
> > > >>>>>>>>>>>>>>>>>>> in a
> > > >>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > > >> .partitions
> > > >>>> and
> > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be
> > > >> applied
> > > >>>> to
> > > >>>>>> all
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> consistency.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> > > >> "When a
> > > >>>>>> Kafka
> > > >>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>> expanded
> > > >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> > > >> latest
> > > >>>> auto
> > > >>>>>>>>>> offset
> > > >>>>>>>>>>>>>> reset
> > > >>>>>>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
> > > >> those
> > > >>>>>>>>>> partitions
> > > >>>>>>>>>>>>>> before
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > > >>>> record
> > > >>>>>> loss
> > > >>>>>>>>>>>>> issue
> > > >>>>>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > > >>>> .partitions
> > > >>>>>> will
> > > >>>>>>>>>>> take
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> > > >> set
> > > >>>> it
> > > >>>>>>>>>>>>>> by_duration if
> > > >>>>>>>>>>>>>>>>>>>>> needed.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> > > >> Tsai <
> > > >>>>>>>>>>>>>>>>> chia7712@apache.org
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > > >>>> something
> > > >>>>>>>>>>>>>> important
> > > >>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> > > >> to
> > > >>>>>> clarify
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>> few
> > > >>>>>>>>>>>>>>>>>>>>> questions:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> > > >> this
> > > >>>>>> logic to
> > > >>>>>>>>>>>>>> other
> > > >>>>>>>>>>>>>>>>>>> reset
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > > >> earliest
> > > >>>> or
> > > >>>>>>>>>>>>> by_duration
> > > >>>>>>>>>>>>>>>>>>> don't
> > > >>>>>>>>>>>>>>>>>>>>> seem
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> > > >> issue
> > > >>>> when a
> > > >>>>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>> expanded.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > > >>>> configure for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they
> > > >> set
> > > >>>> it to
> > > >>>>>>>>>>>>> earliest
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>> latest,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases.
> > > >> For
> > > >>>>>> example,
> > > >>>>>>>>>>>>> if a
> > > >>>>>>>>>>>>>>>>>>>>> consumer is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > > >>>> created
> > > >>>>>> during
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> downtime,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> > > >> latest
> > > >>>> when
> > > >>>>>>>>>>>>> resuming,
> > > >>>>>>>>>>>>>>>>>>> rather
> > > >>>>>>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > > >> partition
> > > >>>> is
> > > >>>>>>>>>>>>>> technically
> > > >>>>>>>>>>>>>>>>>>>>> "new" to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing
> > > >> a
> > > >>>>>> max.age
> > > >>>>>>>>>>>>>> threshold.
> > > >>>>>>>>>>>>>>>>>>> It
> > > >>>>>>>>>>>>>>>>>>>>> gives
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > > >>>> genuinely
> > > >>>>>>>>>>> "hot/new"
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> what is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> > > >> bit
> > > >>>>>> weird. It
> > > >>>>>>>>>>>>> only
> > > >>>>>>>>>>>>>>>>>>>>> applies when
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> > > >> seems
> > > >>>> that
> > > >>>>>> the
> > > >>>>>>>>>>>>>>>>> motivation
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> > > >> other
> > > >>>>>> values
> > > >>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>>>> by_duration.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> > > >>>> way to
> > > >>>>>>>>>>> control
> > > >>>>>>>>>>>>>>>>> newly
> > > >>>>>>>>>>>>>>>>>>>>> created
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > > >> group
> > > >>>>>> starts.
> > > >>>>>>>>>>>>>> Have we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > >> auto.offset.reset.new
> > > >>>>>>>>>>> .partitions?
> > > >>>>>>>>>>>>>> If
> > > >>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > > >>>> defaults
> > > >>>>>> to
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > > >>>>>> explicitly to
> > > >>>>>>>>>>>>>>>>> customize
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > >>>>>> s7133700@gmail.com>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> > > >> 晚上10:37
> > > >>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> > > >> The
> > > >>>>>>>>>>>>>> PartitionMetadata
> > > >>>>>>>>>>>>>>>>>>>>> struct
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > > >>>> computed
> > > >>>>>>>>>>>>>> server-side
> > > >>>>>>>>>>>>>>>>>>> by
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > >> partition_creation_time.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > >>>>>>>>>>>>>> MembershipManager
> > > >>>>>>>>>>>>>>>>>>>>> detects
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > > >>>> metadata
> > > >>>>>> for
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> affected
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> > > >> even
> > > >>>> if
> > > >>>>>> the
> > > >>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>> ID
> > > >>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > > >>>> supported
> > > >>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> > > >> time.
> > > >>>> If the
> > > >>>>>>>>>>>>>> negotiated
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > > >>>> PartitionAgeMs
> > > >>>>>> at
> > > >>>>>>>>>> all
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > > >>>> back to
> > > >>>>>>>>>> latest
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> risking
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> > > >> signal.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > >>>> 2026年4月29日
> > > >>>>>>>>>>> 下午4:04
> > > >>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > > >>>> 'age'
> > > >>>>>>>>>>>>> resolution
> > > >>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> > > >> the
> > > >>>>>> control
> > > >>>>>>>>>>>>> plane
> > > >>>>>>>>>>>>>>>>>>> clean.
> > > >>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > > >> introducing
> > > >>>>>>>>>>> inter-broker
> > > >>>>>>>>>>>>>>>>> clock
> > > >>>>>>>>>>>>>>>>>>>>> skew.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> > > >> single
> > > >>>>>> source of
> > > >>>>>>>>>>>>> truth
> > > >>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> time.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > > >>>> should be
> > > >>>>>>>>>>>>>> negligible.
> > > >>>>>>>>>>>>>>>>>>>>> Given
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > > >>>> configured in
> > > >>>>>>>>>>> minutes
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>> hours, a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > > >>>> brokers
> > > >>>>>>>>>> won't
> > > >>>>>>>>>>>>>> impact
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > >> dev@kafka.apache.org>
> > > >>>> 於
> > > >>>>>>>>>>>>> 2026年4月29日
> > > >>>>>>>>>>>>>>>>>>> 下午3:29
> > > >>>>>>>>>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > > >>>> previous
> > > >>>>>>>>>>>>>> conversation
> > > >>>>>>>>>>>>>>>>>>> but I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> > > >> flow
> > > >>>> with
> > > >>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> > > >> new
> > > >>>>>> boolean;
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> KIP
> > > >>>>>>>>>>>>>>>>>>>>> states
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> > > >> this
> > > >>>>>> boolean
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>>>> set.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a
> > > >> new
> > > >>>>>>>>>> partition,
> > > >>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> > > >> receive
> > > >>>> the
> > > >>>>>> ages.
> > > >>>>>>>>>>> Is
> > > >>>>>>>>>>>>>> my
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> > > >> flow
> > > >>>> and
> > > >>>>>> also
> > > >>>>>>>>>>>>>> explain
> > > >>>>>>>>>>>>>>>>>>>>> how it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > > >> offsets,
> > > >>>> fetch
> > > >>>>>>>>>>>>> offsets,
> > > >>>>>>>>>>>>>>>>>>> etc.).
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > > >>>> wonder
> > > >>>>>> if
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> > > >> right
> > > >>>>>> place
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>> given
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > >> Alternatively, it
> > > >>>>>> could
> > > >>>>>>>>>>>>> simply
> > > >>>>>>>>>>>>>>>>>>>>> include
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > > >>>>>> cautious
> > > >>>>>>>>>>> about
> > > >>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > > >>>> unrelated
> > > >>>>>>>>>>> concepts.
> > > >>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>> API
> > > >>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> > > >> revoking
> > > >>>>>>>>>>> partitions.
> > > >>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>> fact
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > >> corresponding
> > > >>>>>> Streams
> > > >>>>>>>>>> API
> > > >>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>>>>> suggests
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> > > >> would we
> > > >>>> do
> > > >>>>>> if we
> > > >>>>>>>>>>>>>> want to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > > >> Muralidhar
> > > >>>>>> Basani
> > > >>>>>>>>>>> via
> > > >>>>>>>>>>>>>> dev
> > > >>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> > > >> know
> > > >>>>>> about
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> gap.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > >>>>>>>>>>>>>> RequestPartitionAges
> > > >>>>>>>>>>>>>>>>>>>>> field.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > > >>>>>> PartitionAges on
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > > >>>>>> recommended
> > > >>>>>>>>>>>>> value
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on
> > > >> the
> > > >>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > >>>>>>>>>>>>>>>>>>>>> ?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > > >>>> guess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes
> > > >> to
> > > >>>> Kafka
> > > >>>>>>>>>>>>> Streams,
> > > >>>>>>>>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > >>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> > > >> it?
> > > >>>> This
> > > >>>>>> is
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> most
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > > >>>>>> mistakenly
> > > >>>>>>>>>>>>>> configure
> > > >>>>>>>>>>>>>>>>>>>>> it. Or
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > > >> consumer
> > > >>>> falls
> > > >>>>>>>>>> back
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>> earliest"
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > > >>>> per-partition
> > > >>>>>>>>>> which
> > > >>>>>>>>>>>>>> isn't
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > > >> "consumer
> > > >>>>>> resolves
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> initial
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > > >>>>>> earliest was
> > > >>>>>>>>>>>>>> applied
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > > >>>> config is
> > > >>>>>>>>>>>>>> unchanged.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > >>>>>>>>>>>>>> s7133700@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> > > >>>> change.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > > >> 於
> > > >>>>>>>>>> 2026年4月28日
> > > >>>>>>>>>>>>>> 晚上8:03
> > > >>>>>>>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > > >> partition
> > > >>>>>> creation
> > > >>>>>>>>>>> time
> > > >>>>>>>>>>>>>> via
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> > > >> users to
> > > >>>>>>>>>> diagnose
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > >>>> auto.offset.reset.latest.max.age
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> > > >> on
> > > >>>>>> KIP-1327
> > > >>>>>>>>>>>>>> Prevent
> > > >>>>>>>>>>>>>>>>> Hot
> > > >>>>>>>>>>>>>>>>>>>>> Data
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> > > >> newly
> > > >>>>>> expanded
> > > >>>>>>>>>>>>> (hot)
> > > >>>>>>>>>>>>>>>>>>>>> partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > > >> threshold
> > > >>>>>>>>>>> automatically
> > > >>>>>>>>>>>>>> fall
> > > >>>>>>>>>>>>>>>>>>>>> back
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> > > >> forcing a
> > > >>>>>> full
> > > >>>>>>>>>>>>>> historical
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Jiunn-Yang,

Thanks for the reply.

Changing the default value only makes sense if it benefits most users.
Most users
probably never increase the number of partitions. Changing the default
config value gives them no benefit and only introduces a probably
surprising behavior
change.

Jun

On Tue, Jul 7, 2026 at 1:22 PM Chia-Ping Tsai <ch...@apache.org> wrote:

> hi Jiunn
>
> +1 to Approach A :)
>
> Best,
> Chia-Ping
>
> On 2026/07/07 11:55:14 黃竣陽 wrote:
> > Hi Jun, Chia-Ping,
> >
> > I'd like to first explain why I believe changing the default is
> worthwhile.
> >
> > Documentation can only help users who already know about the problem.
> > Because the failure mode is silent, most users discover it only after
> data has
> > already been missed. A safer default such as by_duration:PT5S protects
> users
> > automatically, without requiring prior awareness of the issue.
> >
> > Given that, the question is not whether the issue exists, but how best
> to introduce
> > a safer default.
> >
> > Approach A — Follow a deprecation cycle and land the change in 5.0
> > - Keep the current default (`latest`) in upcoming 4.x releases, but emit
> a warning that
> > the default will change to `by_duration:PT5S` in 5.0 and explain how
> users can retain
> > the current behavior explicitly.
> > - Change the default to `by_duration:PT5S` in 5.0.
> > - Update shell tools (for example, the console consumer) to explicitly
> use `latest`, preserving
> > the expected "show only new messages" behavior.
> >
> > Approach B — Change the default in a 4.4 minor release
> > - Change the default to `by_duration:PT5S` in a 4.x minor release.
> > - Leave shell tools unchanged.
> > - Emit a warning explaining the change and how users can explicitly
> restore `latest`.
> >
> > I favor Approach B:
> >
> > Silent data loss during partition expansion is a correctness issue, not
> a usability issue.
> > Delaying the safer default until 5.0 leaves users relying on the
> implicit default exposed unnecessarily.
> > Users who need latest can simply pin it explicitly.
> > I don't think shell tools need special handling. by_duration:PT5S only
> replays a few seconds of data,
> > which is negligible in practice and preferable to silently missing
> records. Keeping the behavior consistent
> > is simpler.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Jun Rao via dev <de...@kafka.apache.org> 於 2026年7月7日 清晨5:49 寫道:
> > >
> > > Hi, Chia-Ping,
> > >
> > > I am not sure that we need an out-of-the-box solution. The proposal in
> the
> > > KIP is also not out-of-the-box and requires users to opt-in.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> > >
> > >> hi Jun
> > >>
> > >> Yes, updating the docs requires minimal changes to the codebase, but
> it is
> > >> not an out-of-the-box solution. From my perspective, this issue can be
> > >> addressed by setting the console consumer's policy back to latest.
> > >>
> > >> Let's see what Jiunn thinks
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >> On 2026/07/06 15:59:37 Jun Rao via dev wrote:
> > >>> Hi, Chia-Ping,
> > >>>
> > >>> One potential downside of changing the default is that some existing
> > >> tools
> > >>> (e.g. console consumer) may depend on this behavior. For example, a
> user
> > >>> testing the system may expect to see only new messages and could be
> > >>> surprised by the change. Another option is to leave the default
> > >> unchanged,
> > >>> but document clearly how to prevent losing messages on new partitions
> > >>> through by_duration. The users can then make the corresponding config
> > >>> changes if needed.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jun
> > >>>
> > >>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org>
> > >> wrote:
> > >>>
> > >>>> Hi Jun,
> > >>>>
> > >>>> Thanks for the feedback. I agree with your conclusion. It would be
> > >> better
> > >>>> to "make haste" with a new KIP to address this. We can focus the
> > >> discussion
> > >>>> on the following points:
> > >>>>
> > >>>> 1) Should we change the default reset policy to by_duration for all
> > >>>> consumers?
> > >>>> 2) What is the optimal duration to prevent data loss while avoiding
> > >> large
> > >>>> backlogs? Note that the classic consumer has a 3-second heartbeat
> > >> interval,
> > >>>> whereas the async consumer uses 5 seconds.
> > >>>> 3) What are the potential side effects of replacing latest with
> > >>>> by_duration as the default behavior?
> > >>>>
> > >>>> Best,
> > >>>> Chia-Ping
> > >>>>
> > >>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> > >>>>> Hi, Chia-Ping,
> > >>>>>
> > >>>>> I agree that using by_duration with a value of 5s to 10s addresses
> > >> most
> > >>>>> common issues with expanded partitions, given the default
> > >>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that
> as
> > >> the
> > >>>> new
> > >>>>> default. The current proposal may address some other rarer cases,
> > >> but it
> > >>>>> feels a bit like over engineering.
> > >>>>>
> > >>>>> Thanks,
> > >>>>>
> > >>>>> Jun
> > >>>>>
> > >>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <
> chia7712@apache.org>
> > >>>> wrote:
> > >>>>>
> > >>>>>> - Short durations fail during extended downtimes, leading to the
> > >> exact
> > >>>>>> silent data
> > >>>>>> loss this KIP aims to prevent.
> > >>>>>>
> > >>>>>> Losing the records produced to expanded partitions during a
> > >> downtime is
> > >>>>>> actually the expected behavior for the latest policy. This is
> > >> exactly
> > >>>> why,
> > >>>>>> in the current KIP proposal, we prefer by_duration over earliest
> > >> for
> > >>>> the
> > >>>>>> expanded partitions—we want to avoid forcing users to consume a
> > >> massive
> > >>>>>> accumulated backlog just because a partition is technically "new".
> > >>>>>>
> > >>>>>> To quote my previous comment from May 15:
> > >>>>>> "...if a consumer is offline for a while and a new partition is
> > >> created
> > >>>>>> during that downtime, the user might actually want to skip to
> > >> latest
> > >>>> when
> > >>>>>> resuming, rather than reading from earliest just because the
> > >> partition
> > >>>> is
> > >>>>>> technically 'new' to the group."
> > >>>>>>
> > >>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote:
> > >>>>>>> Hello all,
> > >>>>>>>
> > >>>>>>> The metadata blindness and consumer downtime windows are
> > >> inherently
> > >>>>>>> unpredictable, making a fixed by_duration approach unviable. No
> > >>>> static
> > >>>>>>> duration can accommodate both extremes:
> > >>>>>>>
> > >>>>>>> - Short durations fail during extended downtimes, leading to the
> > >>>> exact
> > >>>>>> silent data
> > >>>>>>> loss this KIP aims to prevent.
> > >>>>>>> - Long durations (calibrated for worst-case scenarios) cause
> > >> routine
> > >>>>>> restarts to
> > >>>>>>> trigger unnecessary mass replays, undermining the latest offset
> > >> reset
> > >>>>>> policy.
> > >>>>>>>
> > >>>>>>> This is why the proposed solution uses creation time. It
> > >> provides a
> > >>>>>> discrete,
> > >>>>>>> deterministic signal—verifying whether a partition is actually
> > >> newer
> > >>>>>> than the group,
> > >>>>>>> rather than relying on an arbitrary time window that has no
> > >>>> correlation
> > >>>>>> with the consumer’s
> > >>>>>>> actual downtime.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>>>
> > >>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > >>>>>>>>
> > >>>>>>>> Hi all,
> > >>>>>>>>
> > >>>>>>>> I'd like to revisit Jun previous comment about using
> > >>>> by_duration=5s. I
> > >>>>>> realize now that having a small backlog is actually an acceptable
> > >>>>>> trade-off, especially since the latest policy inherently brings a
> > >> tiny
> > >>>>>> backlog to starting consumers anyway due to the time gap between
> > >>>> fetching
> > >>>>>> the offset and actually consuming.
> > >>>>>>>>
> > >>>>>>>> If we bring the simple by_duration=5s solution back to the
> > >> KIP, I'd
> > >>>>>> like to discuss a further step: Should we just change the default
> > >>>> policy
> > >>>>>> from latest to by_duration=5s specifically for the new consumer
> and
> > >>>> share
> > >>>>>> consumer?
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> Chia-Ping
> > >>>>>>>>
> > >>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > >>>>>>>>> hi Jun
> > >>>>>>>>>
> > >>>>>>>>> The most important part of this story is how users should
> > >> expect
> > >>>> the
> > >>>>>> data
> > >>>>>>>>> they can see when using the latest or by_duration policy with
> > >>>> expanded
> > >>>>>>>>> partitions.
> > >>>>>>>>>
> > >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > >>>>>>>>> non-deterministic, which means users will either read too many
> > >>>>>> historical
> > >>>>>>>>> records from existing partitions or lose some records from
> > >>>> expanded
> > >>>>>>>>> partitions.
> > >>>>>>>>>
> > >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit
> > >> hard to
> > >>>>>>>>> understand, and that is why I preferred having a whole new
> > >> policy
> > >>>>>> based
> > >>>>>>>>> entirely on group creation time (KIP-1282)
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Chia-Ping
> > >>>>>>>>>
> > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > >> 上午1:08寫道:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the reply. I am still trying to understand the
> > >> value
> > >>>> of
> > >>>>>> the new
> > >>>>>>>>>> configs with the KIP.
> > >>>>>>>>>>
> > >>>>>>>>>> The motivation of the KIP is that a user doesn't want to
> > >> miss the
> > >>>>>> data if
> > >>>>>>>>>> the backlog is small. The backlog of the existing partition
> > >> is
> > >>>> easy
> > >>>>>> to
> > >>>>>>>>>> understand because it relates to retention time. The backlog
> > >> for
> > >>>> the
> > >>>>>> new
> > >>>>>>>>>> partition is a bit subtle to understand since it depends on
> > >> the
> > >>>>>> metadata
> > >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user
> > >>>> needs
> > >>>>>> to
> > >>>>>>>>>> understand the metadata refresh delay on the consumer side
> > >> and
> > >>>> use
> > >>>>>> it to
> > >>>>>>>>>> set the config.
> > >>>>>>>>>>
> > >>>>>>>>>> Now, let's consider the alternative: setting the same value
> > >> for
> > >>>> the
> > >>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> > >> this
> > >>>>>> approach.
> > >>>>>>>>>> 1. It computes the seek target client-side as now() -
> > >> duration,
> > >>>> which
> > >>>>>>>>>> introduces clock skew across consumers and forces operators
> > >> to
> > >>>> choose
> > >>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> > >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
> > >> failed
> > >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> > >>>>>> potentially
> > >>>>>>>>>> miss records produced between attempts.
> > >>>>>>>>>> 3. It applies uniformly to all partitions without committed
> > >>>> offsets,
> > >>>>>> and
> > >>>>>>>>>> cannot distinguish newly expanded partitions from
> > >> long-existing
> > >>>>>> partitions
> > >>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> > >>>>>>>>>>
> > >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a
> > >> bit
> > >>>>>> buffer to
> > >>>>>>>>>> the metadata refresh delay. We could also consider improving
> > >> the
> > >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> > >>>> typically
> > >>>>>> low
> > >>>>>>>>>> (in the order of minutes with the classic consumer and tens
> > >> of
> > >>>>>> seconds with
> > >>>>>>>>>> the new consumer). If a user is ok with reading that much
> > >> backlog
> > >>>>>> for new
> > >>>>>>>>>> partitions, it seems they will be ok doing the same for
> > >> existing
> > >>>>>>>>>> partitions.
> > >>>>>>>>>>
> > >>>>>>>>>> So, instead of introducing a new config, could we just reuse
> > >> the
> > >>>>>> existing
> > >>>>>>>>>> config with better documentation and/or implementation?
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > >> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>
> > >>>>>>>>>>> You're right that group creation time is the more intuitive
> > >>>> answer
> > >>>>>> at
> > >>>>>>>>>>> first glance,
> > >>>>>>>>>>> the KIP's own motivation talks about partitions that
> > >> "predate
> > >>>> the
> > >>>>>> group"
> > >>>>>>>>>>> vs partitions
> > >>>>>>>>>>> "created during group runtime," which directly points to a
> > >>>>>>>>>> group-lifecycle
> > >>>>>>>>>>> classifier.
> > >>>>>>>>>>> I'd like to walk through why we landed on partition age,
> > >> and the
> > >>>>>>>>>>> trade-offs we considered.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We evaluated three candidate signals:
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. `by_duration:5secs`
> > >>>>>>>>>>>
> > >>>>>>>>>>> This covers the metadata blindness window, but has issues
> > >> the
> > >>>> KIP
> > >>>>>>>>>>> currently documents
> > >>>>>>>>>>> under "Why not use `by_duration`?":
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
> > >> across
> > >>>>>> consumers.
> > >>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > >> potentially
> > >>>>>> missing
> > >>>>>>>>>>> records produced between
> > >>>>>>>>>>> attempts.
> > >>>>>>>>>>> - It applies uniformly to all partitions without committed
> > >>>> offsets,
> > >>>>>>>>>>> including pre-existing partitions
> > >>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. Group creation time as classifier
> > >>>>>>>>>>>
> > >>>>>>>>>>> This works cleanly when the consumer is actively running.
> > >> Our
> > >>>>>> concern
> > >>>>>>>>>>> is the idle / late-rejoin case:
> > >>>>>>>>>>>
> > >>>>>>>>>>> T=0:         Group created.
> > >>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > >>>>>>>>>>> T=50:        Partition added during the idle window.
> > >>>>>>>>>>> T=100:       Consumer resumes.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Under group creation time, the new partition is classified
> > >> as
> > >>>> new
> > >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
> > >> from
> > >>>> T=50.
> > >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> > >> data
> > >>>>>> that
> > >>>>>>>>>>> the consumer accepts as lost — that is precisely the
> > >> contract of
> > >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
> > >>>> treat
> > >>>>>>>>>>> the new partition differently; both contain backlog
> > >> accumulated
> > >>>>>> during
> > >>>>>>>>>>> the same idle window.
> > >>>>>>>>>>>
> > >>>>>>>>>>> This aligns with the "backlog is backlog” principle you
> > >> raised
> > >>>> in
> > >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
> > >> backlog
> > >>>> on
> > >>>>>>>>>>> every other partition during the same idle period; forcing
> > >>>> 0-backlog
> > >>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> > >> that
> > >>>>>>>>>>> tolerance.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 3. Partition age vs threshold
> > >>>>>>>>>>>
> > >>>>>>>>>>> Partition age corresponds to the actual silent data loss
> > >> window,
> > >>>>>>>>>>> the gap between partition creation and the consumer’s
> > >> metadata
> > >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent:
> > >> the
> > >>>>>>>>>>> consumer had no opportunity to know about the partition.
> > >> Outside
> > >>>>>> this
> > >>>>>>>>>>> window, missing data reflects either:
> > >>>>>>>>>>>
> > >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> > >> consumers,
> > >>>> or
> > >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
> > >> via
> > >>>> reset
> > >>>>>>>>>> policy.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We did not choose partition age because it is more elegant
> > >> than
> > >>>>>> group
> > >>>>>>>>>>> creation time — we chose it because its failure mode
> > >> (requires a
> > >>>>>>>>>>> threshold) is
> > >>>>>>>>>>> less invasive than the failure mode of group creation time
> > >>>>>> (overrides
> > >>>>>>>>>>> user-stated
> > >>>>>>>>>>> `latest` intent during idle periods).
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best Regards,
> > >>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> > >> 寫道:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Relying on both creation times will create an inconsistent
> > >>>>>> scenario. A
> > >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
> > >> seek
> > >>>> to the
> > >>>>>>>>>>>> beginning for the partitions created later than the group.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> > >>>> inconsistency
> > >>>>>>>>>>> using a
> > >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> > >> consensus,
> > >>>>>> KIP-1327
> > >>>>>>>>>>> goes
> > >>>>>>>>>>>> back to using flexible configurations to prevent users from
> > >>>> falling
> > >>>>>>>>>> into
> > >>>>>>>>>>>> that pitfall.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > >>>> 上午6:49寫道:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still
> > >> feels
> > >>>>>> weird to
> > >>>>>>>>>>> me.
> > >>>>>>>>>>>>> It
> > >>>>>>>>>>>>> categorizes partitions as new or existing based on the
> > >>>> partition
> > >>>>>>>>>>> creation
> > >>>>>>>>>>>>> time. Intuitively, the categorization should be based on
> > >> the
> > >>>> group
> > >>>>>>>>>>> creation
> > >>>>>>>>>>>>> time: all partitions existing when the group is created
> > >> are
> > >>>>>> existing
> > >>>>>>>>>> and
> > >>>>>>>>>>>>> all partitions created after the group creation are new
> > >>>>>> partitions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > >>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> > >>>>>>>>>>>>>> discussion, I will close the vote.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hello Jian,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for your feedback,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
> > >> task,
> > >>>> not an
> > >>>>>>>>>> edge
> > >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
> > >> clarify
> > >>>> that
> > >>>>>> this
> > >>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
> > >>>>>>>>>> demonstrates
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> need for this optimization. For example:
> > >>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> > >>>> Kafka: To
> > >>>>>>>>>>>>> balance
> > >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> > >> created
> > >>>>>> with a
> > >>>>>>>>>>>>>> moderate
> > >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
> > >>>> time,
> > >>>>>> it is
> > >>>>>>>>>>>>>> often
> > >>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> > >>>> accommodate
> > >>>>>> the
> > >>>>>>>>>>>>> higher
> > >>>>>>>>>>>>>>>> workload.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Regards
> > >>>>>>>>>>>>>>>> Jian
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> > >> 晚上8:29
> > >>>> 寫道:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> > >> when
> > >>>>>>>>>> describing
> > >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
> > >> them to
> > >>>>>>>>>> describe
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> > >>>> applying
> > >>>>>>>>>> these
> > >>>>>>>>>>>>>> terms
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> > >>>> whether
> > >>>>>> a
> > >>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
> > >>>> they
> > >>>>>> won't
> > >>>>>>>>>>>>>> silently
> > >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
> > >> their
> > >>>>>> active
> > >>>>>>>>>>>>>>>>> consumption.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > >>>> motivation
> > >>>>>>>>>>> section.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > >> 2026年5月30日
> > >>>>>> 凌晨1:12 寫道:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > >>>>>> motivation for
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> > >>>> partitions
> > >>>>>>>>>> without
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> > >>>> partition
> > >>>>>> (hot)
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> > >> consumer
> > >>>> has
> > >>>>>> never
> > >>>>>>>>>>>>> seen
> > >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
> > >> hot
> > >>>>>> partition
> > >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> > >> also
> > >>>>>>>>>>> insufficient
> > >>>>>>>>>>>>>>>>>>> because:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
> > >> varies
> > >>>> across
> > >>>>>>>>>> nodes
> > >>>>>>>>>>>>>> due
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> > >> large
> > >>>>>>>>>> duration,
> > >>>>>>>>>>>>>>>>>>> causing
> > >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
> > >> seek
> > >>>> time
> > >>>>>> on
> > >>>>>>>>>>>>> retry,
> > >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
> > >> data
> > >>>>>> loss."
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
> > >> these
> > >>>> issues
> > >>>>>>>>>>>>> persist,
> > >>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > >> situations
> > >>>>>> don't
> > >>>>>>>>>>>>> need a
> > >>>>>>>>>>>>>>>>>>> common
> > >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > >> situations,
> > >>>> they
> > >>>>>> can
> > >>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>>>>> customized logic using
> > >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > >>>> s7133700@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > >>>> should
> > >>>>>> always
> > >>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >>>>  any
> > >>>>>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
> > >> version
> > >>>>>> mismatch
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
> > >> occurs
> > >>>> inside
> > >>>>>>>>>>>>> poll().
> > >>>>>>>>>>>>>>>>>>>>> Therefore, the
> > >>>>>>>>>>>>>>>>>>>>> user would encounter an
> > >> UnsupportedVersionException
> > >>>> from
> > >>>>>>>>>> poll().
> > >>>>>>>>>>>>>> I’ll
> > >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > >> 2026年5月17日
> > >>>>>> 下午4:50 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> > >> this
> > >>>>>>>>>> partition
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> > >> as
> > >>>>>>>>>>>>>>>>> broker_current_time
> > >>>>>>>>>>>>>>>>>>> -
> > >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > >>>> does not
> > >>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > >>>> should
> > >>>>>> always
> > >>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >>>>  any
> > >>>>>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> > >> when
> > >>>>>> calling
> > >>>>>>>>>>>>>>>>> `poll()`?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> > >> based on
> > >>>> the
> > >>>>>>>>>>>>>> discussion
> > >>>>>>>>>>>>>>>>>>>>>>> feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > >> reset
> > >>>>>>>>>> behavior
> > >>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to
> > >> all
> > >>>>>>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> > >> resets to
> > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > >>>>>>>>>>>>>>>>>>>>>>> config setting
> > >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > >>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>> continues
> > >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > >> represented
> > >>>> by a
> > >>>>>>>>>>> separate
> > >>>>>>>>>>>>>>>>>>>>> internal config
> > >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently
> > >> fixed
> > >>>> to
> > >>>>>>>>>>>>> earliest).
> > >>>>>>>>>>>>>>>>> This
> > >>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > >>>> public
> > >>>>>>>>>>>>>> user-facing
> > >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > >> 2026年5月16日
> > >>>>>> 清晨7:46
> > >>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> > >> is
> > >>>> listed
> > >>>>>>>>>>> below:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
> > >>>> default
> > >>>>>>>>>> value
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > >>>> by_duration and
> > >>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > >>>> auto.offset.reset=earliest.
> > >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> > >>>>>> internal
> > >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > >>>> .partitions=earliest
> > >>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> > >> cases
> > >>>> of
> > >>>>>> other
> > >>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>> in a
> > >>>>>>>>>>>>>>>>>>>>> separate KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > >> .partitions
> > >>>> and
> > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be
> > >> applied
> > >>>> to
> > >>>>>> all
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> consistency.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> > >> "When a
> > >>>>>> Kafka
> > >>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> expanded
> > >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> > >> latest
> > >>>> auto
> > >>>>>>>>>> offset
> > >>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
> > >> those
> > >>>>>>>>>> partitions
> > >>>>>>>>>>>>>> before
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > >>>> record
> > >>>>>> loss
> > >>>>>>>>>>>>> issue
> > >>>>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > >>>> .partitions
> > >>>>>> will
> > >>>>>>>>>>> take
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> > >> set
> > >>>> it
> > >>>>>>>>>>>>>> by_duration if
> > >>>>>>>>>>>>>>>>>>>>> needed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> > >> Tsai <
> > >>>>>>>>>>>>>>>>> chia7712@apache.org
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > >>>> something
> > >>>>>>>>>>>>>> important
> > >>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> > >> to
> > >>>>>> clarify
> > >>>>>>>>>> with
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>>> few
> > >>>>>>>>>>>>>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> > >> this
> > >>>>>> logic to
> > >>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > >> earliest
> > >>>> or
> > >>>>>>>>>>>>> by_duration
> > >>>>>>>>>>>>>>>>>>> don't
> > >>>>>>>>>>>>>>>>>>>>> seem
> > >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> > >> issue
> > >>>> when a
> > >>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> expanded.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > >>>> configure for
> > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they
> > >> set
> > >>>> it to
> > >>>>>>>>>>>>> earliest
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases.
> > >> For
> > >>>>>> example,
> > >>>>>>>>>>>>> if a
> > >>>>>>>>>>>>>>>>>>>>> consumer is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > >>>> created
> > >>>>>> during
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> downtime,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> > >> latest
> > >>>> when
> > >>>>>>>>>>>>> resuming,
> > >>>>>>>>>>>>>>>>>>> rather
> > >>>>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > >> partition
> > >>>> is
> > >>>>>>>>>>>>>> technically
> > >>>>>>>>>>>>>>>>>>>>> "new" to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing
> > >> a
> > >>>>>> max.age
> > >>>>>>>>>>>>>> threshold.
> > >>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>> gives
> > >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > >>>> genuinely
> > >>>>>>>>>>> "hot/new"
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> what is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> > >> bit
> > >>>>>> weird. It
> > >>>>>>>>>>>>> only
> > >>>>>>>>>>>>>>>>>>>>> applies when
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> > >> seems
> > >>>> that
> > >>>>>> the
> > >>>>>>>>>>>>>>>>> motivation
> > >>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> > >> other
> > >>>>>> values
> > >>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>>>> by_duration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> > >>>> way to
> > >>>>>>>>>>> control
> > >>>>>>>>>>>>>>>>> newly
> > >>>>>>>>>>>>>>>>>>>>> created
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > >> group
> > >>>>>> starts.
> > >>>>>>>>>>>>>> Have we
> > >>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > >> auto.offset.reset.new
> > >>>>>>>>>>> .partitions?
> > >>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > >>>> defaults
> > >>>>>> to
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > >>>>>> explicitly to
> > >>>>>>>>>>>>>>>>> customize
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > >>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> > >> 晚上10:37
> > >>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> > >> The
> > >>>>>>>>>>>>>> PartitionMetadata
> > >>>>>>>>>>>>>>>>>>>>> struct
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > >>>> computed
> > >>>>>>>>>>>>>> server-side
> > >>>>>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > >> partition_creation_time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > >>>>>>>>>>>>>> MembershipManager
> > >>>>>>>>>>>>>>>>>>>>> detects
> > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > >>>> metadata
> > >>>>>> for
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> affected
> > >>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> > >> even
> > >>>> if
> > >>>>>> the
> > >>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>> ID
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > >>>> supported
> > >>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> > >> time.
> > >>>> If the
> > >>>>>>>>>>>>>> negotiated
> > >>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > >>>> PartitionAgeMs
> > >>>>>> at
> > >>>>>>>>>> all
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > >>>> back to
> > >>>>>>>>>> latest
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> risking
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> > >> signal.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > >>>> 2026年4月29日
> > >>>>>>>>>>> 下午4:04
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > >>>> 'age'
> > >>>>>>>>>>>>> resolution
> > >>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> > >> the
> > >>>>>> control
> > >>>>>>>>>>>>> plane
> > >>>>>>>>>>>>>>>>>>> clean.
> > >>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>> main
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > >> introducing
> > >>>>>>>>>>> inter-broker
> > >>>>>>>>>>>>>>>>> clock
> > >>>>>>>>>>>>>>>>>>>>> skew.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> > >> single
> > >>>>>> source of
> > >>>>>>>>>>>>> truth
> > >>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > >>>> should be
> > >>>>>>>>>>>>>> negligible.
> > >>>>>>>>>>>>>>>>>>>>> Given
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > >>>> configured in
> > >>>>>>>>>>> minutes
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> hours, a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > >>>> brokers
> > >>>>>>>>>> won't
> > >>>>>>>>>>>>>> impact
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > >> dev@kafka.apache.org>
> > >>>> 於
> > >>>>>>>>>>>>> 2026年4月29日
> > >>>>>>>>>>>>>>>>>>> 下午3:29
> > >>>>>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > >>>> previous
> > >>>>>>>>>>>>>> conversation
> > >>>>>>>>>>>>>>>>>>> but I
> > >>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> > >> flow
> > >>>> with
> > >>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> > >> new
> > >>>>>> boolean;
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>>>> states
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> > >> this
> > >>>>>> boolean
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>> set.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a
> > >> new
> > >>>>>>>>>> partition,
> > >>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> > >> receive
> > >>>> the
> > >>>>>> ages.
> > >>>>>>>>>>> Is
> > >>>>>>>>>>>>>> my
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> > >> flow
> > >>>> and
> > >>>>>> also
> > >>>>>>>>>>>>>> explain
> > >>>>>>>>>>>>>>>>>>>>> how it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > >> offsets,
> > >>>> fetch
> > >>>>>>>>>>>>> offsets,
> > >>>>>>>>>>>>>>>>>>> etc.).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > >>>> wonder
> > >>>>>> if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> > >> right
> > >>>>>> place
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>> given
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > >> Alternatively, it
> > >>>>>> could
> > >>>>>>>>>>>>> simply
> > >>>>>>>>>>>>>>>>>>>>> include
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > >>>>>> cautious
> > >>>>>>>>>>> about
> > >>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > >>>> unrelated
> > >>>>>>>>>>> concepts.
> > >>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>> API
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> > >> revoking
> > >>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>> fact
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > >> corresponding
> > >>>>>> Streams
> > >>>>>>>>>> API
> > >>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>> suggests
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> > >> would we
> > >>>> do
> > >>>>>> if we
> > >>>>>>>>>>>>>> want to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > >> Muralidhar
> > >>>>>> Basani
> > >>>>>>>>>>> via
> > >>>>>>>>>>>>>> dev
> > >>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> > >> know
> > >>>>>> about
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> gap.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > >>>>>>>>>>>>>> RequestPartitionAges
> > >>>>>>>>>>>>>>>>>>>>> field.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > >>>>>> PartitionAges on
> > >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > >>>>>> recommended
> > >>>>>>>>>>>>> value
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on
> > >> the
> > >>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > >>>>>>>>>>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > >>>> guess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes
> > >> to
> > >>>> Kafka
> > >>>>>>>>>>>>> Streams,
> > >>>>>>>>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > >>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> > >> it?
> > >>>> This
> > >>>>>> is
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > >>>>>> mistakenly
> > >>>>>>>>>>>>>> configure
> > >>>>>>>>>>>>>>>>>>>>> it. Or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > >> consumer
> > >>>> falls
> > >>>>>>>>>> back
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> earliest"
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > >>>> per-partition
> > >>>>>>>>>> which
> > >>>>>>>>>>>>>> isn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > >> "consumer
> > >>>>>> resolves
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> initial
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > >>>>>> earliest was
> > >>>>>>>>>>>>>> applied
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > >>>> config is
> > >>>>>>>>>>>>>> unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > >>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> > >>>> change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > >> 於
> > >>>>>>>>>> 2026年4月28日
> > >>>>>>>>>>>>>> 晚上8:03
> > >>>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > >> partition
> > >>>>>> creation
> > >>>>>>>>>>> time
> > >>>>>>>>>>>>>> via
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> > >> users to
> > >>>>>>>>>> diagnose
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > >>>> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> > >> on
> > >>>>>> KIP-1327
> > >>>>>>>>>>>>>> Prevent
> > >>>>>>>>>>>>>>>>> Hot
> > >>>>>>>>>>>>>>>>>>>>> Data
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>>>
> > >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> > >> newly
> > >>>>>> expanded
> > >>>>>>>>>>>>> (hot)
> > >>>>>>>>>>>>>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > >> threshold
> > >>>>>>>>>>> automatically
> > >>>>>>>>>>>>>> fall
> > >>>>>>>>>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> > >> forcing a
> > >>>>>> full
> > >>>>>>>>>>>>>> historical
> > >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Jiunn

+1 to Approach A :)

Best,
Chia-Ping

On 2026/07/07 11:55:14 黃竣陽 wrote:
> Hi Jun, Chia-Ping,
> 
> I'd like to first explain why I believe changing the default is worthwhile. 
> 
> Documentation can only help users who already know about the problem. 
> Because the failure mode is silent, most users discover it only after data has 
> already been missed. A safer default such as by_duration:PT5S protects users
> automatically, without requiring prior awareness of the issue. 
> 
> Given that, the question is not whether the issue exists, but how best to introduce 
> a safer default.
> 
> Approach A — Follow a deprecation cycle and land the change in 5.0 
> - Keep the current default (`latest`) in upcoming 4.x releases, but emit a warning that 
> the default will change to `by_duration:PT5S` in 5.0 and explain how users can retain 
> the current behavior explicitly. 
> - Change the default to `by_duration:PT5S` in 5.0. 
> - Update shell tools (for example, the console consumer) to explicitly use `latest`, preserving 
> the expected "show only new messages" behavior.
> 
> Approach B — Change the default in a 4.4 minor release 
> - Change the default to `by_duration:PT5S` in a 4.x minor release. 
> - Leave shell tools unchanged. 
> - Emit a warning explaining the change and how users can explicitly restore `latest`.
> 
> I favor Approach B: 
> 
> Silent data loss during partition expansion is a correctness issue, not a usability issue. 
> Delaying the safer default until 5.0 leaves users relying on the implicit default exposed unnecessarily. 
> Users who need latest can simply pin it explicitly. 
> I don't think shell tools need special handling. by_duration:PT5S only replays a few seconds of data, 
> which is negligible in practice and preferable to silently missing records. Keeping the behavior consistent 
> is simpler.
> 
> Best Regards,
> Jiunn-Yang
> 
> > Jun Rao via dev <de...@kafka.apache.org> 於 2026年7月7日 清晨5:49 寫道:
> > 
> > Hi, Chia-Ping,
> > 
> > I am not sure that we need an out-of-the-box solution. The proposal in the
> > KIP is also not out-of-the-box and requires users to opt-in.
> > 
> > Thanks,
> > 
> > Jun
> > 
> > On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> > 
> >> hi Jun
> >> 
> >> Yes, updating the docs requires minimal changes to the codebase, but it is
> >> not an out-of-the-box solution. From my perspective, this issue can be
> >> addressed by setting the console consumer's policy back to latest.
> >> 
> >> Let's see what Jiunn thinks
> >> 
> >> Best,
> >> Chia-Ping
> >> 
> >> On 2026/07/06 15:59:37 Jun Rao via dev wrote:
> >>> Hi, Chia-Ping,
> >>> 
> >>> One potential downside of changing the default is that some existing
> >> tools
> >>> (e.g. console consumer) may depend on this behavior. For example, a user
> >>> testing the system may expect to see only new messages and could be
> >>> surprised by the change. Another option is to leave the default
> >> unchanged,
> >>> but document clearly how to prevent losing messages on new partitions
> >>> through by_duration. The users can then make the corresponding config
> >>> changes if needed.
> >>> 
> >>> Thanks,
> >>> 
> >>> Jun
> >>> 
> >>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org>
> >> wrote:
> >>> 
> >>>> Hi Jun,
> >>>> 
> >>>> Thanks for the feedback. I agree with your conclusion. It would be
> >> better
> >>>> to "make haste" with a new KIP to address this. We can focus the
> >> discussion
> >>>> on the following points:
> >>>> 
> >>>> 1) Should we change the default reset policy to by_duration for all
> >>>> consumers?
> >>>> 2) What is the optimal duration to prevent data loss while avoiding
> >> large
> >>>> backlogs? Note that the classic consumer has a 3-second heartbeat
> >> interval,
> >>>> whereas the async consumer uses 5 seconds.
> >>>> 3) What are the potential side effects of replacing latest with
> >>>> by_duration as the default behavior?
> >>>> 
> >>>> Best,
> >>>> Chia-Ping
> >>>> 
> >>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> >>>>> Hi, Chia-Ping,
> >>>>> 
> >>>>> I agree that using by_duration with a value of 5s to 10s addresses
> >> most
> >>>>> common issues with expanded partitions, given the default
> >>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that as
> >> the
> >>>> new
> >>>>> default. The current proposal may address some other rarer cases,
> >> but it
> >>>>> feels a bit like over engineering.
> >>>>> 
> >>>>> Thanks,
> >>>>> 
> >>>>> Jun
> >>>>> 
> >>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org>
> >>>> wrote:
> >>>>> 
> >>>>>> - Short durations fail during extended downtimes, leading to the
> >> exact
> >>>>>> silent data
> >>>>>> loss this KIP aims to prevent.
> >>>>>> 
> >>>>>> Losing the records produced to expanded partitions during a
> >> downtime is
> >>>>>> actually the expected behavior for the latest policy. This is
> >> exactly
> >>>> why,
> >>>>>> in the current KIP proposal, we prefer by_duration over earliest
> >> for
> >>>> the
> >>>>>> expanded partitions—we want to avoid forcing users to consume a
> >> massive
> >>>>>> accumulated backlog just because a partition is technically "new".
> >>>>>> 
> >>>>>> To quote my previous comment from May 15:
> >>>>>> "...if a consumer is offline for a while and a new partition is
> >> created
> >>>>>> during that downtime, the user might actually want to skip to
> >> latest
> >>>> when
> >>>>>> resuming, rather than reading from earliest just because the
> >> partition
> >>>> is
> >>>>>> technically 'new' to the group."
> >>>>>> 
> >>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote:
> >>>>>>> Hello all,
> >>>>>>> 
> >>>>>>> The metadata blindness and consumer downtime windows are
> >> inherently
> >>>>>>> unpredictable, making a fixed by_duration approach unviable. No
> >>>> static
> >>>>>>> duration can accommodate both extremes:
> >>>>>>> 
> >>>>>>> - Short durations fail during extended downtimes, leading to the
> >>>> exact
> >>>>>> silent data
> >>>>>>> loss this KIP aims to prevent.
> >>>>>>> - Long durations (calibrated for worst-case scenarios) cause
> >> routine
> >>>>>> restarts to
> >>>>>>> trigger unnecessary mass replays, undermining the latest offset
> >> reset
> >>>>>> policy.
> >>>>>>> 
> >>>>>>> This is why the proposed solution uses creation time. It
> >> provides a
> >>>>>> discrete,
> >>>>>>> deterministic signal—verifying whether a partition is actually
> >> newer
> >>>>>> than the group,
> >>>>>>> rather than relying on an arbitrary time window that has no
> >>>> correlation
> >>>>>> with the consumer’s
> >>>>>>> actual downtime.
> >>>>>>> 
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>> 
> >>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> >>>>>>>> 
> >>>>>>>> Hi all,
> >>>>>>>> 
> >>>>>>>> I'd like to revisit Jun previous comment about using
> >>>> by_duration=5s. I
> >>>>>> realize now that having a small backlog is actually an acceptable
> >>>>>> trade-off, especially since the latest policy inherently brings a
> >> tiny
> >>>>>> backlog to starting consumers anyway due to the time gap between
> >>>> fetching
> >>>>>> the offset and actually consuming.
> >>>>>>>> 
> >>>>>>>> If we bring the simple by_duration=5s solution back to the
> >> KIP, I'd
> >>>>>> like to discuss a further step: Should we just change the default
> >>>> policy
> >>>>>> from latest to by_duration=5s specifically for the new consumer and
> >>>> share
> >>>>>> consumer?
> >>>>>>>> 
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>> 
> >>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> >>>>>>>>> hi Jun
> >>>>>>>>> 
> >>>>>>>>> The most important part of this story is how users should
> >> expect
> >>>> the
> >>>>>> data
> >>>>>>>>> they can see when using the latest or by_duration policy with
> >>>> expanded
> >>>>>>>>> partitions.
> >>>>>>>>> 
> >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> >>>>>>>>> non-deterministic, which means users will either read too many
> >>>>>> historical
> >>>>>>>>> records from existing partitions or lose some records from
> >>>> expanded
> >>>>>>>>> partitions.
> >>>>>>>>> 
> >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit
> >> hard to
> >>>>>>>>> understand, and that is why I preferred having a whole new
> >> policy
> >>>>>> based
> >>>>>>>>> entirely on group creation time (KIP-1282)
> >>>>>>>>> 
> >>>>>>>>> Best,
> >>>>>>>>> Chia-Ping
> >>>>>>>>> 
> >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> >> 上午1:08寫道:
> >>>>>>>>> 
> >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> >>>>>>>>>> 
> >>>>>>>>>> Thanks for the reply. I am still trying to understand the
> >> value
> >>>> of
> >>>>>> the new
> >>>>>>>>>> configs with the KIP.
> >>>>>>>>>> 
> >>>>>>>>>> The motivation of the KIP is that a user doesn't want to
> >> miss the
> >>>>>> data if
> >>>>>>>>>> the backlog is small. The backlog of the existing partition
> >> is
> >>>> easy
> >>>>>> to
> >>>>>>>>>> understand because it relates to retention time. The backlog
> >> for
> >>>> the
> >>>>>> new
> >>>>>>>>>> partition is a bit subtle to understand since it depends on
> >> the
> >>>>>> metadata
> >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user
> >>>> needs
> >>>>>> to
> >>>>>>>>>> understand the metadata refresh delay on the consumer side
> >> and
> >>>> use
> >>>>>> it to
> >>>>>>>>>> set the config.
> >>>>>>>>>> 
> >>>>>>>>>> Now, let's consider the alternative: setting the same value
> >> for
> >>>> the
> >>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> >> this
> >>>>>> approach.
> >>>>>>>>>> 1. It computes the seek target client-side as now() -
> >> duration,
> >>>> which
> >>>>>>>>>> introduces clock skew across consumers and forces operators
> >> to
> >>>> choose
> >>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
> >> failed
> >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> >>>>>> potentially
> >>>>>>>>>> miss records produced between attempts.
> >>>>>>>>>> 3. It applies uniformly to all partitions without committed
> >>>> offsets,
> >>>>>> and
> >>>>>>>>>> cannot distinguish newly expanded partitions from
> >> long-existing
> >>>>>> partitions
> >>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> >>>>>>>>>> 
> >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a
> >> bit
> >>>>>> buffer to
> >>>>>>>>>> the metadata refresh delay. We could also consider improving
> >> the
> >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> >>>> typically
> >>>>>> low
> >>>>>>>>>> (in the order of minutes with the classic consumer and tens
> >> of
> >>>>>> seconds with
> >>>>>>>>>> the new consumer). If a user is ok with reading that much
> >> backlog
> >>>>>> for new
> >>>>>>>>>> partitions, it seems they will be ok doing the same for
> >> existing
> >>>>>>>>>> partitions.
> >>>>>>>>>> 
> >>>>>>>>>> So, instead of introducing a new config, could we just reuse
> >> the
> >>>>>> existing
> >>>>>>>>>> config with better documentation and/or implementation?
> >>>>>>>>>> 
> >>>>>>>>>> Jun
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> >> wrote:
> >>>>>>>>>> 
> >>>>>>>>>>> Hello Jun,
> >>>>>>>>>>> 
> >>>>>>>>>>> You're right that group creation time is the more intuitive
> >>>> answer
> >>>>>> at
> >>>>>>>>>>> first glance,
> >>>>>>>>>>> the KIP's own motivation talks about partitions that
> >> "predate
> >>>> the
> >>>>>> group"
> >>>>>>>>>>> vs partitions
> >>>>>>>>>>> "created during group runtime," which directly points to a
> >>>>>>>>>> group-lifecycle
> >>>>>>>>>>> classifier.
> >>>>>>>>>>> I'd like to walk through why we landed on partition age,
> >> and the
> >>>>>>>>>>> trade-offs we considered.
> >>>>>>>>>>> 
> >>>>>>>>>>> We evaluated three candidate signals:
> >>>>>>>>>>> 
> >>>>>>>>>>> 1. `by_duration:5secs`
> >>>>>>>>>>> 
> >>>>>>>>>>> This covers the metadata blindness window, but has issues
> >> the
> >>>> KIP
> >>>>>>>>>>> currently documents
> >>>>>>>>>>> under "Why not use `by_duration`?":
> >>>>>>>>>>> 
> >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
> >> across
> >>>>>> consumers.
> >>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> >> potentially
> >>>>>> missing
> >>>>>>>>>>> records produced between
> >>>>>>>>>>> attempts.
> >>>>>>>>>>> - It applies uniformly to all partitions without committed
> >>>> offsets,
> >>>>>>>>>>> including pre-existing partitions
> >>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> >>>>>>>>>>> 
> >>>>>>>>>>> 2. Group creation time as classifier
> >>>>>>>>>>> 
> >>>>>>>>>>> This works cleanly when the consumer is actively running.
> >> Our
> >>>>>> concern
> >>>>>>>>>>> is the idle / late-rejoin case:
> >>>>>>>>>>> 
> >>>>>>>>>>> T=0:         Group created.
> >>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> >>>>>>>>>>> T=50:        Partition added during the idle window.
> >>>>>>>>>>> T=100:       Consumer resumes.
> >>>>>>>>>>> 
> >>>>>>>>>>> Under group creation time, the new partition is classified
> >> as
> >>>> new
> >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
> >> from
> >>>> T=50.
> >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> >> data
> >>>>>> that
> >>>>>>>>>>> the consumer accepts as lost — that is precisely the
> >> contract of
> >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
> >>>> treat
> >>>>>>>>>>> the new partition differently; both contain backlog
> >> accumulated
> >>>>>> during
> >>>>>>>>>>> the same idle window.
> >>>>>>>>>>> 
> >>>>>>>>>>> This aligns with the "backlog is backlog” principle you
> >> raised
> >>>> in
> >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
> >> backlog
> >>>> on
> >>>>>>>>>>> every other partition during the same idle period; forcing
> >>>> 0-backlog
> >>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> >> that
> >>>>>>>>>>> tolerance.
> >>>>>>>>>>> 
> >>>>>>>>>>> 3. Partition age vs threshold
> >>>>>>>>>>> 
> >>>>>>>>>>> Partition age corresponds to the actual silent data loss
> >> window,
> >>>>>>>>>>> the gap between partition creation and the consumer’s
> >> metadata
> >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent:
> >> the
> >>>>>>>>>>> consumer had no opportunity to know about the partition.
> >> Outside
> >>>>>> this
> >>>>>>>>>>> window, missing data reflects either:
> >>>>>>>>>>> 
> >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> >> consumers,
> >>>> or
> >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
> >> via
> >>>> reset
> >>>>>>>>>> policy.
> >>>>>>>>>>> 
> >>>>>>>>>>> We did not choose partition age because it is more elegant
> >> than
> >>>>>> group
> >>>>>>>>>>> creation time — we chose it because its failure mode
> >> (requires a
> >>>>>>>>>>> threshold) is
> >>>>>>>>>>> less invasive than the failure mode of group creation time
> >>>>>> (overrides
> >>>>>>>>>>> user-stated
> >>>>>>>>>>> `latest` intent during idle periods).
> >>>>>>>>>>> 
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>> 
> >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> >> 寫道:
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Relying on both creation times will create an inconsistent
> >>>>>> scenario. A
> >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
> >> seek
> >>>> to the
> >>>>>>>>>>>> beginning for the partitions created later than the group.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> >>>> inconsistency
> >>>>>>>>>>> using a
> >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> >> consensus,
> >>>>>> KIP-1327
> >>>>>>>>>>> goes
> >>>>>>>>>>>> back to using flexible configurations to prevent users from
> >>>> falling
> >>>>>>>>>> into
> >>>>>>>>>>>> that pitfall.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> >>>> 上午6:49寫道:
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still
> >> feels
> >>>>>> weird to
> >>>>>>>>>>> me.
> >>>>>>>>>>>>> It
> >>>>>>>>>>>>> categorizes partitions as new or existing based on the
> >>>> partition
> >>>>>>>>>>> creation
> >>>>>>>>>>>>> time. Intuitively, the categorization should be based on
> >> the
> >>>> group
> >>>>>>>>>>> creation
> >>>>>>>>>>>>> time: all partitions existing when the group is created
> >> are
> >>>>>> existing
> >>>>>>>>>> and
> >>>>>>>>>>>>> all partitions created after the group creation are new
> >>>>>> partitions.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> >>>>>>>>>>>>>> discussion, I will close the vote.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Hello Jian,
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Thanks for your feedback,
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
> >> task,
> >>>> not an
> >>>>>>>>>> edge
> >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
> >> clarify
> >>>> that
> >>>>>> this
> >>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
> >>>>>>>>>> demonstrates
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> need for this optimization. For example:
> >>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> >>>> Kafka: To
> >>>>>>>>>>>>> balance
> >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> >> created
> >>>>>> with a
> >>>>>>>>>>>>>> moderate
> >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
> >>>> time,
> >>>>>> it is
> >>>>>>>>>>>>>> often
> >>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> >>>> accommodate
> >>>>>> the
> >>>>>>>>>>>>> higher
> >>>>>>>>>>>>>>>> workload.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Regards
> >>>>>>>>>>>>>>>> Jian
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> >> 晚上8:29
> >>>> 寫道:
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> >> when
> >>>>>>>>>> describing
> >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
> >> them to
> >>>>>>>>>> describe
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> >>>> applying
> >>>>>>>>>> these
> >>>>>>>>>>>>>> terms
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> >>>> whether
> >>>>>> a
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
> >>>> they
> >>>>>> won't
> >>>>>>>>>>>>>> silently
> >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
> >> their
> >>>>>> active
> >>>>>>>>>>>>>>>>> consumption.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> >>>> motivation
> >>>>>>>>>>> section.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> >> 2026年5月30日
> >>>>>> 凌晨1:12 寫道:
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> >>>>>> motivation for
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> >>>> partitions
> >>>>>>>>>> without
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> >>>> partition
> >>>>>> (hot)
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> >> consumer
> >>>> has
> >>>>>> never
> >>>>>>>>>>>>> seen
> >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
> >> hot
> >>>>>> partition
> >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> >> also
> >>>>>>>>>>> insufficient
> >>>>>>>>>>>>>>>>>>> because:
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
> >> varies
> >>>> across
> >>>>>>>>>> nodes
> >>>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> >> large
> >>>>>>>>>> duration,
> >>>>>>>>>>>>>>>>>>> causing
> >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
> >> seek
> >>>> time
> >>>>>> on
> >>>>>>>>>>>>> retry,
> >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
> >> data
> >>>>>> loss."
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
> >> these
> >>>> issues
> >>>>>>>>>>>>> persist,
> >>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> >> situations
> >>>>>> don't
> >>>>>>>>>>>>> need a
> >>>>>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> >> situations,
> >>>> they
> >>>>>> can
> >>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>> customized logic using
> >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> >>>> s7133700@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> >>>> should
> >>>>>> always
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> >>>>  any
> >>>>>>>>>>>>>> field
> >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
> >> version
> >>>>>> mismatch
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
> >> occurs
> >>>> inside
> >>>>>>>>>>>>> poll().
> >>>>>>>>>>>>>>>>>>>>> Therefore, the
> >>>>>>>>>>>>>>>>>>>>> user would encounter an
> >> UnsupportedVersionException
> >>>> from
> >>>>>>>>>> poll().
> >>>>>>>>>>>>>> I’ll
> >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> >> 2026年5月17日
> >>>>>> 下午4:50 寫道:
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> >> this
> >>>>>>>>>> partition
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> >> as
> >>>>>>>>>>>>>>>>> broker_current_time
> >>>>>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> >>>> does not
> >>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> >>>> should
> >>>>>> always
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> >>>>  any
> >>>>>>>>>>>>>> field
> >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> >> when
> >>>>>> calling
> >>>>>>>>>>>>>>>>> `poll()`?
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> >> based on
> >>>> the
> >>>>>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>>>>>>>>> feedback.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> >> reset
> >>>>>>>>>> behavior
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to
> >> all
> >>>>>>>>>>>>>> auto.offset.reset
> >>>>>>>>>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> >> resets to
> >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions
> >>>>>>>>>>>>>>>>>>>>>>> config setting
> >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> >>>>>>>>>> auto.offset.reset
> >>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>> continues
> >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> >> represented
> >>>> by a
> >>>>>>>>>>> separate
> >>>>>>>>>>>>>>>>>>>>> internal config
> >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently
> >> fixed
> >>>> to
> >>>>>>>>>>>>> earliest).
> >>>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>>>>> decoupled design makes
> >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> >>>> public
> >>>>>>>>>>>>>> user-facing
> >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> >> 2026年5月16日
> >>>>>> 清晨7:46
> >>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> >> is
> >>>> listed
> >>>>>>>>>>> below:
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
> >>>> default
> >>>>>>>>>> value
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> >>>> by_duration and
> >>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> does not change the logic of
> >>>> auto.offset.reset=earliest.
> >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> >>>>>> internal
> >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> >>>> .partitions=earliest
> >>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> >> cases
> >>>> of
> >>>>>> other
> >>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>>> in a
> >>>>>>>>>>>>>>>>>>>>> separate KIP.
> >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> >> .partitions
> >>>> and
> >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be
> >> applied
> >>>> to
> >>>>>> all
> >>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> consistency.
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> >> "When a
> >>>>>> Kafka
> >>>>>>>>>>>>> topic
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> expanded
> >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> >> latest
> >>>> auto
> >>>>>>>>>> offset
> >>>>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
> >> those
> >>>>>>>>>> partitions
> >>>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> >>>> record
> >>>>>> loss
> >>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> >>>> .partitions
> >>>>>> will
> >>>>>>>>>>> take
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> >> set
> >>>> it
> >>>>>>>>>>>>>> by_duration if
> >>>>>>>>>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> >> Tsai <
> >>>>>>>>>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> >>>> something
> >>>>>>>>>>>>>> important
> >>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> >> to
> >>>>>> clarify
> >>>>>>>>>> with
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>> few
> >>>>>>>>>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> >> this
> >>>>>> logic to
> >>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> >> earliest
> >>>> or
> >>>>>>>>>>>>> by_duration
> >>>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> >> issue
> >>>> when a
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> expanded.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> >>>> configure for
> >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they
> >> set
> >>>> it to
> >>>>>>>>>>>>> earliest
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases.
> >> For
> >>>>>> example,
> >>>>>>>>>>>>> if a
> >>>>>>>>>>>>>>>>>>>>> consumer is
> >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> >>>> created
> >>>>>> during
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> downtime,
> >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> >> latest
> >>>> when
> >>>>>>>>>>>>> resuming,
> >>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> >> partition
> >>>> is
> >>>>>>>>>>>>>> technically
> >>>>>>>>>>>>>>>>>>>>> "new" to
> >>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing
> >> a
> >>>>>> max.age
> >>>>>>>>>>>>>> threshold.
> >>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>> gives
> >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> >>>> genuinely
> >>>>>>>>>>> "hot/new"
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> what is
> >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> >> bit
> >>>>>> weird. It
> >>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>> applies when
> >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> >> seems
> >>>> that
> >>>>>> the
> >>>>>>>>>>>>>>>>> motivation
> >>>>>>>>>>>>>>>>>>>>>>>>>> equally
> >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> >> other
> >>>>>> values
> >>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>> by_duration.
> >>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> >>>> way to
> >>>>>>>>>>> control
> >>>>>>>>>>>>>>>>> newly
> >>>>>>>>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> >> group
> >>>>>> starts.
> >>>>>>>>>>>>>> Have we
> >>>>>>>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> >> auto.offset.reset.new
> >>>>>>>>>>> .partitions?
> >>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> >>>> defaults
> >>>>>> to
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> >>>>>> explicitly to
> >>>>>>>>>>>>>>>>> customize
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> >>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> >> 晚上10:37
> >>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> >> The
> >>>>>>>>>>>>>> PartitionMetadata
> >>>>>>>>>>>>>>>>>>>>> struct
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> >>>> computed
> >>>>>>>>>>>>>> server-side
> >>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> >> partition_creation_time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> >>>>>>>>>>>>>> MembershipManager
> >>>>>>>>>>>>>>>>>>>>> detects
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> >>>> metadata
> >>>>>> for
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> >> even
> >>>> if
> >>>>>> the
> >>>>>>>>>>>>> topic
> >>>>>>>>>>>>>> ID
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> >>>> supported
> >>>>>>>>>>>>>>>>>>>>> MetadataResponse
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> >> time.
> >>>> If the
> >>>>>>>>>>>>>> negotiated
> >>>>>>>>>>>>>>>>>>>>>>>>>> version is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> >>>> PartitionAgeMs
> >>>>>> at
> >>>>>>>>>> all
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> >>>> back to
> >>>>>>>>>> latest
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> risking
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> >> signal.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> >>>> 2026年4月29日
> >>>>>>>>>>> 下午4:04
> >>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> >>>> 'age'
> >>>>>>>>>>>>> resolution
> >>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> >> the
> >>>>>> control
> >>>>>>>>>>>>> plane
> >>>>>>>>>>>>>>>>>>> clean.
> >>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> >> introducing
> >>>>>>>>>>> inter-broker
> >>>>>>>>>>>>>>>>> clock
> >>>>>>>>>>>>>>>>>>>>> skew.
> >>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> >> single
> >>>>>> source of
> >>>>>>>>>>>>> truth
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> >>>> should be
> >>>>>>>>>>>>>> negligible.
> >>>>>>>>>>>>>>>>>>>>> Given
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> >>>> configured in
> >>>>>>>>>>> minutes
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> hours, a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> >>>> brokers
> >>>>>>>>>> won't
> >>>>>>>>>>>>>> impact
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> >> dev@kafka.apache.org>
> >>>> 於
> >>>>>>>>>>>>> 2026年4月29日
> >>>>>>>>>>>>>>>>>>> 下午3:29
> >>>>>>>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> >>>> previous
> >>>>>>>>>>>>>> conversation
> >>>>>>>>>>>>>>>>>>> but I
> >>>>>>>>>>>>>>>>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> >> flow
> >>>> with
> >>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> >> new
> >>>>>> boolean;
> >>>>>>>>>> the
> >>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>> states
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> >> this
> >>>>>> boolean
> >>>>>>>>>> is
> >>>>>>>>>>>>>> set.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a
> >> new
> >>>>>>>>>> partition,
> >>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> >> receive
> >>>> the
> >>>>>> ages.
> >>>>>>>>>>> Is
> >>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> >> flow
> >>>> and
> >>>>>> also
> >>>>>>>>>>>>>> explain
> >>>>>>>>>>>>>>>>>>>>> how it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> >> offsets,
> >>>> fetch
> >>>>>>>>>>>>> offsets,
> >>>>>>>>>>>>>>>>>>> etc.).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> >>>> wonder
> >>>>>> if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> >> right
> >>>>>> place
> >>>>>>>>>> for
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>> given
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> >> Alternatively, it
> >>>>>> could
> >>>>>>>>>>>>> simply
> >>>>>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> >>>>>> cautious
> >>>>>>>>>>> about
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> >>>> unrelated
> >>>>>>>>>>> concepts.
> >>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>> API
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> >> revoking
> >>>>>>>>>>> partitions.
> >>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>> fact
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> >> corresponding
> >>>>>> Streams
> >>>>>>>>>> API
> >>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>> suggests
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> >> would we
> >>>> do
> >>>>>> if we
> >>>>>>>>>>>>>> want to
> >>>>>>>>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> >> Muralidhar
> >>>>>> Basani
> >>>>>>>>>>> via
> >>>>>>>>>>>>>> dev
> >>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> >> know
> >>>>>> about
> >>>>>>>>>> the
> >>>>>>>>>>>>>> gap.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> >>>>>>>>>>>>>> RequestPartitionAges
> >>>>>>>>>>>>>>>>>>>>> field.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> >>>>>> PartitionAges on
> >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> >>>>>> recommended
> >>>>>>>>>>>>> value
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on
> >> the
> >>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> >>>>>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> >>>> guess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes
> >> to
> >>>> Kafka
> >>>>>>>>>>>>> Streams,
> >>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> >>>>>>>>>>>>> auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> >> it?
> >>>> This
> >>>>>> is
> >>>>>>>>>> the
> >>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> >>>>>> mistakenly
> >>>>>>>>>>>>>> configure
> >>>>>>>>>>>>>>>>>>>>> it. Or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> >> consumer
> >>>> falls
> >>>>>>>>>> back
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> earliest"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> >>>> per-partition
> >>>>>>>>>> which
> >>>>>>>>>>>>>> isn't
> >>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> >> "consumer
> >>>>>> resolves
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> initial
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> >>>>>> earliest was
> >>>>>>>>>>>>>> applied
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> >>>> config is
> >>>>>>>>>>>>>> unchanged.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> >>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> >>>> change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> >> 於
> >>>>>>>>>> 2026年4月28日
> >>>>>>>>>>>>>> 晚上8:03
> >>>>>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> >> partition
> >>>>>> creation
> >>>>>>>>>>> time
> >>>>>>>>>>>>>> via
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> >> users to
> >>>>>>>>>> diagnose
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> >>>> auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> >> on
> >>>>>> KIP-1327
> >>>>>>>>>>>>>> Prevent
> >>>>>>>>>>>>>>>>> Hot
> >>>>>>>>>>>>>>>>>>>>> Data
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>> 
> >>>> 
> >> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> >> newly
> >>>>>> expanded
> >>>>>>>>>>>>> (hot)
> >>>>>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> >> threshold
> >>>>>>>>>>> automatically
> >>>>>>>>>>>>>> fall
> >>>>>>>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> >> forcing a
> >>>>>> full
> >>>>>>>>>>>>>> historical
> >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>> 
> >>>> 
> >>> 
> >> 
> 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hi Jun, Chia-Ping,

I'd like to first explain why I believe changing the default is worthwhile. 

Documentation can only help users who already know about the problem. 
Because the failure mode is silent, most users discover it only after data has 
already been missed. A safer default such as by_duration:PT5S protects users
automatically, without requiring prior awareness of the issue. 

Given that, the question is not whether the issue exists, but how best to introduce 
a safer default.

Approach A — Follow a deprecation cycle and land the change in 5.0 
- Keep the current default (`latest`) in upcoming 4.x releases, but emit a warning that 
the default will change to `by_duration:PT5S` in 5.0 and explain how users can retain 
the current behavior explicitly. 
- Change the default to `by_duration:PT5S` in 5.0. 
- Update shell tools (for example, the console consumer) to explicitly use `latest`, preserving 
the expected "show only new messages" behavior.

Approach B — Change the default in a 4.4 minor release 
- Change the default to `by_duration:PT5S` in a 4.x minor release. 
- Leave shell tools unchanged. 
- Emit a warning explaining the change and how users can explicitly restore `latest`.

I favor Approach B: 

Silent data loss during partition expansion is a correctness issue, not a usability issue. 
Delaying the safer default until 5.0 leaves users relying on the implicit default exposed unnecessarily. 
Users who need latest can simply pin it explicitly. 
I don't think shell tools need special handling. by_duration:PT5S only replays a few seconds of data, 
which is negligible in practice and preferable to silently missing records. Keeping the behavior consistent 
is simpler.

Best Regards,
Jiunn-Yang

> Jun Rao via dev <de...@kafka.apache.org> 於 2026年7月7日 清晨5:49 寫道:
> 
> Hi, Chia-Ping,
> 
> I am not sure that we need an out-of-the-box solution. The proposal in the
> KIP is also not out-of-the-box and requires users to opt-in.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> 
>> hi Jun
>> 
>> Yes, updating the docs requires minimal changes to the codebase, but it is
>> not an out-of-the-box solution. From my perspective, this issue can be
>> addressed by setting the console consumer's policy back to latest.
>> 
>> Let's see what Jiunn thinks
>> 
>> Best,
>> Chia-Ping
>> 
>> On 2026/07/06 15:59:37 Jun Rao via dev wrote:
>>> Hi, Chia-Ping,
>>> 
>>> One potential downside of changing the default is that some existing
>> tools
>>> (e.g. console consumer) may depend on this behavior. For example, a user
>>> testing the system may expect to see only new messages and could be
>>> surprised by the change. Another option is to leave the default
>> unchanged,
>>> but document clearly how to prevent losing messages on new partitions
>>> through by_duration. The users can then make the corresponding config
>>> changes if needed.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org>
>> wrote:
>>> 
>>>> Hi Jun,
>>>> 
>>>> Thanks for the feedback. I agree with your conclusion. It would be
>> better
>>>> to "make haste" with a new KIP to address this. We can focus the
>> discussion
>>>> on the following points:
>>>> 
>>>> 1) Should we change the default reset policy to by_duration for all
>>>> consumers?
>>>> 2) What is the optimal duration to prevent data loss while avoiding
>> large
>>>> backlogs? Note that the classic consumer has a 3-second heartbeat
>> interval,
>>>> whereas the async consumer uses 5 seconds.
>>>> 3) What are the potential side effects of replacing latest with
>>>> by_duration as the default behavior?
>>>> 
>>>> Best,
>>>> Chia-Ping
>>>> 
>>>> On 2026/06/30 20:39:24 Jun Rao via dev wrote:
>>>>> Hi, Chia-Ping,
>>>>> 
>>>>> I agree that using by_duration with a value of 5s to 10s addresses
>> most
>>>>> common issues with expanded partitions, given the default
>>>>> heartbeat.interval.ms is 3 seconds. We can consider setting that as
>> the
>>>> new
>>>>> default. The current proposal may address some other rarer cases,
>> but it
>>>>> feels a bit like over engineering.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jun
>>>>> 
>>>>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org>
>>>> wrote:
>>>>> 
>>>>>> - Short durations fail during extended downtimes, leading to the
>> exact
>>>>>> silent data
>>>>>> loss this KIP aims to prevent.
>>>>>> 
>>>>>> Losing the records produced to expanded partitions during a
>> downtime is
>>>>>> actually the expected behavior for the latest policy. This is
>> exactly
>>>> why,
>>>>>> in the current KIP proposal, we prefer by_duration over earliest
>> for
>>>> the
>>>>>> expanded partitions—we want to avoid forcing users to consume a
>> massive
>>>>>> accumulated backlog just because a partition is technically "new".
>>>>>> 
>>>>>> To quote my previous comment from May 15:
>>>>>> "...if a consumer is offline for a while and a new partition is
>> created
>>>>>> during that downtime, the user might actually want to skip to
>> latest
>>>> when
>>>>>> resuming, rather than reading from earliest just because the
>> partition
>>>> is
>>>>>> technically 'new' to the group."
>>>>>> 
>>>>>> On 2026/06/30 12:12:47 黃竣陽 wrote:
>>>>>>> Hello all,
>>>>>>> 
>>>>>>> The metadata blindness and consumer downtime windows are
>> inherently
>>>>>>> unpredictable, making a fixed by_duration approach unviable. No
>>>> static
>>>>>>> duration can accommodate both extremes:
>>>>>>> 
>>>>>>> - Short durations fail during extended downtimes, leading to the
>>>> exact
>>>>>> silent data
>>>>>>> loss this KIP aims to prevent.
>>>>>>> - Long durations (calibrated for worst-case scenarios) cause
>> routine
>>>>>> restarts to
>>>>>>> trigger unnecessary mass replays, undermining the latest offset
>> reset
>>>>>> policy.
>>>>>>> 
>>>>>>> This is why the proposed solution uses creation time. It
>> provides a
>>>>>> discrete,
>>>>>>> deterministic signal—verifying whether a partition is actually
>> newer
>>>>>> than the group,
>>>>>>> rather than relying on an arbitrary time window that has no
>>>> correlation
>>>>>> with the consumer’s
>>>>>>> actual downtime.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> I'd like to revisit Jun previous comment about using
>>>> by_duration=5s. I
>>>>>> realize now that having a small backlog is actually an acceptable
>>>>>> trade-off, especially since the latest policy inherently brings a
>> tiny
>>>>>> backlog to starting consumers anyway due to the time gap between
>>>> fetching
>>>>>> the offset and actually consuming.
>>>>>>>> 
>>>>>>>> If we bring the simple by_duration=5s solution back to the
>> KIP, I'd
>>>>>> like to discuss a further step: Should we just change the default
>>>> policy
>>>>>> from latest to by_duration=5s specifically for the new consumer and
>>>> share
>>>>>> consumer?
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>> 
>>>>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
>>>>>>>>> hi Jun
>>>>>>>>> 
>>>>>>>>> The most important part of this story is how users should
>> expect
>>>> the
>>>>>> data
>>>>>>>>> they can see when using the latest or by_duration policy with
>>>> expanded
>>>>>>>>> partitions.
>>>>>>>>> 
>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
>>>>>>>>> non-deterministic, which means users will either read too many
>>>>>> historical
>>>>>>>>> records from existing partitions or lose some records from
>>>> expanded
>>>>>>>>> partitions.
>>>>>>>>> 
>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit
>> hard to
>>>>>>>>> understand, and that is why I preferred having a whole new
>> policy
>>>>>> based
>>>>>>>>> entirely on group creation time (KIP-1282)
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
>> 上午1:08寫道:
>>>>>>>>> 
>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply. I am still trying to understand the
>> value
>>>> of
>>>>>> the new
>>>>>>>>>> configs with the KIP.
>>>>>>>>>> 
>>>>>>>>>> The motivation of the KIP is that a user doesn't want to
>> miss the
>>>>>> data if
>>>>>>>>>> the backlog is small. The backlog of the existing partition
>> is
>>>> easy
>>>>>> to
>>>>>>>>>> understand because it relates to retention time. The backlog
>> for
>>>> the
>>>>>> new
>>>>>>>>>> partition is a bit subtle to understand since it depends on
>> the
>>>>>> metadata
>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user
>>>> needs
>>>>>> to
>>>>>>>>>> understand the metadata refresh delay on the consumer side
>> and
>>>> use
>>>>>> it to
>>>>>>>>>> set the config.
>>>>>>>>>> 
>>>>>>>>>> Now, let's consider the alternative: setting the same value
>> for
>>>> the
>>>>>>>>>> existing by_duration policy. The KIP lists three issues with
>> this
>>>>>> approach.
>>>>>>>>>> 1. It computes the seek target client-side as now() -
>> duration,
>>>> which
>>>>>>>>>> introduces clock skew across consumers and forces operators
>> to
>>>> choose
>>>>>>>>>> overly large durations, causing unnecessary reprocessing.
>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
>> failed
>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
>>>>>> potentially
>>>>>>>>>> miss records produced between attempts.
>>>>>>>>>> 3. It applies uniformly to all partitions without committed
>>>> offsets,
>>>>>> and
>>>>>>>>>> cannot distinguish newly expanded partitions from
>> long-existing
>>>>>> partitions
>>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
>>>>>>>>>> 
>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a
>> bit
>>>>>> buffer to
>>>>>>>>>> the metadata refresh delay. We could also consider improving
>> the
>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
>>>> typically
>>>>>> low
>>>>>>>>>> (in the order of minutes with the classic consumer and tens
>> of
>>>>>> seconds with
>>>>>>>>>> the new consumer). If a user is ok with reading that much
>> backlog
>>>>>> for new
>>>>>>>>>> partitions, it seems they will be ok doing the same for
>> existing
>>>>>>>>>> partitions.
>>>>>>>>>> 
>>>>>>>>>> So, instead of introducing a new config, could we just reuse
>> the
>>>>>> existing
>>>>>>>>>> config with better documentation and/or implementation?
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello Jun,
>>>>>>>>>>> 
>>>>>>>>>>> You're right that group creation time is the more intuitive
>>>> answer
>>>>>> at
>>>>>>>>>>> first glance,
>>>>>>>>>>> the KIP's own motivation talks about partitions that
>> "predate
>>>> the
>>>>>> group"
>>>>>>>>>>> vs partitions
>>>>>>>>>>> "created during group runtime," which directly points to a
>>>>>>>>>> group-lifecycle
>>>>>>>>>>> classifier.
>>>>>>>>>>> I'd like to walk through why we landed on partition age,
>> and the
>>>>>>>>>>> trade-offs we considered.
>>>>>>>>>>> 
>>>>>>>>>>> We evaluated three candidate signals:
>>>>>>>>>>> 
>>>>>>>>>>> 1. `by_duration:5secs`
>>>>>>>>>>> 
>>>>>>>>>>> This covers the metadata blindness window, but has issues
>> the
>>>> KIP
>>>>>>>>>>> currently documents
>>>>>>>>>>> under "Why not use `by_duration`?":
>>>>>>>>>>> 
>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
>> across
>>>>>> consumers.
>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
>> potentially
>>>>>> missing
>>>>>>>>>>> records produced between
>>>>>>>>>>> attempts.
>>>>>>>>>>> - It applies uniformly to all partitions without committed
>>>> offsets,
>>>>>>>>>>> including pre-existing partitions
>>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
>>>>>>>>>>> 
>>>>>>>>>>> 2. Group creation time as classifier
>>>>>>>>>>> 
>>>>>>>>>>> This works cleanly when the consumer is actively running.
>> Our
>>>>>> concern
>>>>>>>>>>> is the idle / late-rejoin case:
>>>>>>>>>>> 
>>>>>>>>>>> T=0:         Group created.
>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>>>>>>>>> T=50:        Partition added during the idle window.
>>>>>>>>>>> T=100:       Consumer resumes.
>>>>>>>>>>> 
>>>>>>>>>>> Under group creation time, the new partition is classified
>> as
>>>> new
>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
>> from
>>>> T=50.
>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
>> data
>>>>>> that
>>>>>>>>>>> the consumer accepts as lost — that is precisely the
>> contract of
>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
>>>> treat
>>>>>>>>>>> the new partition differently; both contain backlog
>> accumulated
>>>>>> during
>>>>>>>>>>> the same idle window.
>>>>>>>>>>> 
>>>>>>>>>>> This aligns with the "backlog is backlog” principle you
>> raised
>>>> in
>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
>> backlog
>>>> on
>>>>>>>>>>> every other partition during the same idle period; forcing
>>>> 0-backlog
>>>>>>>>>>> tolerance only on new partitions would be inconsistent with
>> that
>>>>>>>>>>> tolerance.
>>>>>>>>>>> 
>>>>>>>>>>> 3. Partition age vs threshold
>>>>>>>>>>> 
>>>>>>>>>>> Partition age corresponds to the actual silent data loss
>> window,
>>>>>>>>>>> the gap between partition creation and the consumer’s
>> metadata
>>>>>>>>>>> refresh. Within this window, data loss is genuinely silent:
>> the
>>>>>>>>>>> consumer had no opportunity to know about the partition.
>> Outside
>>>>>> this
>>>>>>>>>>> window, missing data reflects either:
>>>>>>>>>>> 
>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
>> consumers,
>>>> or
>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
>> via
>>>> reset
>>>>>>>>>> policy.
>>>>>>>>>>> 
>>>>>>>>>>> We did not choose partition age because it is more elegant
>> than
>>>>>> group
>>>>>>>>>>> creation time — we chose it because its failure mode
>> (requires a
>>>>>>>>>>> threshold) is
>>>>>>>>>>> less invasive than the failure mode of group creation time
>>>>>> (overrides
>>>>>>>>>>> user-stated
>>>>>>>>>>> `latest` intent during idle periods).
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
>> 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>> 
>>>>>>>>>>>> Relying on both creation times will create an inconsistent
>>>>>> scenario. A
>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
>> seek
>>>> to the
>>>>>>>>>>>> beginning for the partitions created later than the group.
>>>>>>>>>>>> 
>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
>>>> inconsistency
>>>>>>>>>>> using a
>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
>> consensus,
>>>>>> KIP-1327
>>>>>>>>>>> goes
>>>>>>>>>>>> back to using flexible configurations to prevent users from
>>>> falling
>>>>>>>>>> into
>>>>>>>>>>>> that pitfall.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
>>>> 上午6:49寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still
>> feels
>>>>>> weird to
>>>>>>>>>>> me.
>>>>>>>>>>>>> It
>>>>>>>>>>>>> categorizes partitions as new or existing based on the
>>>> partition
>>>>>>>>>>> creation
>>>>>>>>>>>>> time. Intuitively, the categorization should be based on
>> the
>>>> group
>>>>>>>>>>> creation
>>>>>>>>>>>>> time: all partitions existing when the group is created
>> are
>>>>>> existing
>>>>>>>>>> and
>>>>>>>>>>>>> all partitions created after the group creation are new
>>>>>> partitions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Manually bumping this thread. If there is no further
>>>>>>>>>>>>>> discussion, I will close the vote.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Jian,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for your feedback,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
>> task,
>>>> not an
>>>>>>>>>> edge
>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
>> clarify
>>>> that
>>>>>> this
>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
>>>>>>>>>> demonstrates
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> need for this optimization. For example:
>>>>>>>>>>>>>>>> A partition expansion is a common operational task in
>>>> Kafka: To
>>>>>>>>>>>>> balance
>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
>> created
>>>>>> with a
>>>>>>>>>>>>>> moderate
>>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
>>>> time,
>>>>>> it is
>>>>>>>>>>>>>> often
>>>>>>>>>>>>>>>> necessary to increase the number of partitions to
>>>> accommodate
>>>>>> the
>>>>>>>>>>>>> higher
>>>>>>>>>>>>>>>> workload.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>> Jian
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
>> 晚上8:29
>>>> 寫道:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
>> when
>>>>>>>>>> describing
>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
>> them to
>>>>>>>>>> describe
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
>>>> applying
>>>>>>>>>> these
>>>>>>>>>>>>>> terms
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
>>>> whether
>>>>>> a
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
>>>> they
>>>>>> won't
>>>>>>>>>>>>>> silently
>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
>> their
>>>>>> active
>>>>>>>>>>>>>>>>> consumption.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
>>>> motivation
>>>>>>>>>>> section.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
>> 2026年5月30日
>>>>>> 凌晨1:12 寫道:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
>>>>>> motivation for
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
>>>> partitions
>>>>>>>>>> without
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
>>>> partition
>>>>>> (hot)
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
>> consumer
>>>> has
>>>>>> never
>>>>>>>>>>>>> seen
>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
>> hot
>>>>>> partition
>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
>> also
>>>>>>>>>>> insufficient
>>>>>>>>>>>>>>>>>>> because:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
>> varies
>>>> across
>>>>>>>>>> nodes
>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
>> large
>>>>>>>>>> duration,
>>>>>>>>>>>>>>>>>>> causing
>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
>> seek
>>>> time
>>>>>> on
>>>>>>>>>>>>> retry,
>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
>> data
>>>>>> loss."
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
>> these
>>>> issues
>>>>>>>>>>>>> persist,
>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
>> situations
>>>>>> don't
>>>>>>>>>>>>> need a
>>>>>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
>> situations,
>>>> they
>>>>>> can
>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>>>>> customized logic using
>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
>>>> s7133700@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
>>>> should
>>>>>> always
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>>>>  any
>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
>> version
>>>>>> mismatch
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
>> occurs
>>>> inside
>>>>>>>>>>>>> poll().
>>>>>>>>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>>>>>>>>> user would encounter an
>> UnsupportedVersionException
>>>> from
>>>>>>>>>> poll().
>>>>>>>>>>>>>> I’ll
>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>> 2026年5月17日
>>>>>> 下午4:50 寫道:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
>> this
>>>>>>>>>> partition
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
>> as
>>>>>>>>>>>>>>>>> broker_current_time
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
>>>> does not
>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
>>>> should
>>>>>> always
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>>>>  any
>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
>> when
>>>>>> calling
>>>>>>>>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
>> based on
>>>> the
>>>>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
>> reset
>>>>>>>>>> behavior
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to
>> all
>>>>>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
>> resets to
>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions
>>>>>>>>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
>> represented
>>>> by a
>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently
>> fixed
>>>> to
>>>>>>>>>>>>> earliest).
>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
>>>> public
>>>>>>>>>>>>>> user-facing
>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>> 2026年5月16日
>>>>>> 清晨7:46
>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
>> is
>>>> listed
>>>>>>>>>>> below:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
>>>> default
>>>>>>>>>> value
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
>>>> by_duration and
>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> does not change the logic of
>>>> auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
>>>>>> internal
>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
>>>> .partitions=earliest
>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
>> cases
>>>> of
>>>>>> other
>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>> .partitions
>>>> and
>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be
>> applied
>>>> to
>>>>>> all
>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
>> "When a
>>>>>> Kafka
>>>>>>>>>>>>> topic
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
>> latest
>>>> auto
>>>>>>>>>> offset
>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
>> those
>>>>>>>>>> partitions
>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
>>>> record
>>>>>> loss
>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>>>> .partitions
>>>>>> will
>>>>>>>>>>> take
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
>> set
>>>> it
>>>>>>>>>>>>>> by_duration if
>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
>> Tsai <
>>>>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
>>>> something
>>>>>>>>>>>>>> important
>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
>> to
>>>>>> clarify
>>>>>>>>>> with
>>>>>>>>>>>>> a
>>>>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
>> this
>>>>>> logic to
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
>> earliest
>>>> or
>>>>>>>>>>>>> by_duration
>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
>> issue
>>>> when a
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
>>>> configure for
>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they
>> set
>>>> it to
>>>>>>>>>>>>> earliest
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases.
>> For
>>>>>> example,
>>>>>>>>>>>>> if a
>>>>>>>>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
>>>> created
>>>>>> during
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
>> latest
>>>> when
>>>>>>>>>>>>> resuming,
>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
>> partition
>>>> is
>>>>>>>>>>>>>> technically
>>>>>>>>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing
>> a
>>>>>> max.age
>>>>>>>>>>>>>> threshold.
>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
>>>> genuinely
>>>>>>>>>>> "hot/new"
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
>> bit
>>>>>> weird. It
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
>> seems
>>>> that
>>>>>> the
>>>>>>>>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
>> other
>>>>>> values
>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
>>>> way to
>>>>>>>>>>> control
>>>>>>>>>>>>>>>>> newly
>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
>> group
>>>>>> starts.
>>>>>>>>>>>>>> Have we
>>>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
>> auto.offset.reset.new
>>>>>>>>>>> .partitions?
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
>>>> defaults
>>>>>> to
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
>>>>>> explicitly to
>>>>>>>>>>>>>>>>> customize
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
>> 晚上10:37
>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
>> The
>>>>>>>>>>>>>> PartitionMetadata
>>>>>>>>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
>>>> computed
>>>>>>>>>>>>>> server-side
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
>> partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>>>>>>>>> MembershipManager
>>>>>>>>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
>>>> metadata
>>>>>> for
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
>> even
>>>> if
>>>>>> the
>>>>>>>>>>>>> topic
>>>>>>>>>>>>>> ID
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
>>>> supported
>>>>>>>>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
>> time.
>>>> If the
>>>>>>>>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
>>>> PartitionAgeMs
>>>>>> at
>>>>>>>>>> all
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
>>>> back to
>>>>>>>>>> latest
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
>> signal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>>>> 2026年4月29日
>>>>>>>>>>> 下午4:04
>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
>>>> 'age'
>>>>>>>>>>>>> resolution
>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
>> the
>>>>>> control
>>>>>>>>>>>>> plane
>>>>>>>>>>>>>>>>>>> clean.
>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
>> introducing
>>>>>>>>>>> inter-broker
>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
>> single
>>>>>> source of
>>>>>>>>>>>>> truth
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
>>>> should be
>>>>>>>>>>>>>> negligible.
>>>>>>>>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
>>>> configured in
>>>>>>>>>>> minutes
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
>>>> brokers
>>>>>>>>>> won't
>>>>>>>>>>>>>> impact
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
>> dev@kafka.apache.org>
>>>> 於
>>>>>>>>>>>>> 2026年4月29日
>>>>>>>>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
>>>> previous
>>>>>>>>>>>>>> conversation
>>>>>>>>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
>> flow
>>>> with
>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
>> new
>>>>>> boolean;
>>>>>>>>>> the
>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
>> this
>>>>>> boolean
>>>>>>>>>> is
>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a
>> new
>>>>>>>>>> partition,
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
>> receive
>>>> the
>>>>>> ages.
>>>>>>>>>>> Is
>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
>> flow
>>>> and
>>>>>> also
>>>>>>>>>>>>>> explain
>>>>>>>>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
>> offsets,
>>>> fetch
>>>>>>>>>>>>> offsets,
>>>>>>>>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
>>>> wonder
>>>>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
>> right
>>>>>> place
>>>>>>>>>> for
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
>> Alternatively, it
>>>>>> could
>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
>>>>>> cautious
>>>>>>>>>>> about
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
>>>> unrelated
>>>>>>>>>>> concepts.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
>> revoking
>>>>>>>>>>> partitions.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
>> corresponding
>>>>>> Streams
>>>>>>>>>> API
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
>> would we
>>>> do
>>>>>> if we
>>>>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
>> Muralidhar
>>>>>> Basani
>>>>>>>>>>> via
>>>>>>>>>>>>>> dev
>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
>> know
>>>>>> about
>>>>>>>>>> the
>>>>>>>>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>>>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
>>>>>> PartitionAges on
>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
>>>>>> recommended
>>>>>>>>>>>>> value
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on
>> the
>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
>>>> guess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes
>> to
>>>> Kafka
>>>>>>>>>>>>> Streams,
>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>>>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
>> it?
>>>> This
>>>>>> is
>>>>>>>>>> the
>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
>>>>>> mistakenly
>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
>> consumer
>>>> falls
>>>>>>>>>> back
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
>>>> per-partition
>>>>>>>>>> which
>>>>>>>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
>> "consumer
>>>>>> resolves
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
>>>>>> earliest was
>>>>>>>>>>>>>> applied
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
>>>> config is
>>>>>>>>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
>>>> change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
>> 於
>>>>>>>>>> 2026年4月28日
>>>>>>>>>>>>>> 晚上8:03
>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
>> partition
>>>>>> creation
>>>>>>>>>>> time
>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
>> users to
>>>>>>>>>> diagnose
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
>> on
>>>>>> KIP-1327
>>>>>>>>>>>>>> Prevent
>>>>>>>>>>>>>>>>> Hot
>>>>>>>>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>> 
>>>> 
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
>> newly
>>>>>> expanded
>>>>>>>>>>>>> (hot)
>>>>>>>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
>> threshold
>>>>>>>>>>> automatically
>>>>>>>>>>>>>> fall
>>>>>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
>> forcing a
>>>>>> full
>>>>>>>>>>>>>> historical
>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping,

I am not sure that we need an out-of-the-box solution. The proposal in the
KIP is also not out-of-the-box and requires users to opt-in.

Thanks,

Jun

On Mon, Jul 6, 2026 at 11:34 AM Chia-Ping Tsai <ch...@apache.org> wrote:

> hi Jun
>
> Yes, updating the docs requires minimal changes to the codebase, but it is
> not an out-of-the-box solution. From my perspective, this issue can be
> addressed by setting the console consumer's policy back to latest.
>
> Let's see what Jiunn thinks
>
> Best,
> Chia-Ping
>
> On 2026/07/06 15:59:37 Jun Rao via dev wrote:
> > Hi, Chia-Ping,
> >
> > One potential downside of changing the default is that some existing
> tools
> > (e.g. console consumer) may depend on this behavior. For example, a user
> > testing the system may expect to see only new messages and could be
> > surprised by the change. Another option is to leave the default
> unchanged,
> > but document clearly how to prevent losing messages on new partitions
> > through by_duration. The users can then make the corresponding config
> > changes if needed.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >
> > > Hi Jun,
> > >
> > > Thanks for the feedback. I agree with your conclusion. It would be
> better
> > > to "make haste" with a new KIP to address this. We can focus the
> discussion
> > > on the following points:
> > >
> > > 1) Should we change the default reset policy to by_duration for all
> > > consumers?
> > > 2) What is the optimal duration to prevent data loss while avoiding
> large
> > > backlogs? Note that the classic consumer has a 3-second heartbeat
> interval,
> > > whereas the async consumer uses 5 seconds.
> > > 3) What are the potential side effects of replacing latest with
> > > by_duration as the default behavior?
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> > > > Hi, Chia-Ping,
> > > >
> > > > I agree that using by_duration with a value of 5s to 10s addresses
> most
> > > > common issues with expanded partitions, given the default
> > > > heartbeat.interval.ms is 3 seconds. We can consider setting that as
> the
> > > new
> > > > default. The current proposal may address some other rarer cases,
> but it
> > > > feels a bit like over engineering.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org>
> > > wrote:
> > > >
> > > > > - Short durations fail during extended downtimes, leading to the
> exact
> > > > > silent data
> > > > > loss this KIP aims to prevent.
> > > > >
> > > > > Losing the records produced to expanded partitions during a
> downtime is
> > > > > actually the expected behavior for the latest policy. This is
> exactly
> > > why,
> > > > > in the current KIP proposal, we prefer by_duration over earliest
> for
> > > the
> > > > > expanded partitions—we want to avoid forcing users to consume a
> massive
> > > > > accumulated backlog just because a partition is technically "new".
> > > > >
> > > > > To quote my previous comment from May 15:
> > > > > "...if a consumer is offline for a while and a new partition is
> created
> > > > > during that downtime, the user might actually want to skip to
> latest
> > > when
> > > > > resuming, rather than reading from earliest just because the
> partition
> > > is
> > > > > technically 'new' to the group."
> > > > >
> > > > > On 2026/06/30 12:12:47 黃竣陽 wrote:
> > > > > > Hello all,
> > > > > >
> > > > > > The metadata blindness and consumer downtime windows are
> inherently
> > > > > > unpredictable, making a fixed by_duration approach unviable. No
> > > static
> > > > > > duration can accommodate both extremes:
> > > > > >
> > > > > > - Short durations fail during extended downtimes, leading to the
> > > exact
> > > > > silent data
> > > > > > loss this KIP aims to prevent.
> > > > > > - Long durations (calibrated for worst-case scenarios) cause
> routine
> > > > > restarts to
> > > > > > trigger unnecessary mass replays, undermining the latest offset
> reset
> > > > > policy.
> > > > > >
> > > > > > This is why the proposed solution uses creation time. It
> provides a
> > > > > discrete,
> > > > > > deterministic signal—verifying whether a partition is actually
> newer
> > > > > than the group,
> > > > > > rather than relying on an arbitrary time window that has no
> > > correlation
> > > > > with the consumer’s
> > > > > > actual downtime.
> > > > > >
> > > > > > Best Regards,
> > > > > > Jiunn-Yang
> > > > > >
> > > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to revisit Jun previous comment about using
> > > by_duration=5s. I
> > > > > realize now that having a small backlog is actually an acceptable
> > > > > trade-off, especially since the latest policy inherently brings a
> tiny
> > > > > backlog to starting consumers anyway due to the time gap between
> > > fetching
> > > > > the offset and actually consuming.
> > > > > > >
> > > > > > > If we bring the simple by_duration=5s solution back to the
> KIP, I'd
> > > > > like to discuss a further step: Should we just change the default
> > > policy
> > > > > from latest to by_duration=5s specifically for the new consumer and
> > > share
> > > > > consumer?
> > > > > > >
> > > > > > > Best,
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > > > > > >> hi Jun
> > > > > > >>
> > > > > > >> The most important part of this story is how users should
> expect
> > > the
> > > > > data
> > > > > > >> they can see when using the latest or by_duration policy with
> > > expanded
> > > > > > >> partitions.
> > > > > > >>
> > > > > > >> Yes, the by_duration policy can minimize data loss, but it is
> > > > > > >> non-deterministic, which means users will either read too many
> > > > > historical
> > > > > > >> records from existing partitions or lose some records from
> > > expanded
> > > > > > >> partitions.
> > > > > > >>
> > > > > > >> Also, I agree that auto.offset.reset.max.age.ms is a bit
> hard to
> > > > > > >> understand, and that is why I preferred having a whole new
> policy
> > > > > based
> > > > > > >> entirely on group creation time (KIP-1282)
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Chia-Ping
> > > > > > >>
> > > > > > >> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> 上午1:08寫道:
> > > > > > >>
> > > > > > >>> Hi, Chia-Ping and Jiunn-Yang,
> > > > > > >>>
> > > > > > >>> Thanks for the reply. I am still trying to understand the
> value
> > > of
> > > > > the new
> > > > > > >>> configs with the KIP.
> > > > > > >>>
> > > > > > >>> The motivation of the KIP is that a user doesn't want to
> miss the
> > > > > data if
> > > > > > >>> the backlog is small. The backlog of the existing partition
> is
> > > easy
> > > > > to
> > > > > > >>> understand because it relates to retention time. The backlog
> for
> > > the
> > > > > new
> > > > > > >>> partition is a bit subtle to understand since it depends on
> the
> > > > > metadata
> > > > > > >>> refresh delay. To set auto.offset.reset.max.age.ms, the user
> > > needs
> > > > > to
> > > > > > >>> understand the metadata refresh delay on the consumer side
> and
> > > use
> > > > > it to
> > > > > > >>> set the config.
> > > > > > >>>
> > > > > > >>> Now, let's consider the alternative: setting the same value
> for
> > > the
> > > > > > >>> existing by_duration policy. The KIP lists three issues with
> this
> > > > > approach.
> > > > > > >>> 1. It computes the seek target client-side as now() -
> duration,
> > > which
> > > > > > >>> introduces clock skew across consumers and forces operators
> to
> > > choose
> > > > > > >>> overly large durations, causing unnecessary reprocessing.
> > > > > > >>> 2. The target timestamp is recomputed on each retry, so
> failed
> > > > > > >>> ListOffsetsRequest retries can shift the target forward and
> > > > > potentially
> > > > > > >>> miss records produced between attempts.
> > > > > > >>> 3. It applies uniformly to all partitions without committed
> > > offsets,
> > > > > and
> > > > > > >>> cannot distinguish newly expanded partitions from
> long-existing
> > > > > partitions
> > > > > > >>> newly assigned to the group, leading to unnecessary replay.
> > > > > > >>>
> > > > > > >>> Issues 1 and 2 are uncommon and can be mitigated by adding a
> bit
> > > > > buffer to
> > > > > > >>> the metadata refresh delay. We could also consider improving
> the
> > > > > > >>> implementation. For issue 3, the metadata refresh delay is
> > > typically
> > > > > low
> > > > > > >>> (in the order of minutes with the classic consumer and tens
> of
> > > > > seconds with
> > > > > > >>> the new consumer). If a user is ok with reading that much
> backlog
> > > > > for new
> > > > > > >>> partitions, it seems they will be ok doing the same for
> existing
> > > > > > >>> partitions.
> > > > > > >>>
> > > > > > >>> So, instead of introducing a new config, could we just reuse
> the
> > > > > existing
> > > > > > >>> config with better documentation and/or implementation?
> > > > > > >>>
> > > > > > >>> Jun
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > > > > > >>>
> > > > > > >>>> Hello Jun,
> > > > > > >>>>
> > > > > > >>>> You're right that group creation time is the more intuitive
> > > answer
> > > > > at
> > > > > > >>>> first glance,
> > > > > > >>>> the KIP's own motivation talks about partitions that
> "predate
> > > the
> > > > > group"
> > > > > > >>>> vs partitions
> > > > > > >>>> "created during group runtime," which directly points to a
> > > > > > >>> group-lifecycle
> > > > > > >>>> classifier.
> > > > > > >>>> I'd like to walk through why we landed on partition age,
> and the
> > > > > > >>>> trade-offs we considered.
> > > > > > >>>>
> > > > > > >>>> We evaluated three candidate signals:
> > > > > > >>>>
> > > > > > >>>> 1. `by_duration:5secs`
> > > > > > >>>>
> > > > > > >>>> This covers the metadata blindness window, but has issues
> the
> > > KIP
> > > > > > >>>> currently documents
> > > > > > >>>> under "Why not use `by_duration`?":
> > > > > > >>>>
> > > > > > >>>> - Client-side `now() - duration` introduces clock skew
> across
> > > > > consumers.
> > > > > > >>>> - `ListOffsets` retries shift the target forward,
> potentially
> > > > > missing
> > > > > > >>>> records produced between
> > > > > > >>>> attempts.
> > > > > > >>>> - It applies uniformly to all partitions without committed
> > > offsets,
> > > > > > >>>> including pre-existing partitions
> > > > > > >>>> newly assigned to the group, causing unnecessary replay.
> > > > > > >>>>
> > > > > > >>>> 2. Group creation time as classifier
> > > > > > >>>>
> > > > > > >>>> This works cleanly when the consumer is actively running.
> Our
> > > > > concern
> > > > > > >>>> is the idle / late-rejoin case:
> > > > > > >>>>
> > > > > > >>>> T=0:         Group created.
> > > > > > >>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > > > > >>>> T=50:        Partition added during the idle window.
> > > > > > >>>> T=100:       Consumer resumes.
> > > > > > >>>>
> > > > > > >>>> Under group creation time, the new partition is classified
> as
> > > new
> > > > > > >>>> (`50 > 0`) and reset to `earliest`, replaying everything
> from
> > > T=50.
> > > > > > >>>> But during `[T=1, T=100]`, base partitions also accumulated
> data
> > > > > that
> > > > > > >>>> the consumer accepts as lost — that is precisely the
> contract of
> > > > > > >>>> `auto.offset.reset=latest`. There is no principled reason to
> > > treat
> > > > > > >>>> the new partition differently; both contain backlog
> accumulated
> > > > > during
> > > > > > >>>> the same idle window.
> > > > > > >>>>
> > > > > > >>>> This aligns with the "backlog is backlog” principle you
> raised
> > > in
> > > > > > >>>> the KIP-1282 thread: a `latest` user has tolerated some
> backlog
> > > on
> > > > > > >>>> every other partition during the same idle period; forcing
> > > 0-backlog
> > > > > > >>>> tolerance only on new partitions would be inconsistent with
> that
> > > > > > >>>> tolerance.
> > > > > > >>>>
> > > > > > >>>> 3. Partition age vs threshold
> > > > > > >>>>
> > > > > > >>>> Partition age corresponds to the actual silent data loss
> window,
> > > > > > >>>> the gap between partition creation and the consumer’s
> metadata
> > > > > > >>>> refresh. Within this window, data loss is genuinely silent:
> the
> > > > > > >>>> consumer had no opportunity to know about the partition.
> Outside
> > > > > this
> > > > > > >>>> window, missing data reflects either:
> > > > > > >>>>
> > > > > > >>>> - (a) the user’s tolerated cost of running with idle
> consumers,
> > > or
> > > > > > >>>> - (b) an operational issue to surface via monitoring, not
> via
> > > reset
> > > > > > >>> policy.
> > > > > > >>>>
> > > > > > >>>> We did not choose partition age because it is more elegant
> than
> > > > > group
> > > > > > >>>> creation time — we chose it because its failure mode
> (requires a
> > > > > > >>>> threshold) is
> > > > > > >>>> less invasive than the failure mode of group creation time
> > > > > (overrides
> > > > > > >>>> user-stated
> > > > > > >>>> `latest` intent during idle periods).
> > > > > > >>>>
> > > > > > >>>> Best Regards,
> > > > > > >>>> Jiunn-Yang
> > > > > > >>>>
> > > > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> 寫道:
> > > > > > >>>>>
> > > > > > >>>>> Hi Jun,
> > > > > > >>>>>
> > > > > > >>>>> Relying on both creation times will create an inconsistent
> > > > > scenario. A
> > > > > > >>>>> consumer that lost all offsets due to a long sleep will
> seek
> > > to the
> > > > > > >>>>> beginning for the partitions created later than the group.
> > > > > > >>>>>
> > > > > > >>>>> That is why we initially proposed KIP-1282 to fix the
> > > inconsistency
> > > > > > >>>> using a
> > > > > > >>>>> whole new policy. Since KIP-1282 couldn't reach a
> consensus,
> > > > > KIP-1327
> > > > > > >>>> goes
> > > > > > >>>>> back to using flexible configurations to prevent users from
> > > falling
> > > > > > >>> into
> > > > > > >>>>> that pitfall.
> > > > > > >>>>>
> > > > > > >>>>> Best, Chia-Ping
> > > > > > >>>>>
> > > > > > >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > > 上午6:49寫道:
> > > > > > >>>>>
> > > > > > >>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks for the reply and sorry for the late reply.
> > > > > > >>>>>>
> > > > > > >>>>>> JR1. The design of auto.offset.reset.max.age.ms still
> feels
> > > > > weird to
> > > > > > >>>> me.
> > > > > > >>>>>> It
> > > > > > >>>>>> categorizes partitions as new or existing based on the
> > > partition
> > > > > > >>>> creation
> > > > > > >>>>>> time. Intuitively, the categorization should be based on
> the
> > > group
> > > > > > >>>> creation
> > > > > > >>>>>> time: all partitions existing when the group is created
> are
> > > > > existing
> > > > > > >>> and
> > > > > > >>>>>> all partitions created after the group creation are new
> > > > > partitions.
> > > > > > >>>>>>
> > > > > > >>>>>> Jun
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > > wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Hi all,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Manually bumping this thread. If there is no further
> > > > > > >>>>>>> discussion, I will close the vote.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Best Regards,
> > > > > > >>>>>>> Jiunn-Yang
> > > > > > >>>>>>>
> > > > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Hello Jian,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks for your feedback,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Agreed, partition expansion is a common operational
> task,
> > > not an
> > > > > > >>> edge
> > > > > > >>>>>>>> case. I've updated the Motivation section accordingly.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Best Regards,
> > > > > > >>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Hi Jiunn-Yang:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Thanks for the KIP. I think it would be useful to
> clarify
> > > that
> > > > > this
> > > > > > >>>>>> is a
> > > > > > >>>>>>>>> common scenario rather than an edge case, which further
> > > > > > >>> demonstrates
> > > > > > >>>>>> the
> > > > > > >>>>>>>>> need for this optimization. For example:
> > > > > > >>>>>>>>> A partition expansion is a common operational task in
> > > Kafka: To
> > > > > > >>>>>> balance
> > > > > > >>>>>>>>> resource utilization and cost, topics are typically
> created
> > > > > with a
> > > > > > >>>>>>> moderate
> > > > > > >>>>>>>>> default partition count. However, as traffic grows over
> > > time,
> > > > > it is
> > > > > > >>>>>>> often
> > > > > > >>>>>>>>> necessary to increase the number of partitions to
> > > accommodate
> > > > > the
> > > > > > >>>>>> higher
> > > > > > >>>>>>>>> workload.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Regards
> > > > > > >>>>>>>>> Jian
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Hello chia,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> 晚上8:29
> > > 寫道:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Hi Jiunn-Yang,
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> when
> > > > > > >>> describing
> > > > > > >>>>>>>>>>> partitions in the KIP? I understand you are using
> them to
> > > > > > >>> describe
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>> "freshness" or the users' need for the records, but
> > > applying
> > > > > > >>> these
> > > > > > >>>>>>> terms
> > > > > > >>>>>>>>>> to
> > > > > > >>>>>>>>>>> the partition itself feels a bit unnatural.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> After all, in this scenario, users don't really care
> > > whether
> > > > > a
> > > > > > >>>>>>> partition
> > > > > > >>>>>>>>>> is
> > > > > > >>>>>>>>>>> newly expanded or not. Their only expectation is that
> > > they
> > > > > won't
> > > > > > >>>>>>> silently
> > > > > > >>>>>>>>>>> lose any live records produced to the topic during
> their
> > > > > active
> > > > > > >>>>>>>>>> consumption.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Best, Chia-Ping
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Hello Jun,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > > motivation
> > > > > > >>>> section.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> 2026年5月30日
> > > > > 凌晨1:12 寫道:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > > > > motivation for
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>> KIP.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> The KIP says "The core insight is that not all
> > > partitions
> > > > > > >>> without
> > > > > > >>>>>> a
> > > > > > >>>>>>>>>>>>> committed offset are the same. A newly expanded
> > > partition
> > > > > (hot)
> > > > > > >>>> is
> > > > > > >>>>>>>>>>>>> fundamentally different from a partition the
> consumer
> > > has
> > > > > never
> > > > > > >>>>>> seen
> > > > > > >>>>>>>>>>>>> because it predates the group (cold)." Why is the
> hot
> > > > > partition
> > > > > > >>>>>>>>>>>>> fundamentally different from the cold?
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> also
> > > > > > >>>> insufficient
> > > > > > >>>>>>>>>>>> because:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> - The calculated seek time (now() - duration)
> varies
> > > across
> > > > > > >>> nodes
> > > > > > >>>>>>> due
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> large
> > > > > > >>> duration,
> > > > > > >>>>>>>>>>>> causing
> > > > > > >>>>>>>>>>>>> unnecessary reprocessing.
> > > > > > >>>>>>>>>>>>> - On network errors, the client recalculates the
> seek
> > > time
> > > > > on
> > > > > > >>>>>> retry,
> > > > > > >>>>>>>>>>>>> shifting the target timestamp forward and risking
> data
> > > > > loss."
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> However, both of these situations are rare. If
> these
> > > issues
> > > > > > >>>>>> persist,
> > > > > > >>>>>>>>>> more
> > > > > > >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> situations
> > > > > don't
> > > > > > >>>>>> need a
> > > > > > >>>>>>>>>>>> common
> > > > > > >>>>>>>>>>>>> solution. If users care about those rare
> situations,
> > > they
> > > > > can
> > > > > > >>>>>>> implement
> > > > > > >>>>>>>>>>>>> customized logic using
> > > > > > >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > s7133700@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Hello chia,
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks for the feedback,
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > should
> > > > > always
> > > > > > >>>> be
> > > > > > >>>>>>>>>>>> greater
> > > > > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >   any
> > > > > > >>>>>>> field
> > > > > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> The earliest point at which we can detect the
> version
> > > > > mismatch
> > > > > > >>>> is
> > > > > > >>>>>>>>>> during
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>> first metadata fetch after assignment, which
> occurs
> > > inside
> > > > > > >>>>>> poll().
> > > > > > >>>>>>>>>>>>>> Therefore, the
> > > > > > >>>>>>>>>>>>>> user would encounter an
> UnsupportedVersionException
> > > from
> > > > > > >>> poll().
> > > > > > >>>>>>> I’ll
> > > > > > >>>>>>>>>>>>>> clarify this in the KIP.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> 2026年5月17日
> > > > > 下午4:50 寫道:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> hi Jiunn
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> this
> > > > > > >>> partition
> > > > > > >>>>>> in
> > > > > > >>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> as
> > > > > > >>>>>>>>>> broker_current_time
> > > > > > >>>>>>>>>>>> -
> > > > > > >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > > does not
> > > > > > >>>>>> support
> > > > > > >>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > should
> > > > > always
> > > > > > >>>> be
> > > > > > >>>>>>>>>>>> greater
> > > > > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >   any
> > > > > > >>>>>>> field
> > > > > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> when
> > > > > calling
> > > > > > >>>>>>>>>> `poll()`?
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > > > >>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> based on
> > > the
> > > > > > >>>>>>> discussion
> > > > > > >>>>>>>>>>>>>>>> feedback.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> The updated design decouples the new-partition
> reset
> > > > > > >>> behavior
> > > > > > >>>>>>> from
> > > > > > >>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to
> all
> > > > > > >>>>>>> auto.offset.reset
> > > > > > >>>>>>>>>>>>>> values
> > > > > > >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > > > > >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> resets to
> > > > > > >>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > > > > > >>>>>>>>>>>>>>>> config setting
> > > > > > >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > > > > >>> auto.offset.reset
> > > > > > >>>>>>>>>> policy
> > > > > > >>>>>>>>>>>>>> continues
> > > > > > >>>>>>>>>>>>>>>> to apply unchanged.
> > > > > > >>>>>>>>>>>>>>>> - The new-partition reset behavior is
> represented
> > > by a
> > > > > > >>>> separate
> > > > > > >>>>>>>>>>>>>> internal config
> > > > > > >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently
> fixed
> > > to
> > > > > > >>>>>> earliest).
> > > > > > >>>>>>>>>> This
> > > > > > >>>>>>>>>>>>>> decoupled design makes
> > > > > > >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > > public
> > > > > > >>>>>>> user-facing
> > > > > > >>>>>>>>>>>>>> configuration in a future KIP.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> 2026年5月16日
> > > > > 清晨7:46
> > > > > > >>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> hi Jun
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> is
> > > listed
> > > > > > >>>> below:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
> > > default
> > > > > > >>> value
> > > > > > >>>>>> of
> > > > > > >>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > > by_duration and
> > > > > > >>>>>> latest,
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>> does not change the logic of
> > > auto.offset.reset=earliest.
> > > > > > >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> > > > > internal
> > > > > > >>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > .partitions=earliest
> > > > > > >>>> already
> > > > > > >>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> cases
> > > of
> > > > > other
> > > > > > >>>>>>> values
> > > > > > >>>>>>>>>>>> in a
> > > > > > >>>>>>>>>>>>>> separate KIP.
> > > > > > >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> .partitions
> > > and
> > > > > > >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be
> applied
> > > to
> > > > > all
> > > > > > >>> for
> > > > > > >>>>>>>>>>>>>> consistency.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> WDYT?
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > > > > >>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> "When a
> > > > > Kafka
> > > > > > >>>>>> topic
> > > > > > >>>>>>> is
> > > > > > >>>>>>>>>>>>>> expanded
> > > > > > >>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> latest
> > > auto
> > > > > > >>> offset
> > > > > > >>>>>>> reset
> > > > > > >>>>>>>>>>>>>> policy
> > > > > > >>>>>>>>>>>>>>>>>> will silently miss all records produced to
> those
> > > > > > >>> partitions
> > > > > > >>>>>>> before
> > > > > > >>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > > record
> > > > > loss
> > > > > > >>>>>> issue
> > > > > > >>>>>>>>>> could
> > > > > > >>>>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>>>>> happen, right?
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > > .partitions
> > > > > will
> > > > > > >>>> take
> > > > > > >>>>>>> the
> > > > > > >>>>>>>>>>>> same
> > > > > > >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> set
> > > it
> > > > > > >>>>>>> by_duration if
> > > > > > >>>>>>>>>>>>>> needed.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> Tsai <
> > > > > > >>>>>>>>>> chia7712@apache.org
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > > something
> > > > > > >>>>>>> important
> > > > > > >>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>> your
> > > > > > >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> to
> > > > > clarify
> > > > > > >>> with
> > > > > > >>>>>> a
> > > > > > >>>>>>> few
> > > > > > >>>>>>>>>>>>>> questions:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> this
> > > > > logic to
> > > > > > >>>>>>> other
> > > > > > >>>>>>>>>>>> reset
> > > > > > >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> earliest
> > > or
> > > > > > >>>>>> by_duration
> > > > > > >>>>>>>>>>>> don't
> > > > > > >>>>>>>>>>>>>> seem
> > > > > > >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> issue
> > > when a
> > > > > > >>>>>>> partition
> > > > > > >>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>> expanded.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > > configure for
> > > > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they
> set
> > > it to
> > > > > > >>>>>> earliest
> > > > > > >>>>>>> or
> > > > > > >>>>>>>>>>>>>> latest,
> > > > > > >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases.
> For
> > > > > example,
> > > > > > >>>>>> if a
> > > > > > >>>>>>>>>>>>>> consumer is
> > > > > > >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > > created
> > > > > during
> > > > > > >>>>>> that
> > > > > > >>>>>>>>>>>>>> downtime,
> > > > > > >>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> latest
> > > when
> > > > > > >>>>>> resuming,
> > > > > > >>>>>>>>>>>> rather
> > > > > > >>>>>>>>>>>>>> than
> > > > > > >>>>>>>>>>>>>>>>>>> reading from earliest just because the
> partition
> > > is
> > > > > > >>>>>>> technically
> > > > > > >>>>>>>>>>>>>> "new" to
> > > > > > >>>>>>>>>>>>>>>>>>> the group.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing
> a
> > > > > max.age
> > > > > > >>>>>>> threshold.
> > > > > > >>>>>>>>>>>> It
> > > > > > >>>>>>>>>>>>>> gives
> > > > > > >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > > genuinely
> > > > > > >>>> "hot/new"
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>>>>>>> what is
> > > > > > >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> bit
> > > > > weird. It
> > > > > > >>>>>> only
> > > > > > >>>>>>>>>>>>>> applies when
> > > > > > >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> seems
> > > that
> > > > > the
> > > > > > >>>>>>>>>> motivation
> > > > > > >>>>>>>>>>>>>>>>>>> equally
> > > > > > >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> other
> > > > > values
> > > > > > >>> like
> > > > > > >>>>>>>>>>>>>> by_duration.
> > > > > > >>>>>>>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> > > way to
> > > > > > >>>> control
> > > > > > >>>>>>>>>> newly
> > > > > > >>>>>>>>>>>>>> created
> > > > > > >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> group
> > > > > starts.
> > > > > > >>>>>>> Have we
> > > > > > >>>>>>>>>>>>>>>>>>> considered
> > > > > > >>>>>>>>>>>>>>>>>>>> adding a new config like
> auto.offset.reset.new
> > > > > > >>>> .partitions?
> > > > > > >>>>>>> If
> > > > > > >>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>> new
> > > > > > >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > > defaults
> > > > > to
> > > > > > >>> the
> > > > > > >>>>>>>>>> policy
> > > > > > >>>>>>>>>>>>>> used
> > > > > > >>>>>>>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > > > > explicitly to
> > > > > > >>>>>>>>>> customize
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > > > s7133700@gmail.com>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> 晚上10:37
> > > 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> The
> > > > > > >>>>>>> PartitionMetadata
> > > > > > >>>>>>>>>>>>>> struct
> > > > > > >>>>>>>>>>>>>>>>>>>>> gains a new
> > > > > > >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > > computed
> > > > > > >>>>>>> server-side
> > > > > > >>>>>>>>>>>> by
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>> broker as
> > > > > > >>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> partition_creation_time.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > > > > >>>>>>> MembershipManager
> > > > > > >>>>>>>>>>>>>> detects
> > > > > > >>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > > > >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > > metadata
> > > > > for
> > > > > > >>>> the
> > > > > > >>>>>>>>>>>> affected
> > > > > > >>>>>>>>>>>>>>>>>>> topic
> > > > > > >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > > > > >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> even
> > > if
> > > > > the
> > > > > > >>>>>> topic
> > > > > > >>>>>>> ID
> > > > > > >>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>> already
> > > > > > >>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > > supported
> > > > > > >>>>>>>>>>>>>> MetadataResponse
> > > > > > >>>>>>>>>>>>>>>>>>>>> version via the
> > > > > > >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> time.
> > > If the
> > > > > > >>>>>>> negotiated
> > > > > > >>>>>>>>>>>>>>>>>>> version is
> > > > > > >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > > > >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > > PartitionAgeMs
> > > > > at
> > > > > > >>> all
> > > > > > >>>>>> and
> > > > > > >>>>>>>>>> can
> > > > > > >>>>>>>>>>>>>>>>>>> throw an
> > > > > > >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > > > >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > > back to
> > > > > > >>> latest
> > > > > > >>>>>>> and
> > > > > > >>>>>>>>>>>>>> risking
> > > > > > >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> signal.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > 2026年4月29日
> > > > > > >>>> 下午4:04
> > > > > > >>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > > 'age'
> > > > > > >>>>>> resolution
> > > > > > >>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> the
> > > > > control
> > > > > > >>>>>> plane
> > > > > > >>>>>>>>>>>> clean.
> > > > > > >>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>>>> main
> > > > > > >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> introducing
> > > > > > >>>> inter-broker
> > > > > > >>>>>>>>>> clock
> > > > > > >>>>>>>>>>>>>> skew.
> > > > > > >>>>>>>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> single
> > > > > source of
> > > > > > >>>>>> truth
> > > > > > >>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>> time.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > > should be
> > > > > > >>>>>>> negligible.
> > > > > > >>>>>>>>>>>>>> Given
> > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > > configured in
> > > > > > >>>> minutes
> > > > > > >>>>>>> or
> > > > > > >>>>>>>>>>>>>> hours, a
> > > > > > >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > > brokers
> > > > > > >>> won't
> > > > > > >>>>>>> impact
> > > > > > >>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> dev@kafka.apache.org>
> > > 於
> > > > > > >>>>>> 2026年4月29日
> > > > > > >>>>>>>>>>>> 下午3:29
> > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > > previous
> > > > > > >>>>>>> conversation
> > > > > > >>>>>>>>>>>> but I
> > > > > > >>>>>>>>>>>>>>>>>>> took a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> flow
> > > with
> > > > > the
> > > > > > >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> new
> > > > > boolean;
> > > > > > >>> the
> > > > > > >>>>>>> KIP
> > > > > > >>>>>>>>>>>>>> states
> > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> this
> > > > > boolean
> > > > > > >>> is
> > > > > > >>>>>>> set.
> > > > > > >>>>>>>>>>>>>>>>>>> Implicitly,
> > > > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a
> new
> > > > > > >>> partition,
> > > > > > >>>>>> it
> > > > > > >>>>>>>>>> will
> > > > > > >>>>>>>>>>>>>>>>>>> issue a
> > > > > > >>>>>>>>>>>>>>>>>>>>> new
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> receive
> > > the
> > > > > ages.
> > > > > > >>>> Is
> > > > > > >>>>>>> my
> > > > > > >>>>>>>>>>>>>>>>>>>>> understanding
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> flow
> > > and
> > > > > also
> > > > > > >>>>>>> explain
> > > > > > >>>>>>>>>>>>>> how it
> > > > > > >>>>>>>>>>>>>>>>>>>>> fits
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> offsets,
> > > fetch
> > > > > > >>>>>> offsets,
> > > > > > >>>>>>>>>>>> etc.).
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > > wonder
> > > > > if
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> right
> > > > > place
> > > > > > >>> for
> > > > > > >>>>>>> this
> > > > > > >>>>>>>>>>>> given
> > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>> a new
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> Alternatively, it
> > > > > could
> > > > > > >>>>>> simply
> > > > > > >>>>>>>>>>>>>> include
> > > > > > >>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > > > > cautious
> > > > > > >>>> about
> > > > > > >>>>>>> not
> > > > > > >>>>>>>>>>>>>>>>>>> overloading
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > > unrelated
> > > > > > >>>> concepts.
> > > > > > >>>>>>> The
> > > > > > >>>>>>>>>>>> API
> > > > > > >>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> revoking
> > > > > > >>>> partitions.
> > > > > > >>>>>>> The
> > > > > > >>>>>>>>>>>> fact
> > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> corresponding
> > > > > Streams
> > > > > > >>> API
> > > > > > >>>>>>> also
> > > > > > >>>>>>>>>>>>>> suggests
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> would we
> > > do
> > > > > if we
> > > > > > >>>>>>> want to
> > > > > > >>>>>>>>>>>>>>>>>>> support
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> David
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> Muralidhar
> > > > > Basani
> > > > > > >>>> via
> > > > > > >>>>>>> dev
> > > > > > >>>>>>>>>> <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> know
> > > > > about
> > > > > > >>> the
> > > > > > >>>>>>> gap.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > > > > >>>>>>> RequestPartitionAges
> > > > > > >>>>>>>>>>>>>> field.
> > > > > > >>>>>>>>>>>>>>>>>>> Can a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > > > > PartitionAges on
> > > > > > >>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > > > >>>>>>>>>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > > > > recommended
> > > > > > >>>>>> value
> > > > > > >>>>>>> or
> > > > > > >>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>> ConfigDef
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on
> the
> > > > > > >>>>>>>>>>>> metadata.max.age.ms
> > > > > > >>>>>>>>>>>>>> ?
> > > > > > >>>>>>>>>>>>>>>>>>>>> Sizing
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > > guess.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes
> to
> > > Kafka
> > > > > > >>>>>> Streams,
> > > > > > >>>>>>>>>>>> would
> > > > > > >>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>>>>>> better
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > > > >>>>>> auto.offset.reset.latest.max.age
> > > > > > >>>>>>> to
> > > > > > >>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > > > >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > > > >>>>>>>>>>>>>>>>>>>>> for a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> it?
> > > This
> > > > > is
> > > > > > >>> the
> > > > > > >>>>>>> most
> > > > > > >>>>>>>>>>>>>>>>>>> familiar
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > > > > mistakenly
> > > > > > >>>>>>> configure
> > > > > > >>>>>>>>>>>>>> it. Or
> > > > > > >>>>>>>>>>>>>>>>>>>>> may be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> consumer
> > > falls
> > > > > > >>> back
> > > > > > >>>>>> to
> > > > > > >>>>>>>>>>>>>> earliest"
> > > > > > >>>>>>>>>>>>>>>>>>>>> reads as
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > > per-partition
> > > > > > >>> which
> > > > > > >>>>>>> isn't
> > > > > > >>>>>>>>>>>>>>>>>>> supported.
> > > > > > >>>>>>>>>>>>>>>>>>>>> May
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> "consumer
> > > > > resolves
> > > > > > >>>> the
> > > > > > >>>>>>>>>>>> initial
> > > > > > >>>>>>>>>>>>>>>>>>>>> position to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > > > > earliest was
> > > > > > >>>>>>> applied
> > > > > > >>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > > config is
> > > > > > >>>>>>> unchanged.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > > > > >>>>>>> s7133700@gmail.com>
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> > > change.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> 於
> > > > > > >>> 2026年4月28日
> > > > > > >>>>>>> 晚上8:03
> > > > > > >>>>>>>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> partition
> > > > > creation
> > > > > > >>>> time
> > > > > > >>>>>>> via
> > > > > > >>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>> Admin
> > > > > > >>>>>>>>>>>>>>>>>>>>> API?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> users to
> > > > > > >>> diagnose
> > > > > > >>>>>> and
> > > > > > >>>>>>>>>>>>>>>>>>> troubleshoot
> > > > > > >>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > auto.offset.reset.latest.max.age
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> on
> > > > > KIP-1327
> > > > > > >>>>>>> Prevent
> > > > > > >>>>>>>>>> Hot
> > > > > > >>>>>>>>>>>>>> Data
> > > > > > >>>>>>>>>>>>>>>>>>>>> Loss
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> newly
> > > > > expanded
> > > > > > >>>>>> (hot)
> > > > > > >>>>>>>>>>>>>> partitions
> > > > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> threshold
> > > > > > >>>> automatically
> > > > > > >>>>>>> fall
> > > > > > >>>>>>>>>>>>>> back
> > > > > > >>>>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> forcing a
> > > > > full
> > > > > > >>>>>>> historical
> > > > > > >>>>>>>>>>>>>>>>>>> reprocess.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Jun

Yes, updating the docs requires minimal changes to the codebase, but it is not an out-of-the-box solution. From my perspective, this issue can be addressed by setting the console consumer's policy back to latest.

Let's see what Jiunn thinks

Best,
Chia-Ping

On 2026/07/06 15:59:37 Jun Rao via dev wrote:
> Hi, Chia-Ping,
> 
> One potential downside of changing the default is that some existing tools
> (e.g. console consumer) may depend on this behavior. For example, a user
> testing the system may expect to see only new messages and could be
> surprised by the change. Another option is to leave the default unchanged,
> but document clearly how to prevent losing messages on new partitions
> through by_duration. The users can then make the corresponding config
> changes if needed.
> 
> Thanks,
> 
> Jun
> 
> On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> 
> > Hi Jun,
> >
> > Thanks for the feedback. I agree with your conclusion. It would be better
> > to "make haste" with a new KIP to address this. We can focus the discussion
> > on the following points:
> >
> > 1) Should we change the default reset policy to by_duration for all
> > consumers?
> > 2) What is the optimal duration to prevent data loss while avoiding large
> > backlogs? Note that the classic consumer has a 3-second heartbeat interval,
> > whereas the async consumer uses 5 seconds.
> > 3) What are the potential side effects of replacing latest with
> > by_duration as the default behavior?
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> > > Hi, Chia-Ping,
> > >
> > > I agree that using by_duration with a value of 5s to 10s addresses most
> > > common issues with expanded partitions, given the default
> > > heartbeat.interval.ms is 3 seconds. We can consider setting that as the
> > new
> > > default. The current proposal may address some other rarer cases, but it
> > > feels a bit like over engineering.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org>
> > wrote:
> > >
> > > > - Short durations fail during extended downtimes, leading to the exact
> > > > silent data
> > > > loss this KIP aims to prevent.
> > > >
> > > > Losing the records produced to expanded partitions during a downtime is
> > > > actually the expected behavior for the latest policy. This is exactly
> > why,
> > > > in the current KIP proposal, we prefer by_duration over earliest for
> > the
> > > > expanded partitions—we want to avoid forcing users to consume a massive
> > > > accumulated backlog just because a partition is technically "new".
> > > >
> > > > To quote my previous comment from May 15:
> > > > "...if a consumer is offline for a while and a new partition is created
> > > > during that downtime, the user might actually want to skip to latest
> > when
> > > > resuming, rather than reading from earliest just because the partition
> > is
> > > > technically 'new' to the group."
> > > >
> > > > On 2026/06/30 12:12:47 黃竣陽 wrote:
> > > > > Hello all,
> > > > >
> > > > > The metadata blindness and consumer downtime windows are inherently
> > > > > unpredictable, making a fixed by_duration approach unviable. No
> > static
> > > > > duration can accommodate both extremes:
> > > > >
> > > > > - Short durations fail during extended downtimes, leading to the
> > exact
> > > > silent data
> > > > > loss this KIP aims to prevent.
> > > > > - Long durations (calibrated for worst-case scenarios) cause routine
> > > > restarts to
> > > > > trigger unnecessary mass replays, undermining the latest offset reset
> > > > policy.
> > > > >
> > > > > This is why the proposed solution uses creation time. It provides a
> > > > discrete,
> > > > > deterministic signal—verifying whether a partition is actually newer
> > > > than the group,
> > > > > rather than relying on an arbitrary time window that has no
> > correlation
> > > > with the consumer’s
> > > > > actual downtime.
> > > > >
> > > > > Best Regards,
> > > > > Jiunn-Yang
> > > > >
> > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to revisit Jun previous comment about using
> > by_duration=5s. I
> > > > realize now that having a small backlog is actually an acceptable
> > > > trade-off, especially since the latest policy inherently brings a tiny
> > > > backlog to starting consumers anyway due to the time gap between
> > fetching
> > > > the offset and actually consuming.
> > > > > >
> > > > > > If we bring the simple by_duration=5s solution back to the KIP, I'd
> > > > like to discuss a further step: Should we just change the default
> > policy
> > > > from latest to by_duration=5s specifically for the new consumer and
> > share
> > > > consumer?
> > > > > >
> > > > > > Best,
> > > > > > Chia-Ping
> > > > > >
> > > > > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > > > > >> hi Jun
> > > > > >>
> > > > > >> The most important part of this story is how users should expect
> > the
> > > > data
> > > > > >> they can see when using the latest or by_duration policy with
> > expanded
> > > > > >> partitions.
> > > > > >>
> > > > > >> Yes, the by_duration policy can minimize data loss, but it is
> > > > > >> non-deterministic, which means users will either read too many
> > > > historical
> > > > > >> records from existing partitions or lose some records from
> > expanded
> > > > > >> partitions.
> > > > > >>
> > > > > >> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
> > > > > >> understand, and that is why I preferred having a whole new policy
> > > > based
> > > > > >> entirely on group creation time (KIP-1282)
> > > > > >>
> > > > > >> Best,
> > > > > >> Chia-Ping
> > > > > >>
> > > > > >> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> > > > > >>
> > > > > >>> Hi, Chia-Ping and Jiunn-Yang,
> > > > > >>>
> > > > > >>> Thanks for the reply. I am still trying to understand the value
> > of
> > > > the new
> > > > > >>> configs with the KIP.
> > > > > >>>
> > > > > >>> The motivation of the KIP is that a user doesn't want to miss the
> > > > data if
> > > > > >>> the backlog is small. The backlog of the existing partition is
> > easy
> > > > to
> > > > > >>> understand because it relates to retention time. The backlog for
> > the
> > > > new
> > > > > >>> partition is a bit subtle to understand since it depends on the
> > > > metadata
> > > > > >>> refresh delay. To set auto.offset.reset.max.age.ms, the user
> > needs
> > > > to
> > > > > >>> understand the metadata refresh delay on the consumer side and
> > use
> > > > it to
> > > > > >>> set the config.
> > > > > >>>
> > > > > >>> Now, let's consider the alternative: setting the same value for
> > the
> > > > > >>> existing by_duration policy. The KIP lists three issues with this
> > > > approach.
> > > > > >>> 1. It computes the seek target client-side as now() - duration,
> > which
> > > > > >>> introduces clock skew across consumers and forces operators to
> > choose
> > > > > >>> overly large durations, causing unnecessary reprocessing.
> > > > > >>> 2. The target timestamp is recomputed on each retry, so failed
> > > > > >>> ListOffsetsRequest retries can shift the target forward and
> > > > potentially
> > > > > >>> miss records produced between attempts.
> > > > > >>> 3. It applies uniformly to all partitions without committed
> > offsets,
> > > > and
> > > > > >>> cannot distinguish newly expanded partitions from long-existing
> > > > partitions
> > > > > >>> newly assigned to the group, leading to unnecessary replay.
> > > > > >>>
> > > > > >>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
> > > > buffer to
> > > > > >>> the metadata refresh delay. We could also consider improving the
> > > > > >>> implementation. For issue 3, the metadata refresh delay is
> > typically
> > > > low
> > > > > >>> (in the order of minutes with the classic consumer and tens of
> > > > seconds with
> > > > > >>> the new consumer). If a user is ok with reading that much backlog
> > > > for new
> > > > > >>> partitions, it seems they will be ok doing the same for existing
> > > > > >>> partitions.
> > > > > >>>
> > > > > >>> So, instead of introducing a new config, could we just reuse the
> > > > existing
> > > > > >>> config with better documentation and/or implementation?
> > > > > >>>
> > > > > >>> Jun
> > > > > >>>
> > > > > >>>
> > > > > >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > > >>>
> > > > > >>>> Hello Jun,
> > > > > >>>>
> > > > > >>>> You're right that group creation time is the more intuitive
> > answer
> > > > at
> > > > > >>>> first glance,
> > > > > >>>> the KIP's own motivation talks about partitions that "predate
> > the
> > > > group"
> > > > > >>>> vs partitions
> > > > > >>>> "created during group runtime," which directly points to a
> > > > > >>> group-lifecycle
> > > > > >>>> classifier.
> > > > > >>>> I'd like to walk through why we landed on partition age, and the
> > > > > >>>> trade-offs we considered.
> > > > > >>>>
> > > > > >>>> We evaluated three candidate signals:
> > > > > >>>>
> > > > > >>>> 1. `by_duration:5secs`
> > > > > >>>>
> > > > > >>>> This covers the metadata blindness window, but has issues the
> > KIP
> > > > > >>>> currently documents
> > > > > >>>> under "Why not use `by_duration`?":
> > > > > >>>>
> > > > > >>>> - Client-side `now() - duration` introduces clock skew across
> > > > consumers.
> > > > > >>>> - `ListOffsets` retries shift the target forward, potentially
> > > > missing
> > > > > >>>> records produced between
> > > > > >>>> attempts.
> > > > > >>>> - It applies uniformly to all partitions without committed
> > offsets,
> > > > > >>>> including pre-existing partitions
> > > > > >>>> newly assigned to the group, causing unnecessary replay.
> > > > > >>>>
> > > > > >>>> 2. Group creation time as classifier
> > > > > >>>>
> > > > > >>>> This works cleanly when the consumer is actively running. Our
> > > > concern
> > > > > >>>> is the idle / late-rejoin case:
> > > > > >>>>
> > > > > >>>> T=0:         Group created.
> > > > > >>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > > > >>>> T=50:        Partition added during the idle window.
> > > > > >>>> T=100:       Consumer resumes.
> > > > > >>>>
> > > > > >>>> Under group creation time, the new partition is classified as
> > new
> > > > > >>>> (`50 > 0`) and reset to `earliest`, replaying everything from
> > T=50.
> > > > > >>>> But during `[T=1, T=100]`, base partitions also accumulated data
> > > > that
> > > > > >>>> the consumer accepts as lost — that is precisely the contract of
> > > > > >>>> `auto.offset.reset=latest`. There is no principled reason to
> > treat
> > > > > >>>> the new partition differently; both contain backlog accumulated
> > > > during
> > > > > >>>> the same idle window.
> > > > > >>>>
> > > > > >>>> This aligns with the "backlog is backlog” principle you raised
> > in
> > > > > >>>> the KIP-1282 thread: a `latest` user has tolerated some backlog
> > on
> > > > > >>>> every other partition during the same idle period; forcing
> > 0-backlog
> > > > > >>>> tolerance only on new partitions would be inconsistent with that
> > > > > >>>> tolerance.
> > > > > >>>>
> > > > > >>>> 3. Partition age vs threshold
> > > > > >>>>
> > > > > >>>> Partition age corresponds to the actual silent data loss window,
> > > > > >>>> the gap between partition creation and the consumer’s metadata
> > > > > >>>> refresh. Within this window, data loss is genuinely silent: the
> > > > > >>>> consumer had no opportunity to know about the partition. Outside
> > > > this
> > > > > >>>> window, missing data reflects either:
> > > > > >>>>
> > > > > >>>> - (a) the user’s tolerated cost of running with idle consumers,
> > or
> > > > > >>>> - (b) an operational issue to surface via monitoring, not via
> > reset
> > > > > >>> policy.
> > > > > >>>>
> > > > > >>>> We did not choose partition age because it is more elegant than
> > > > group
> > > > > >>>> creation time — we chose it because its failure mode (requires a
> > > > > >>>> threshold) is
> > > > > >>>> less invasive than the failure mode of group creation time
> > > > (overrides
> > > > > >>>> user-stated
> > > > > >>>> `latest` intent during idle periods).
> > > > > >>>>
> > > > > >>>> Best Regards,
> > > > > >>>> Jiunn-Yang
> > > > > >>>>
> > > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > > > > >>>>>
> > > > > >>>>> Hi Jun,
> > > > > >>>>>
> > > > > >>>>> Relying on both creation times will create an inconsistent
> > > > scenario. A
> > > > > >>>>> consumer that lost all offsets due to a long sleep will seek
> > to the
> > > > > >>>>> beginning for the partitions created later than the group.
> > > > > >>>>>
> > > > > >>>>> That is why we initially proposed KIP-1282 to fix the
> > inconsistency
> > > > > >>>> using a
> > > > > >>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
> > > > KIP-1327
> > > > > >>>> goes
> > > > > >>>>> back to using flexible configurations to prevent users from
> > falling
> > > > > >>> into
> > > > > >>>>> that pitfall.
> > > > > >>>>>
> > > > > >>>>> Best, Chia-Ping
> > > > > >>>>>
> > > > > >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > 上午6:49寫道:
> > > > > >>>>>
> > > > > >>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>
> > > > > >>>>>> Thanks for the reply and sorry for the late reply.
> > > > > >>>>>>
> > > > > >>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels
> > > > weird to
> > > > > >>>> me.
> > > > > >>>>>> It
> > > > > >>>>>> categorizes partitions as new or existing based on the
> > partition
> > > > > >>>> creation
> > > > > >>>>>> time. Intuitively, the categorization should be based on the
> > group
> > > > > >>>> creation
> > > > > >>>>>> time: all partitions existing when the group is created are
> > > > existing
> > > > > >>> and
> > > > > >>>>>> all partitions created after the group creation are new
> > > > partitions.
> > > > > >>>>>>
> > > > > >>>>>> Jun
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hi all,
> > > > > >>>>>>>
> > > > > >>>>>>> Manually bumping this thread. If there is no further
> > > > > >>>>>>> discussion, I will close the vote.
> > > > > >>>>>>>
> > > > > >>>>>>> Best Regards,
> > > > > >>>>>>> Jiunn-Yang
> > > > > >>>>>>>
> > > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Hello Jian,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks for your feedback,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Agreed, partition expansion is a common operational task,
> > not an
> > > > > >>> edge
> > > > > >>>>>>>> case. I've updated the Motivation section accordingly.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Best Regards,
> > > > > >>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>
> > > > > >>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Hi Jiunn-Yang:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks for the KIP. I think it would be useful to clarify
> > that
> > > > this
> > > > > >>>>>> is a
> > > > > >>>>>>>>> common scenario rather than an edge case, which further
> > > > > >>> demonstrates
> > > > > >>>>>> the
> > > > > >>>>>>>>> need for this optimization. For example:
> > > > > >>>>>>>>> A partition expansion is a common operational task in
> > Kafka: To
> > > > > >>>>>> balance
> > > > > >>>>>>>>> resource utilization and cost, topics are typically created
> > > > with a
> > > > > >>>>>>> moderate
> > > > > >>>>>>>>> default partition count. However, as traffic grows over
> > time,
> > > > it is
> > > > > >>>>>>> often
> > > > > >>>>>>>>> necessary to increase the number of partitions to
> > accommodate
> > > > the
> > > > > >>>>>> higher
> > > > > >>>>>>>>> workload.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Regards
> > > > > >>>>>>>>> Jian
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Hello chia,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29
> > 寫道:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Hi Jiunn-Yang,
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> > > > > >>> describing
> > > > > >>>>>>>>>>> partitions in the KIP? I understand you are using them to
> > > > > >>> describe
> > > > > >>>>>> the
> > > > > >>>>>>>>>>> "freshness" or the users' need for the records, but
> > applying
> > > > > >>> these
> > > > > >>>>>>> terms
> > > > > >>>>>>>>>> to
> > > > > >>>>>>>>>>> the partition itself feels a bit unnatural.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> After all, in this scenario, users don't really care
> > whether
> > > > a
> > > > > >>>>>>> partition
> > > > > >>>>>>>>>> is
> > > > > >>>>>>>>>>> newly expanded or not. Their only expectation is that
> > they
> > > > won't
> > > > > >>>>>>> silently
> > > > > >>>>>>>>>>> lose any live records produced to the topic during their
> > > > active
> > > > > >>>>>>>>>> consumption.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Best, Chia-Ping
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Hello Jun,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > motivation
> > > > > >>>> section.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
> > > > 凌晨1:12 寫道:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > > > motivation for
> > > > > >>>>>> the
> > > > > >>>>>>>>>> KIP.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> The KIP says "The core insight is that not all
> > partitions
> > > > > >>> without
> > > > > >>>>>> a
> > > > > >>>>>>>>>>>>> committed offset are the same. A newly expanded
> > partition
> > > > (hot)
> > > > > >>>> is
> > > > > >>>>>>>>>>>>> fundamentally different from a partition the consumer
> > has
> > > > never
> > > > > >>>>>> seen
> > > > > >>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> > > > partition
> > > > > >>>>>>>>>>>>> fundamentally different from the cold?
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> > > > > >>>> insufficient
> > > > > >>>>>>>>>>>> because:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> - The calculated seek time (now() - duration) varies
> > across
> > > > > >>> nodes
> > > > > >>>>>>> due
> > > > > >>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> > > > > >>> duration,
> > > > > >>>>>>>>>>>> causing
> > > > > >>>>>>>>>>>>> unnecessary reprocessing.
> > > > > >>>>>>>>>>>>> - On network errors, the client recalculates the seek
> > time
> > > > on
> > > > > >>>>>> retry,
> > > > > >>>>>>>>>>>>> shifting the target timestamp forward and risking data
> > > > loss."
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> However, both of these situations are rare. If these
> > issues
> > > > > >>>>>> persist,
> > > > > >>>>>>>>>> more
> > > > > >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations
> > > > don't
> > > > > >>>>>> need a
> > > > > >>>>>>>>>>>> common
> > > > > >>>>>>>>>>>>> solution. If users care about those rare situations,
> > they
> > > > can
> > > > > >>>>>>> implement
> > > > > >>>>>>>>>>>>> customized logic using
> > > > > >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > s7133700@gmail.com>
> > > > > >>> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Hello chia,
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Thanks for the feedback,
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value
> > should
> > > > always
> > > > > >>>> be
> > > > > >>>>>>>>>>>> greater
> > > > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > > > >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> >   any
> > > > > >>>>>>> field
> > > > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> The earliest point at which we can detect the version
> > > > mismatch
> > > > > >>>> is
> > > > > >>>>>>>>>> during
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>> first metadata fetch after assignment, which occurs
> > inside
> > > > > >>>>>> poll().
> > > > > >>>>>>>>>>>>>> Therefore, the
> > > > > >>>>>>>>>>>>>> user would encounter an UnsupportedVersionException
> > from
> > > > > >>> poll().
> > > > > >>>>>>> I’ll
> > > > > >>>>>>>>>>>>>> clarify this in the KIP.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
> > > > 下午4:50 寫道:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> hi Jiunn
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> > > > > >>> partition
> > > > > >>>>>> in
> > > > > >>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> > > > > >>>>>>>>>> broker_current_time
> > > > > >>>>>>>>>>>> -
> > > > > >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > does not
> > > > > >>>>>> support
> > > > > >>>>>>>>>> this
> > > > > >>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value
> > should
> > > > always
> > > > > >>>> be
> > > > > >>>>>>>>>>>> greater
> > > > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> >   any
> > > > > >>>>>>> field
> > > > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
> > > > calling
> > > > > >>>>>>>>>> `poll()`?
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > > >>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on
> > the
> > > > > >>>>>>> discussion
> > > > > >>>>>>>>>>>>>>>> feedback.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
> > > > > >>> behavior
> > > > > >>>>>>> from
> > > > > >>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> > > > > >>>>>>> auto.offset.reset
> > > > > >>>>>>>>>>>>>> values
> > > > > >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > > > >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> > > > > >>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > > > > >>>>>>>>>>>>>>>> config setting
> > > > > >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > > > >>> auto.offset.reset
> > > > > >>>>>>>>>> policy
> > > > > >>>>>>>>>>>>>> continues
> > > > > >>>>>>>>>>>>>>>> to apply unchanged.
> > > > > >>>>>>>>>>>>>>>> - The new-partition reset behavior is represented
> > by a
> > > > > >>>> separate
> > > > > >>>>>>>>>>>>>> internal config
> > > > > >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed
> > to
> > > > > >>>>>> earliest).
> > > > > >>>>>>>>>> This
> > > > > >>>>>>>>>>>>>> decoupled design makes
> > > > > >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > public
> > > > > >>>>>>> user-facing
> > > > > >>>>>>>>>>>>>> configuration in a future KIP.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
> > > > 清晨7:46
> > > > > >>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> hi Jun
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is
> > listed
> > > > > >>>> below:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
> > default
> > > > > >>> value
> > > > > >>>>>> of
> > > > > >>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > by_duration and
> > > > > >>>>>> latest,
> > > > > >>>>>>> and
> > > > > >>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>> does not change the logic of
> > auto.offset.reset=earliest.
> > > > > >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> > > > internal
> > > > > >>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > .partitions=earliest
> > > > > >>>> already
> > > > > >>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases
> > of
> > > > other
> > > > > >>>>>>> values
> > > > > >>>>>>>>>>>> in a
> > > > > >>>>>>>>>>>>>> separate KIP.
> > > > > >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions
> > and
> > > > > >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied
> > to
> > > > all
> > > > > >>> for
> > > > > >>>>>>>>>>>>>> consistency.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> WDYT?
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > > > >>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a
> > > > Kafka
> > > > > >>>>>> topic
> > > > > >>>>>>> is
> > > > > >>>>>>>>>>>>>> expanded
> > > > > >>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest
> > auto
> > > > > >>> offset
> > > > > >>>>>>> reset
> > > > > >>>>>>>>>>>>>> policy
> > > > > >>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> > > > > >>> partitions
> > > > > >>>>>>> before
> > > > > >>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > record
> > > > loss
> > > > > >>>>>> issue
> > > > > >>>>>>>>>> could
> > > > > >>>>>>>>>>>>>> also
> > > > > >>>>>>>>>>>>>>>>>> happen, right?
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > .partitions
> > > > will
> > > > > >>>> take
> > > > > >>>>>>> the
> > > > > >>>>>>>>>>>> same
> > > > > >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set
> > it
> > > > > >>>>>>> by_duration if
> > > > > >>>>>>>>>>>>>> needed.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > > > > >>>>>>>>>> chia7712@apache.org
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> hi Jun
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > something
> > > > > >>>>>>> important
> > > > > >>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>> your
> > > > > >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
> > > > clarify
> > > > > >>> with
> > > > > >>>>>> a
> > > > > >>>>>>> few
> > > > > >>>>>>>>>>>>>> questions:
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
> > > > logic to
> > > > > >>>>>>> other
> > > > > >>>>>>>>>>>> reset
> > > > > >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest
> > or
> > > > > >>>>>> by_duration
> > > > > >>>>>>>>>>>> don't
> > > > > >>>>>>>>>>>>>> seem
> > > > > >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue
> > when a
> > > > > >>>>>>> partition
> > > > > >>>>>>>>>> is
> > > > > >>>>>>>>>>>>>> expanded.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > configure for
> > > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set
> > it to
> > > > > >>>>>> earliest
> > > > > >>>>>>> or
> > > > > >>>>>>>>>>>>>> latest,
> > > > > >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> > > > example,
> > > > > >>>>>> if a
> > > > > >>>>>>>>>>>>>> consumer is
> > > > > >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > created
> > > > during
> > > > > >>>>>> that
> > > > > >>>>>>>>>>>>>> downtime,
> > > > > >>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest
> > when
> > > > > >>>>>> resuming,
> > > > > >>>>>>>>>>>> rather
> > > > > >>>>>>>>>>>>>> than
> > > > > >>>>>>>>>>>>>>>>>>> reading from earliest just because the partition
> > is
> > > > > >>>>>>> technically
> > > > > >>>>>>>>>>>>>> "new" to
> > > > > >>>>>>>>>>>>>>>>>>> the group.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
> > > > max.age
> > > > > >>>>>>> threshold.
> > > > > >>>>>>>>>>>> It
> > > > > >>>>>>>>>>>>>> gives
> > > > > >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > genuinely
> > > > > >>>> "hot/new"
> > > > > >>>>>>> and
> > > > > >>>>>>>>>>>>>> what is
> > > > > >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
> > > > weird. It
> > > > > >>>>>> only
> > > > > >>>>>>>>>>>>>> applies when
> > > > > >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems
> > that
> > > > the
> > > > > >>>>>>>>>> motivation
> > > > > >>>>>>>>>>>>>>>>>>> equally
> > > > > >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
> > > > values
> > > > > >>> like
> > > > > >>>>>>>>>>>>>> by_duration.
> > > > > >>>>>>>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> > way to
> > > > > >>>> control
> > > > > >>>>>>>>>> newly
> > > > > >>>>>>>>>>>>>> created
> > > > > >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
> > > > starts.
> > > > > >>>>>>> Have we
> > > > > >>>>>>>>>>>>>>>>>>> considered
> > > > > >>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > > > > >>>> .partitions?
> > > > > >>>>>>> If
> > > > > >>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>> new
> > > > > >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > defaults
> > > > to
> > > > > >>> the
> > > > > >>>>>>>>>> policy
> > > > > >>>>>>>>>>>>>> used
> > > > > >>>>>>>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > > > explicitly to
> > > > > >>>>>>>>>> customize
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > > s7133700@gmail.com>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37
> > 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > > > > >>>>>>> PartitionMetadata
> > > > > >>>>>>>>>>>>>> struct
> > > > > >>>>>>>>>>>>>>>>>>>>> gains a new
> > > > > >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > computed
> > > > > >>>>>>> server-side
> > > > > >>>>>>>>>>>> by
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>> broker as
> > > > > >>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > > > >>>>>>> MembershipManager
> > > > > >>>>>>>>>>>>>> detects
> > > > > >>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > > >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > metadata
> > > > for
> > > > > >>>> the
> > > > > >>>>>>>>>>>> affected
> > > > > >>>>>>>>>>>>>>>>>>> topic
> > > > > >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > > > >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even
> > if
> > > > the
> > > > > >>>>>> topic
> > > > > >>>>>>> ID
> > > > > >>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>> already
> > > > > >>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > supported
> > > > > >>>>>>>>>>>>>> MetadataResponse
> > > > > >>>>>>>>>>>>>>>>>>>>> version via the
> > > > > >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time.
> > If the
> > > > > >>>>>>> negotiated
> > > > > >>>>>>>>>>>>>>>>>>> version is
> > > > > >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > > >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > PartitionAgeMs
> > > > at
> > > > > >>> all
> > > > > >>>>>> and
> > > > > >>>>>>>>>> can
> > > > > >>>>>>>>>>>>>>>>>>> throw an
> > > > > >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > > >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > back to
> > > > > >>> latest
> > > > > >>>>>>> and
> > > > > >>>>>>>>>>>>>> risking
> > > > > >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > 2026年4月29日
> > > > > >>>> 下午4:04
> > > > > >>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > 'age'
> > > > > >>>>>> resolution
> > > > > >>>>>>>>>> from
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
> > > > control
> > > > > >>>>>> plane
> > > > > >>>>>>>>>>>> clean.
> > > > > >>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>>>> main
> > > > > >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > > > > >>>> inter-broker
> > > > > >>>>>>>>>> clock
> > > > > >>>>>>>>>>>>>> skew.
> > > > > >>>>>>>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
> > > > source of
> > > > > >>>>>> truth
> > > > > >>>>>>>>>> for
> > > > > >>>>>>>>>>>>>> time.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > should be
> > > > > >>>>>>> negligible.
> > > > > >>>>>>>>>>>>>> Given
> > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > configured in
> > > > > >>>> minutes
> > > > > >>>>>>> or
> > > > > >>>>>>>>>>>>>> hours, a
> > > > > >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > brokers
> > > > > >>> won't
> > > > > >>>>>>> impact
> > > > > >>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org>
> > 於
> > > > > >>>>>> 2026年4月29日
> > > > > >>>>>>>>>>>> 下午3:29
> > > > > >>>>>>>>>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > previous
> > > > > >>>>>>> conversation
> > > > > >>>>>>>>>>>> but I
> > > > > >>>>>>>>>>>>>>>>>>> took a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow
> > with
> > > > the
> > > > > >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
> > > > boolean;
> > > > > >>> the
> > > > > >>>>>>> KIP
> > > > > >>>>>>>>>>>>>> states
> > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
> > > > boolean
> > > > > >>> is
> > > > > >>>>>>> set.
> > > > > >>>>>>>>>>>>>>>>>>> Implicitly,
> > > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> > > > > >>> partition,
> > > > > >>>>>> it
> > > > > >>>>>>>>>> will
> > > > > >>>>>>>>>>>>>>>>>>> issue a
> > > > > >>>>>>>>>>>>>>>>>>>>> new
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive
> > the
> > > > ages.
> > > > > >>>> Is
> > > > > >>>>>>> my
> > > > > >>>>>>>>>>>>>>>>>>>>> understanding
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow
> > and
> > > > also
> > > > > >>>>>>> explain
> > > > > >>>>>>>>>>>>>> how it
> > > > > >>>>>>>>>>>>>>>>>>>>> fits
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets,
> > fetch
> > > > > >>>>>> offsets,
> > > > > >>>>>>>>>>>> etc.).
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > wonder
> > > > if
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
> > > > place
> > > > > >>> for
> > > > > >>>>>>> this
> > > > > >>>>>>>>>>>> given
> > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>> a new
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it
> > > > could
> > > > > >>>>>> simply
> > > > > >>>>>>>>>>>>>> include
> > > > > >>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > > > cautious
> > > > > >>>> about
> > > > > >>>>>>> not
> > > > > >>>>>>>>>>>>>>>>>>> overloading
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > unrelated
> > > > > >>>> concepts.
> > > > > >>>>>>> The
> > > > > >>>>>>>>>>>> API
> > > > > >>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > > > > >>>> partitions.
> > > > > >>>>>>> The
> > > > > >>>>>>>>>>>> fact
> > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
> > > > Streams
> > > > > >>> API
> > > > > >>>>>>> also
> > > > > >>>>>>>>>>>>>> suggests
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we
> > do
> > > > if we
> > > > > >>>>>>> want to
> > > > > >>>>>>>>>>>>>>>>>>> support
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> David
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar
> > > > Basani
> > > > > >>>> via
> > > > > >>>>>>> dev
> > > > > >>>>>>>>>> <
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
> > > > about
> > > > > >>> the
> > > > > >>>>>>> gap.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > > > >>>>>>> RequestPartitionAges
> > > > > >>>>>>>>>>>>>> field.
> > > > > >>>>>>>>>>>>>>>>>>> Can a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > > > PartitionAges on
> > > > > >>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > > >>>>>>>>>>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > > > recommended
> > > > > >>>>>> value
> > > > > >>>>>>> or
> > > > > >>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>> ConfigDef
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > > > > >>>>>>>>>>>> metadata.max.age.ms
> > > > > >>>>>>>>>>>>>> ?
> > > > > >>>>>>>>>>>>>>>>>>>>> Sizing
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > guess.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to
> > Kafka
> > > > > >>>>>> Streams,
> > > > > >>>>>>>>>>>> would
> > > > > >>>>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>>>>>> better
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > > >>>>>> auto.offset.reset.latest.max.age
> > > > > >>>>>>> to
> > > > > >>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > > >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > > >>>>>>>>>>>>>>>>>>>>> for a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it?
> > This
> > > > is
> > > > > >>> the
> > > > > >>>>>>> most
> > > > > >>>>>>>>>>>>>>>>>>> familiar
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > > > mistakenly
> > > > > >>>>>>> configure
> > > > > >>>>>>>>>>>>>> it. Or
> > > > > >>>>>>>>>>>>>>>>>>>>> may be
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer
> > falls
> > > > > >>> back
> > > > > >>>>>> to
> > > > > >>>>>>>>>>>>>> earliest"
> > > > > >>>>>>>>>>>>>>>>>>>>> reads as
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > per-partition
> > > > > >>> which
> > > > > >>>>>>> isn't
> > > > > >>>>>>>>>>>>>>>>>>> supported.
> > > > > >>>>>>>>>>>>>>>>>>>>> May
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
> > > > resolves
> > > > > >>>> the
> > > > > >>>>>>>>>>>> initial
> > > > > >>>>>>>>>>>>>>>>>>>>> position to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > > > earliest was
> > > > > >>>>>>> applied
> > > > > >>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > config is
> > > > > >>>>>>> unchanged.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > > > >>>>>>> s7133700@gmail.com>
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> > change.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > > >>> 2026年4月28日
> > > > > >>>>>>> 晚上8:03
> > > > > >>>>>>>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
> > > > creation
> > > > > >>>> time
> > > > > >>>>>>> via
> > > > > >>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>> Admin
> > > > > >>>>>>>>>>>>>>>>>>>>> API?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> > > > > >>> diagnose
> > > > > >>>>>> and
> > > > > >>>>>>>>>>>>>>>>>>> troubleshoot
> > > > > >>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > auto.offset.reset.latest.max.age
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> > > > KIP-1327
> > > > > >>>>>>> Prevent
> > > > > >>>>>>>>>> Hot
> > > > > >>>>>>>>>>>>>> Data
> > > > > >>>>>>>>>>>>>>>>>>>>> Loss
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>
> > > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
> > > > expanded
> > > > > >>>>>> (hot)
> > > > > >>>>>>>>>>>>>> partitions
> > > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > > > > >>>> automatically
> > > > > >>>>>>> fall
> > > > > >>>>>>>>>>>>>> back
> > > > > >>>>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a
> > > > full
> > > > > >>>>>>> historical
> > > > > >>>>>>>>>>>>>>>>>>> reprocess.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping,

One potential downside of changing the default is that some existing tools
(e.g. console consumer) may depend on this behavior. For example, a user
testing the system may expect to see only new messages and could be
surprised by the change. Another option is to leave the default unchanged,
but document clearly how to prevent losing messages on new partitions
through by_duration. The users can then make the corresponding config
changes if needed.

Thanks,

Jun

On Wed, Jul 1, 2026 at 9:39 AM Chia-Ping Tsai <ch...@apache.org> wrote:

> Hi Jun,
>
> Thanks for the feedback. I agree with your conclusion. It would be better
> to "make haste" with a new KIP to address this. We can focus the discussion
> on the following points:
>
> 1) Should we change the default reset policy to by_duration for all
> consumers?
> 2) What is the optimal duration to prevent data loss while avoiding large
> backlogs? Note that the classic consumer has a 3-second heartbeat interval,
> whereas the async consumer uses 5 seconds.
> 3) What are the potential side effects of replacing latest with
> by_duration as the default behavior?
>
> Best,
> Chia-Ping
>
> On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> > Hi, Chia-Ping,
> >
> > I agree that using by_duration with a value of 5s to 10s addresses most
> > common issues with expanded partitions, given the default
> > heartbeat.interval.ms is 3 seconds. We can consider setting that as the
> new
> > default. The current proposal may address some other rarer cases, but it
> > feels a bit like over engineering.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >
> > > - Short durations fail during extended downtimes, leading to the exact
> > > silent data
> > > loss this KIP aims to prevent.
> > >
> > > Losing the records produced to expanded partitions during a downtime is
> > > actually the expected behavior for the latest policy. This is exactly
> why,
> > > in the current KIP proposal, we prefer by_duration over earliest for
> the
> > > expanded partitions—we want to avoid forcing users to consume a massive
> > > accumulated backlog just because a partition is technically "new".
> > >
> > > To quote my previous comment from May 15:
> > > "...if a consumer is offline for a while and a new partition is created
> > > during that downtime, the user might actually want to skip to latest
> when
> > > resuming, rather than reading from earliest just because the partition
> is
> > > technically 'new' to the group."
> > >
> > > On 2026/06/30 12:12:47 黃竣陽 wrote:
> > > > Hello all,
> > > >
> > > > The metadata blindness and consumer downtime windows are inherently
> > > > unpredictable, making a fixed by_duration approach unviable. No
> static
> > > > duration can accommodate both extremes:
> > > >
> > > > - Short durations fail during extended downtimes, leading to the
> exact
> > > silent data
> > > > loss this KIP aims to prevent.
> > > > - Long durations (calibrated for worst-case scenarios) cause routine
> > > restarts to
> > > > trigger unnecessary mass replays, undermining the latest offset reset
> > > policy.
> > > >
> > > > This is why the proposed solution uses creation time. It provides a
> > > discrete,
> > > > deterministic signal—verifying whether a partition is actually newer
> > > than the group,
> > > > rather than relying on an arbitrary time window that has no
> correlation
> > > with the consumer’s
> > > > actual downtime.
> > > >
> > > > Best Regards,
> > > > Jiunn-Yang
> > > >
> > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to revisit Jun previous comment about using
> by_duration=5s. I
> > > realize now that having a small backlog is actually an acceptable
> > > trade-off, especially since the latest policy inherently brings a tiny
> > > backlog to starting consumers anyway due to the time gap between
> fetching
> > > the offset and actually consuming.
> > > > >
> > > > > If we bring the simple by_duration=5s solution back to the KIP, I'd
> > > like to discuss a further step: Should we just change the default
> policy
> > > from latest to by_duration=5s specifically for the new consumer and
> share
> > > consumer?
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > > > >> hi Jun
> > > > >>
> > > > >> The most important part of this story is how users should expect
> the
> > > data
> > > > >> they can see when using the latest or by_duration policy with
> expanded
> > > > >> partitions.
> > > > >>
> > > > >> Yes, the by_duration policy can minimize data loss, but it is
> > > > >> non-deterministic, which means users will either read too many
> > > historical
> > > > >> records from existing partitions or lose some records from
> expanded
> > > > >> partitions.
> > > > >>
> > > > >> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
> > > > >> understand, and that is why I preferred having a whole new policy
> > > based
> > > > >> entirely on group creation time (KIP-1282)
> > > > >>
> > > > >> Best,
> > > > >> Chia-Ping
> > > > >>
> > > > >> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> > > > >>
> > > > >>> Hi, Chia-Ping and Jiunn-Yang,
> > > > >>>
> > > > >>> Thanks for the reply. I am still trying to understand the value
> of
> > > the new
> > > > >>> configs with the KIP.
> > > > >>>
> > > > >>> The motivation of the KIP is that a user doesn't want to miss the
> > > data if
> > > > >>> the backlog is small. The backlog of the existing partition is
> easy
> > > to
> > > > >>> understand because it relates to retention time. The backlog for
> the
> > > new
> > > > >>> partition is a bit subtle to understand since it depends on the
> > > metadata
> > > > >>> refresh delay. To set auto.offset.reset.max.age.ms, the user
> needs
> > > to
> > > > >>> understand the metadata refresh delay on the consumer side and
> use
> > > it to
> > > > >>> set the config.
> > > > >>>
> > > > >>> Now, let's consider the alternative: setting the same value for
> the
> > > > >>> existing by_duration policy. The KIP lists three issues with this
> > > approach.
> > > > >>> 1. It computes the seek target client-side as now() - duration,
> which
> > > > >>> introduces clock skew across consumers and forces operators to
> choose
> > > > >>> overly large durations, causing unnecessary reprocessing.
> > > > >>> 2. The target timestamp is recomputed on each retry, so failed
> > > > >>> ListOffsetsRequest retries can shift the target forward and
> > > potentially
> > > > >>> miss records produced between attempts.
> > > > >>> 3. It applies uniformly to all partitions without committed
> offsets,
> > > and
> > > > >>> cannot distinguish newly expanded partitions from long-existing
> > > partitions
> > > > >>> newly assigned to the group, leading to unnecessary replay.
> > > > >>>
> > > > >>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
> > > buffer to
> > > > >>> the metadata refresh delay. We could also consider improving the
> > > > >>> implementation. For issue 3, the metadata refresh delay is
> typically
> > > low
> > > > >>> (in the order of minutes with the classic consumer and tens of
> > > seconds with
> > > > >>> the new consumer). If a user is ok with reading that much backlog
> > > for new
> > > > >>> partitions, it seems they will be ok doing the same for existing
> > > > >>> partitions.
> > > > >>>
> > > > >>> So, instead of introducing a new config, could we just reuse the
> > > existing
> > > > >>> config with better documentation and/or implementation?
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>>
> > > > >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hello Jun,
> > > > >>>>
> > > > >>>> You're right that group creation time is the more intuitive
> answer
> > > at
> > > > >>>> first glance,
> > > > >>>> the KIP's own motivation talks about partitions that "predate
> the
> > > group"
> > > > >>>> vs partitions
> > > > >>>> "created during group runtime," which directly points to a
> > > > >>> group-lifecycle
> > > > >>>> classifier.
> > > > >>>> I'd like to walk through why we landed on partition age, and the
> > > > >>>> trade-offs we considered.
> > > > >>>>
> > > > >>>> We evaluated three candidate signals:
> > > > >>>>
> > > > >>>> 1. `by_duration:5secs`
> > > > >>>>
> > > > >>>> This covers the metadata blindness window, but has issues the
> KIP
> > > > >>>> currently documents
> > > > >>>> under "Why not use `by_duration`?":
> > > > >>>>
> > > > >>>> - Client-side `now() - duration` introduces clock skew across
> > > consumers.
> > > > >>>> - `ListOffsets` retries shift the target forward, potentially
> > > missing
> > > > >>>> records produced between
> > > > >>>> attempts.
> > > > >>>> - It applies uniformly to all partitions without committed
> offsets,
> > > > >>>> including pre-existing partitions
> > > > >>>> newly assigned to the group, causing unnecessary replay.
> > > > >>>>
> > > > >>>> 2. Group creation time as classifier
> > > > >>>>
> > > > >>>> This works cleanly when the consumer is actively running. Our
> > > concern
> > > > >>>> is the idle / late-rejoin case:
> > > > >>>>
> > > > >>>> T=0:         Group created.
> > > > >>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > > >>>> T=50:        Partition added during the idle window.
> > > > >>>> T=100:       Consumer resumes.
> > > > >>>>
> > > > >>>> Under group creation time, the new partition is classified as
> new
> > > > >>>> (`50 > 0`) and reset to `earliest`, replaying everything from
> T=50.
> > > > >>>> But during `[T=1, T=100]`, base partitions also accumulated data
> > > that
> > > > >>>> the consumer accepts as lost — that is precisely the contract of
> > > > >>>> `auto.offset.reset=latest`. There is no principled reason to
> treat
> > > > >>>> the new partition differently; both contain backlog accumulated
> > > during
> > > > >>>> the same idle window.
> > > > >>>>
> > > > >>>> This aligns with the "backlog is backlog” principle you raised
> in
> > > > >>>> the KIP-1282 thread: a `latest` user has tolerated some backlog
> on
> > > > >>>> every other partition during the same idle period; forcing
> 0-backlog
> > > > >>>> tolerance only on new partitions would be inconsistent with that
> > > > >>>> tolerance.
> > > > >>>>
> > > > >>>> 3. Partition age vs threshold
> > > > >>>>
> > > > >>>> Partition age corresponds to the actual silent data loss window,
> > > > >>>> the gap between partition creation and the consumer’s metadata
> > > > >>>> refresh. Within this window, data loss is genuinely silent: the
> > > > >>>> consumer had no opportunity to know about the partition. Outside
> > > this
> > > > >>>> window, missing data reflects either:
> > > > >>>>
> > > > >>>> - (a) the user’s tolerated cost of running with idle consumers,
> or
> > > > >>>> - (b) an operational issue to surface via monitoring, not via
> reset
> > > > >>> policy.
> > > > >>>>
> > > > >>>> We did not choose partition age because it is more elegant than
> > > group
> > > > >>>> creation time — we chose it because its failure mode (requires a
> > > > >>>> threshold) is
> > > > >>>> less invasive than the failure mode of group creation time
> > > (overrides
> > > > >>>> user-stated
> > > > >>>> `latest` intent during idle periods).
> > > > >>>>
> > > > >>>> Best Regards,
> > > > >>>> Jiunn-Yang
> > > > >>>>
> > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > > > >>>>>
> > > > >>>>> Hi Jun,
> > > > >>>>>
> > > > >>>>> Relying on both creation times will create an inconsistent
> > > scenario. A
> > > > >>>>> consumer that lost all offsets due to a long sleep will seek
> to the
> > > > >>>>> beginning for the partitions created later than the group.
> > > > >>>>>
> > > > >>>>> That is why we initially proposed KIP-1282 to fix the
> inconsistency
> > > > >>>> using a
> > > > >>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
> > > KIP-1327
> > > > >>>> goes
> > > > >>>>> back to using flexible configurations to prevent users from
> falling
> > > > >>> into
> > > > >>>>> that pitfall.
> > > > >>>>>
> > > > >>>>> Best, Chia-Ping
> > > > >>>>>
> > > > >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> 上午6:49寫道:
> > > > >>>>>
> > > > >>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>
> > > > >>>>>> Thanks for the reply and sorry for the late reply.
> > > > >>>>>>
> > > > >>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels
> > > weird to
> > > > >>>> me.
> > > > >>>>>> It
> > > > >>>>>> categorizes partitions as new or existing based on the
> partition
> > > > >>>> creation
> > > > >>>>>> time. Intuitively, the categorization should be based on the
> group
> > > > >>>> creation
> > > > >>>>>> time: all partitions existing when the group is created are
> > > existing
> > > > >>> and
> > > > >>>>>> all partitions created after the group creation are new
> > > partitions.
> > > > >>>>>>
> > > > >>>>>> Jun
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hi all,
> > > > >>>>>>>
> > > > >>>>>>> Manually bumping this thread. If there is no further
> > > > >>>>>>> discussion, I will close the vote.
> > > > >>>>>>>
> > > > >>>>>>> Best Regards,
> > > > >>>>>>> Jiunn-Yang
> > > > >>>>>>>
> > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > > >>>>>>>>
> > > > >>>>>>>> Hello Jian,
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks for your feedback,
> > > > >>>>>>>>
> > > > >>>>>>>> Agreed, partition expansion is a common operational task,
> not an
> > > > >>> edge
> > > > >>>>>>>> case. I've updated the Motivation section accordingly.
> > > > >>>>>>>>
> > > > >>>>>>>> Best Regards,
> > > > >>>>>>>> Jiunn-Yang
> > > > >>>>>>>>
> > > > >>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > > >>>>>>>>>
> > > > >>>>>>>>> Hi Jiunn-Yang:
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks for the KIP. I think it would be useful to clarify
> that
> > > this
> > > > >>>>>> is a
> > > > >>>>>>>>> common scenario rather than an edge case, which further
> > > > >>> demonstrates
> > > > >>>>>> the
> > > > >>>>>>>>> need for this optimization. For example:
> > > > >>>>>>>>> A partition expansion is a common operational task in
> Kafka: To
> > > > >>>>>> balance
> > > > >>>>>>>>> resource utilization and cost, topics are typically created
> > > with a
> > > > >>>>>>> moderate
> > > > >>>>>>>>> default partition count. However, as traffic grows over
> time,
> > > it is
> > > > >>>>>>> often
> > > > >>>>>>>>> necessary to increase the number of partitions to
> accommodate
> > > the
> > > > >>>>>> higher
> > > > >>>>>>>>> workload.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Regards
> > > > >>>>>>>>> Jian
> > > > >>>>>>>>>
> > > > >>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hello chia,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Best Regards,
> > > > >>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29
> 寫道:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Hi Jiunn-Yang,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> > > > >>> describing
> > > > >>>>>>>>>>> partitions in the KIP? I understand you are using them to
> > > > >>> describe
> > > > >>>>>> the
> > > > >>>>>>>>>>> "freshness" or the users' need for the records, but
> applying
> > > > >>> these
> > > > >>>>>>> terms
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>> the partition itself feels a bit unnatural.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> After all, in this scenario, users don't really care
> whether
> > > a
> > > > >>>>>>> partition
> > > > >>>>>>>>>> is
> > > > >>>>>>>>>>> newly expanded or not. Their only expectation is that
> they
> > > won't
> > > > >>>>>>> silently
> > > > >>>>>>>>>>> lose any live records produced to the topic during their
> > > active
> > > > >>>>>>>>>> consumption.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Best, Chia-Ping
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hello Jun,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> motivation
> > > > >>>> section.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
> > > 凌晨1:12 寫道:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > > motivation for
> > > > >>>>>> the
> > > > >>>>>>>>>> KIP.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> The KIP says "The core insight is that not all
> partitions
> > > > >>> without
> > > > >>>>>> a
> > > > >>>>>>>>>>>>> committed offset are the same. A newly expanded
> partition
> > > (hot)
> > > > >>>> is
> > > > >>>>>>>>>>>>> fundamentally different from a partition the consumer
> has
> > > never
> > > > >>>>>> seen
> > > > >>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> > > partition
> > > > >>>>>>>>>>>>> fundamentally different from the cold?
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> > > > >>>> insufficient
> > > > >>>>>>>>>>>> because:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> - The calculated seek time (now() - duration) varies
> across
> > > > >>> nodes
> > > > >>>>>>> due
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> > > > >>> duration,
> > > > >>>>>>>>>>>> causing
> > > > >>>>>>>>>>>>> unnecessary reprocessing.
> > > > >>>>>>>>>>>>> - On network errors, the client recalculates the seek
> time
> > > on
> > > > >>>>>> retry,
> > > > >>>>>>>>>>>>> shifting the target timestamp forward and risking data
> > > loss."
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> However, both of these situations are rare. If these
> issues
> > > > >>>>>> persist,
> > > > >>>>>>>>>> more
> > > > >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations
> > > don't
> > > > >>>>>> need a
> > > > >>>>>>>>>>>> common
> > > > >>>>>>>>>>>>> solution. If users care about those rare situations,
> they
> > > can
> > > > >>>>>>> implement
> > > > >>>>>>>>>>>>> customized logic using
> > > > >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> s7133700@gmail.com>
> > > > >>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Hello chia,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks for the feedback,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value
> should
> > > always
> > > > >>>> be
> > > > >>>>>>>>>>>> greater
> > > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > > >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>   any
> > > > >>>>>>> field
> > > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> The earliest point at which we can detect the version
> > > mismatch
> > > > >>>> is
> > > > >>>>>>>>>> during
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>> first metadata fetch after assignment, which occurs
> inside
> > > > >>>>>> poll().
> > > > >>>>>>>>>>>>>> Therefore, the
> > > > >>>>>>>>>>>>>> user would encounter an UnsupportedVersionException
> from
> > > > >>> poll().
> > > > >>>>>>> I’ll
> > > > >>>>>>>>>>>>>> clarify this in the KIP.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
> > > 下午4:50 寫道:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> hi Jiunn
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> > > > >>> partition
> > > > >>>>>> in
> > > > >>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> > > > >>>>>>>>>> broker_current_time
> > > > >>>>>>>>>>>> -
> > > > >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> does not
> > > > >>>>>> support
> > > > >>>>>>>>>> this
> > > > >>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> If the creation time exists, the returned value
> should
> > > always
> > > > >>>> be
> > > > >>>>>>>>>>>> greater
> > > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>   any
> > > > >>>>>>> field
> > > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
> > > calling
> > > > >>>>>>>>>> `poll()`?
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > >>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on
> the
> > > > >>>>>>> discussion
> > > > >>>>>>>>>>>>>>>> feedback.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
> > > > >>> behavior
> > > > >>>>>>> from
> > > > >>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> > > > >>>>>>> auto.offset.reset
> > > > >>>>>>>>>>>>>> values
> > > > >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > > >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> > > > >>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > > > >>>>>>>>>>>>>>>> config setting
> > > > >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > > >>> auto.offset.reset
> > > > >>>>>>>>>> policy
> > > > >>>>>>>>>>>>>> continues
> > > > >>>>>>>>>>>>>>>> to apply unchanged.
> > > > >>>>>>>>>>>>>>>> - The new-partition reset behavior is represented
> by a
> > > > >>>> separate
> > > > >>>>>>>>>>>>>> internal config
> > > > >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed
> to
> > > > >>>>>> earliest).
> > > > >>>>>>>>>> This
> > > > >>>>>>>>>>>>>> decoupled design makes
> > > > >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> public
> > > > >>>>>>> user-facing
> > > > >>>>>>>>>>>>>> configuration in a future KIP.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
> > > 清晨7:46
> > > > >>> 寫道:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> hi Jun
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is
> listed
> > > > >>>> below:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a
> default
> > > > >>> value
> > > > >>>>>> of
> > > > >>>>>>>>>>>>>> earliest. It fixes the data loss from both
> by_duration and
> > > > >>>>>> latest,
> > > > >>>>>>> and
> > > > >>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>> does not change the logic of
> auto.offset.reset=earliest.
> > > > >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> > > internal
> > > > >>>>>>>>>>>>>> configuration. auto.offset.reset.new
> .partitions=earliest
> > > > >>>> already
> > > > >>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases
> of
> > > other
> > > > >>>>>>> values
> > > > >>>>>>>>>>>> in a
> > > > >>>>>>>>>>>>>> separate KIP.
> > > > >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions
> and
> > > > >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied
> to
> > > all
> > > > >>> for
> > > > >>>>>>>>>>>>>> consistency.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> WDYT?
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > > >>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a
> > > Kafka
> > > > >>>>>> topic
> > > > >>>>>>> is
> > > > >>>>>>>>>>>>>> expanded
> > > > >>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest
> auto
> > > > >>> offset
> > > > >>>>>>> reset
> > > > >>>>>>>>>>>>>> policy
> > > > >>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> > > > >>> partitions
> > > > >>>>>>> before
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > > >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> record
> > > loss
> > > > >>>>>> issue
> > > > >>>>>>>>>> could
> > > > >>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>>>> happen, right?
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> .partitions
> > > will
> > > > >>>> take
> > > > >>>>>>> the
> > > > >>>>>>>>>>>> same
> > > > >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set
> it
> > > > >>>>>>> by_duration if
> > > > >>>>>>>>>>>>>> needed.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > > > >>>>>>>>>> chia7712@apache.org
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> hi Jun
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> something
> > > > >>>>>>> important
> > > > >>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>> your
> > > > >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
> > > clarify
> > > > >>> with
> > > > >>>>>> a
> > > > >>>>>>> few
> > > > >>>>>>>>>>>>>> questions:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
> > > logic to
> > > > >>>>>>> other
> > > > >>>>>>>>>>>> reset
> > > > >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest
> or
> > > > >>>>>> by_duration
> > > > >>>>>>>>>>>> don't
> > > > >>>>>>>>>>>>>> seem
> > > > >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue
> when a
> > > > >>>>>>> partition
> > > > >>>>>>>>>> is
> > > > >>>>>>>>>>>>>> expanded.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> configure for
> > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set
> it to
> > > > >>>>>> earliest
> > > > >>>>>>> or
> > > > >>>>>>>>>>>>>> latest,
> > > > >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> > > example,
> > > > >>>>>> if a
> > > > >>>>>>>>>>>>>> consumer is
> > > > >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> created
> > > during
> > > > >>>>>> that
> > > > >>>>>>>>>>>>>> downtime,
> > > > >>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest
> when
> > > > >>>>>> resuming,
> > > > >>>>>>>>>>>> rather
> > > > >>>>>>>>>>>>>> than
> > > > >>>>>>>>>>>>>>>>>>> reading from earliest just because the partition
> is
> > > > >>>>>>> technically
> > > > >>>>>>>>>>>>>> "new" to
> > > > >>>>>>>>>>>>>>>>>>> the group.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
> > > max.age
> > > > >>>>>>> threshold.
> > > > >>>>>>>>>>>> It
> > > > >>>>>>>>>>>>>> gives
> > > > >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> genuinely
> > > > >>>> "hot/new"
> > > > >>>>>>> and
> > > > >>>>>>>>>>>>>> what is
> > > > >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
> > > weird. It
> > > > >>>>>> only
> > > > >>>>>>>>>>>>>> applies when
> > > > >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems
> that
> > > the
> > > > >>>>>>>>>> motivation
> > > > >>>>>>>>>>>>>>>>>>> equally
> > > > >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
> > > values
> > > > >>> like
> > > > >>>>>>>>>>>>>> by_duration.
> > > > >>>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> way to
> > > > >>>> control
> > > > >>>>>>>>>> newly
> > > > >>>>>>>>>>>>>> created
> > > > >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
> > > starts.
> > > > >>>>>>> Have we
> > > > >>>>>>>>>>>>>>>>>>> considered
> > > > >>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > > > >>>> .partitions?
> > > > >>>>>>> If
> > > > >>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> defaults
> > > to
> > > > >>> the
> > > > >>>>>>>>>> policy
> > > > >>>>>>>>>>>>>> used
> > > > >>>>>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > > explicitly to
> > > > >>>>>>>>>> customize
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > s7133700@gmail.com>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37
> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > > > >>>>>>> PartitionMetadata
> > > > >>>>>>>>>>>>>> struct
> > > > >>>>>>>>>>>>>>>>>>>>> gains a new
> > > > >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> computed
> > > > >>>>>>> server-side
> > > > >>>>>>>>>>>> by
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>> broker as
> > > > >>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > > >>>>>>> MembershipManager
> > > > >>>>>>>>>>>>>> detects
> > > > >>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> metadata
> > > for
> > > > >>>> the
> > > > >>>>>>>>>>>> affected
> > > > >>>>>>>>>>>>>>>>>>> topic
> > > > >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > > >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even
> if
> > > the
> > > > >>>>>> topic
> > > > >>>>>>> ID
> > > > >>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>> already
> > > > >>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> supported
> > > > >>>>>>>>>>>>>> MetadataResponse
> > > > >>>>>>>>>>>>>>>>>>>>> version via the
> > > > >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time.
> If the
> > > > >>>>>>> negotiated
> > > > >>>>>>>>>>>>>>>>>>> version is
> > > > >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> PartitionAgeMs
> > > at
> > > > >>> all
> > > > >>>>>> and
> > > > >>>>>>>>>> can
> > > > >>>>>>>>>>>>>>>>>>> throw an
> > > > >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> back to
> > > > >>> latest
> > > > >>>>>>> and
> > > > >>>>>>>>>>>>>> risking
> > > > >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> 2026年4月29日
> > > > >>>> 下午4:04
> > > > >>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> 'age'
> > > > >>>>>> resolution
> > > > >>>>>>>>>> from
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
> > > control
> > > > >>>>>> plane
> > > > >>>>>>>>>>>> clean.
> > > > >>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>> main
> > > > >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > > > >>>> inter-broker
> > > > >>>>>>>>>> clock
> > > > >>>>>>>>>>>>>> skew.
> > > > >>>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
> > > source of
> > > > >>>>>> truth
> > > > >>>>>>>>>> for
> > > > >>>>>>>>>>>>>> time.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> should be
> > > > >>>>>>> negligible.
> > > > >>>>>>>>>>>>>> Given
> > > > >>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> configured in
> > > > >>>> minutes
> > > > >>>>>>> or
> > > > >>>>>>>>>>>>>> hours, a
> > > > >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> brokers
> > > > >>> won't
> > > > >>>>>>> impact
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org>
> 於
> > > > >>>>>> 2026年4月29日
> > > > >>>>>>>>>>>> 下午3:29
> > > > >>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> previous
> > > > >>>>>>> conversation
> > > > >>>>>>>>>>>> but I
> > > > >>>>>>>>>>>>>>>>>>> took a
> > > > >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow
> with
> > > the
> > > > >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
> > > boolean;
> > > > >>> the
> > > > >>>>>>> KIP
> > > > >>>>>>>>>>>>>> states
> > > > >>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
> > > boolean
> > > > >>> is
> > > > >>>>>>> set.
> > > > >>>>>>>>>>>>>>>>>>> Implicitly,
> > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> > > > >>> partition,
> > > > >>>>>> it
> > > > >>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>>>>> issue a
> > > > >>>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive
> the
> > > ages.
> > > > >>>> Is
> > > > >>>>>>> my
> > > > >>>>>>>>>>>>>>>>>>>>> understanding
> > > > >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow
> and
> > > also
> > > > >>>>>>> explain
> > > > >>>>>>>>>>>>>> how it
> > > > >>>>>>>>>>>>>>>>>>>>> fits
> > > > >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets,
> fetch
> > > > >>>>>> offsets,
> > > > >>>>>>>>>>>> etc.).
> > > > >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> wonder
> > > if
> > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
> > > place
> > > > >>> for
> > > > >>>>>>> this
> > > > >>>>>>>>>>>> given
> > > > >>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>> a new
> > > > >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it
> > > could
> > > > >>>>>> simply
> > > > >>>>>>>>>>>>>> include
> > > > >>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > > cautious
> > > > >>>> about
> > > > >>>>>>> not
> > > > >>>>>>>>>>>>>>>>>>> overloading
> > > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> unrelated
> > > > >>>> concepts.
> > > > >>>>>>> The
> > > > >>>>>>>>>>>> API
> > > > >>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > > > >>>> partitions.
> > > > >>>>>>> The
> > > > >>>>>>>>>>>> fact
> > > > >>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
> > > Streams
> > > > >>> API
> > > > >>>>>>> also
> > > > >>>>>>>>>>>>>> suggests
> > > > >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we
> do
> > > if we
> > > > >>>>>>> want to
> > > > >>>>>>>>>>>>>>>>>>> support
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>> David
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar
> > > Basani
> > > > >>>> via
> > > > >>>>>>> dev
> > > > >>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
> > > about
> > > > >>> the
> > > > >>>>>>> gap.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > > >>>>>>> RequestPartitionAges
> > > > >>>>>>>>>>>>>> field.
> > > > >>>>>>>>>>>>>>>>>>> Can a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > > PartitionAges on
> > > > >>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > >>>>>>>>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > > recommended
> > > > >>>>>> value
> > > > >>>>>>> or
> > > > >>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>> ConfigDef
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > > > >>>>>>>>>>>> metadata.max.age.ms
> > > > >>>>>>>>>>>>>> ?
> > > > >>>>>>>>>>>>>>>>>>>>> Sizing
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> guess.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to
> Kafka
> > > > >>>>>> Streams,
> > > > >>>>>>>>>>>> would
> > > > >>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>> better
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > >>>>>> auto.offset.reset.latest.max.age
> > > > >>>>>>> to
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > >>>>>>>>>>>>>>>>>>>>> for a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it?
> This
> > > is
> > > > >>> the
> > > > >>>>>>> most
> > > > >>>>>>>>>>>>>>>>>>> familiar
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > > mistakenly
> > > > >>>>>>> configure
> > > > >>>>>>>>>>>>>> it. Or
> > > > >>>>>>>>>>>>>>>>>>>>> may be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer
> falls
> > > > >>> back
> > > > >>>>>> to
> > > > >>>>>>>>>>>>>> earliest"
> > > > >>>>>>>>>>>>>>>>>>>>> reads as
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> per-partition
> > > > >>> which
> > > > >>>>>>> isn't
> > > > >>>>>>>>>>>>>>>>>>> supported.
> > > > >>>>>>>>>>>>>>>>>>>>> May
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
> > > resolves
> > > > >>>> the
> > > > >>>>>>>>>>>> initial
> > > > >>>>>>>>>>>>>>>>>>>>> position to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > > earliest was
> > > > >>>>>>> applied
> > > > >>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> config is
> > > > >>>>>>> unchanged.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > > >>>>>>> s7133700@gmail.com>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> change.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > >>> 2026年4月28日
> > > > >>>>>>> 晚上8:03
> > > > >>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
> > > creation
> > > > >>>> time
> > > > >>>>>>> via
> > > > >>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>> Admin
> > > > >>>>>>>>>>>>>>>>>>>>> API?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> > > > >>> diagnose
> > > > >>>>>> and
> > > > >>>>>>>>>>>>>>>>>>> troubleshoot
> > > > >>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> auto.offset.reset.latest.max.age
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> > > KIP-1327
> > > > >>>>>>> Prevent
> > > > >>>>>>>>>> Hot
> > > > >>>>>>>>>>>>>> Data
> > > > >>>>>>>>>>>>>>>>>>>>> Loss
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
> > > expanded
> > > > >>>>>> (hot)
> > > > >>>>>>>>>>>>>> partitions
> > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > > > >>>> automatically
> > > > >>>>>>> fall
> > > > >>>>>>>>>>>>>> back
> > > > >>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a
> > > full
> > > > >>>>>>> historical
> > > > >>>>>>>>>>>>>>>>>>> reprocess.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello all,

Thanks for the feedback,

I have opened a new discussion thread to discuss changing the 
default reset policy from latest to by_duration:PT5S.
<https://lists.apache.org/thread/njjznyly38td4scb9bjc8v0gq7kc1yqj>

Feedback and comments are very welcome!

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@apache.org> 於 2026年7月2日 凌晨12:38 寫道:
> 
> Hi Jun,
> 
> Thanks for the feedback. I agree with your conclusion. It would be better to "make haste" with a new KIP to address this. We can focus the discussion on the following points:
> 
> 1) Should we change the default reset policy to by_duration for all consumers?
> 2) What is the optimal duration to prevent data loss while avoiding large backlogs? Note that the classic consumer has a 3-second heartbeat interval, whereas the async consumer uses 5 seconds.
> 3) What are the potential side effects of replacing latest with by_duration as the default behavior?
> 
> Best,
> Chia-Ping
> 
> On 2026/06/30 20:39:24 Jun Rao via dev wrote:
>> Hi, Chia-Ping,
>> 
>> I agree that using by_duration with a value of 5s to 10s addresses most
>> common issues with expanded partitions, given the default
>> heartbeat.interval.ms is 3 seconds. We can consider setting that as the new
>> default. The current proposal may address some other rarer cases, but it
>> feels a bit like over engineering.
>> 
>> Thanks,
>> 
>> Jun
>> 
>> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org> wrote:
>> 
>>> - Short durations fail during extended downtimes, leading to the exact
>>> silent data
>>> loss this KIP aims to prevent.
>>> 
>>> Losing the records produced to expanded partitions during a downtime is
>>> actually the expected behavior for the latest policy. This is exactly why,
>>> in the current KIP proposal, we prefer by_duration over earliest for the
>>> expanded partitions—we want to avoid forcing users to consume a massive
>>> accumulated backlog just because a partition is technically "new".
>>> 
>>> To quote my previous comment from May 15:
>>> "...if a consumer is offline for a while and a new partition is created
>>> during that downtime, the user might actually want to skip to latest when
>>> resuming, rather than reading from earliest just because the partition is
>>> technically 'new' to the group."
>>> 
>>> On 2026/06/30 12:12:47 黃竣陽 wrote:
>>>> Hello all,
>>>> 
>>>> The metadata blindness and consumer downtime windows are inherently
>>>> unpredictable, making a fixed by_duration approach unviable. No static
>>>> duration can accommodate both extremes:
>>>> 
>>>> - Short durations fail during extended downtimes, leading to the exact
>>> silent data
>>>> loss this KIP aims to prevent.
>>>> - Long durations (calibrated for worst-case scenarios) cause routine
>>> restarts to
>>>> trigger unnecessary mass replays, undermining the latest offset reset
>>> policy.
>>>> 
>>>> This is why the proposed solution uses creation time. It provides a
>>> discrete,
>>>> deterministic signal—verifying whether a partition is actually newer
>>> than the group,
>>>> rather than relying on an arbitrary time window that has no correlation
>>> with the consumer’s
>>>> actual downtime.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> I'd like to revisit Jun previous comment about using by_duration=5s. I
>>> realize now that having a small backlog is actually an acceptable
>>> trade-off, especially since the latest policy inherently brings a tiny
>>> backlog to starting consumers anyway due to the time gap between fetching
>>> the offset and actually consuming.
>>>>> 
>>>>> If we bring the simple by_duration=5s solution back to the KIP, I'd
>>> like to discuss a further step: Should we just change the default policy
>>> from latest to by_duration=5s specifically for the new consumer and share
>>> consumer?
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
>>>>>> hi Jun
>>>>>> 
>>>>>> The most important part of this story is how users should expect the
>>> data
>>>>>> they can see when using the latest or by_duration policy with expanded
>>>>>> partitions.
>>>>>> 
>>>>>> Yes, the by_duration policy can minimize data loss, but it is
>>>>>> non-deterministic, which means users will either read too many
>>> historical
>>>>>> records from existing partitions or lose some records from expanded
>>>>>> partitions.
>>>>>> 
>>>>>> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
>>>>>> understand, and that is why I preferred having a whole new policy
>>> based
>>>>>> entirely on group creation time (KIP-1282)
>>>>>> 
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>> 
>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>>>>>> 
>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the reply. I am still trying to understand the value of
>>> the new
>>>>>>> configs with the KIP.
>>>>>>> 
>>>>>>> The motivation of the KIP is that a user doesn't want to miss the
>>> data if
>>>>>>> the backlog is small. The backlog of the existing partition is easy
>>> to
>>>>>>> understand because it relates to retention time. The backlog for the
>>> new
>>>>>>> partition is a bit subtle to understand since it depends on the
>>> metadata
>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms, the user needs
>>> to
>>>>>>> understand the metadata refresh delay on the consumer side and use
>>> it to
>>>>>>> set the config.
>>>>>>> 
>>>>>>> Now, let's consider the alternative: setting the same value for the
>>>>>>> existing by_duration policy. The KIP lists three issues with this
>>> approach.
>>>>>>> 1. It computes the seek target client-side as now() - duration, which
>>>>>>> introduces clock skew across consumers and forces operators to choose
>>>>>>> overly large durations, causing unnecessary reprocessing.
>>>>>>> 2. The target timestamp is recomputed on each retry, so failed
>>>>>>> ListOffsetsRequest retries can shift the target forward and
>>> potentially
>>>>>>> miss records produced between attempts.
>>>>>>> 3. It applies uniformly to all partitions without committed offsets,
>>> and
>>>>>>> cannot distinguish newly expanded partitions from long-existing
>>> partitions
>>>>>>> newly assigned to the group, leading to unnecessary replay.
>>>>>>> 
>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
>>> buffer to
>>>>>>> the metadata refresh delay. We could also consider improving the
>>>>>>> implementation. For issue 3, the metadata refresh delay is typically
>>> low
>>>>>>> (in the order of minutes with the classic consumer and tens of
>>> seconds with
>>>>>>> the new consumer). If a user is ok with reading that much backlog
>>> for new
>>>>>>> partitions, it seems they will be ok doing the same for existing
>>>>>>> partitions.
>>>>>>> 
>>>>>>> So, instead of introducing a new config, could we just reuse the
>>> existing
>>>>>>> config with better documentation and/or implementation?
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> 
>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hello Jun,
>>>>>>>> 
>>>>>>>> You're right that group creation time is the more intuitive answer
>>> at
>>>>>>>> first glance,
>>>>>>>> the KIP's own motivation talks about partitions that "predate the
>>> group"
>>>>>>>> vs partitions
>>>>>>>> "created during group runtime," which directly points to a
>>>>>>> group-lifecycle
>>>>>>>> classifier.
>>>>>>>> I'd like to walk through why we landed on partition age, and the
>>>>>>>> trade-offs we considered.
>>>>>>>> 
>>>>>>>> We evaluated three candidate signals:
>>>>>>>> 
>>>>>>>> 1. `by_duration:5secs`
>>>>>>>> 
>>>>>>>> This covers the metadata blindness window, but has issues the KIP
>>>>>>>> currently documents
>>>>>>>> under "Why not use `by_duration`?":
>>>>>>>> 
>>>>>>>> - Client-side `now() - duration` introduces clock skew across
>>> consumers.
>>>>>>>> - `ListOffsets` retries shift the target forward, potentially
>>> missing
>>>>>>>> records produced between
>>>>>>>> attempts.
>>>>>>>> - It applies uniformly to all partitions without committed offsets,
>>>>>>>> including pre-existing partitions
>>>>>>>> newly assigned to the group, causing unnecessary replay.
>>>>>>>> 
>>>>>>>> 2. Group creation time as classifier
>>>>>>>> 
>>>>>>>> This works cleanly when the consumer is actively running. Our
>>> concern
>>>>>>>> is the idle / late-rejoin case:
>>>>>>>> 
>>>>>>>> T=0:         Group created.
>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>>>>>> T=50:        Partition added during the idle window.
>>>>>>>> T=100:       Consumer resumes.
>>>>>>>> 
>>>>>>>> Under group creation time, the new partition is classified as new
>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated data
>>> that
>>>>>>>> the consumer accepts as lost — that is precisely the contract of
>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to treat
>>>>>>>> the new partition differently; both contain backlog accumulated
>>> during
>>>>>>>> the same idle window.
>>>>>>>> 
>>>>>>>> This aligns with the "backlog is backlog” principle you raised in
>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
>>>>>>>> every other partition during the same idle period; forcing 0-backlog
>>>>>>>> tolerance only on new partitions would be inconsistent with that
>>>>>>>> tolerance.
>>>>>>>> 
>>>>>>>> 3. Partition age vs threshold
>>>>>>>> 
>>>>>>>> Partition age corresponds to the actual silent data loss window,
>>>>>>>> the gap between partition creation and the consumer’s metadata
>>>>>>>> refresh. Within this window, data loss is genuinely silent: the
>>>>>>>> consumer had no opportunity to know about the partition. Outside
>>> this
>>>>>>>> window, missing data reflects either:
>>>>>>>> 
>>>>>>>> - (a) the user’s tolerated cost of running with idle consumers, or
>>>>>>>> - (b) an operational issue to surface via monitoring, not via reset
>>>>>>> policy.
>>>>>>>> 
>>>>>>>> We did not choose partition age because it is more elegant than
>>> group
>>>>>>>> creation time — we chose it because its failure mode (requires a
>>>>>>>> threshold) is
>>>>>>>> less invasive than the failure mode of group creation time
>>> (overrides
>>>>>>>> user-stated
>>>>>>>> `latest` intent during idle periods).
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>>>>>>>>> 
>>>>>>>>> Hi Jun,
>>>>>>>>> 
>>>>>>>>> Relying on both creation times will create an inconsistent
>>> scenario. A
>>>>>>>>> consumer that lost all offsets due to a long sleep will seek to the
>>>>>>>>> beginning for the partitions created later than the group.
>>>>>>>>> 
>>>>>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
>>>>>>>> using a
>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
>>> KIP-1327
>>>>>>>> goes
>>>>>>>>> back to using flexible configurations to prevent users from falling
>>>>>>> into
>>>>>>>>> that pitfall.
>>>>>>>>> 
>>>>>>>>> Best, Chia-Ping
>>>>>>>>> 
>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>>>>>> 
>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels
>>> weird to
>>>>>>>> me.
>>>>>>>>>> It
>>>>>>>>>> categorizes partitions as new or existing based on the partition
>>>>>>>> creation
>>>>>>>>>> time. Intuitively, the categorization should be based on the group
>>>>>>>> creation
>>>>>>>>>> time: all partitions existing when the group is created are
>>> existing
>>>>>>> and
>>>>>>>>>> all partitions created after the group creation are new
>>> partitions.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi all,
>>>>>>>>>>> 
>>>>>>>>>>> Manually bumping this thread. If there is no further
>>>>>>>>>>> discussion, I will close the vote.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hello Jian,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for your feedback,
>>>>>>>>>>>> 
>>>>>>>>>>>> Agreed, partition expansion is a common operational task, not an
>>>>>>> edge
>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
>>> this
>>>>>>>>>> is a
>>>>>>>>>>>>> common scenario rather than an edge case, which further
>>>>>>> demonstrates
>>>>>>>>>> the
>>>>>>>>>>>>> need for this optimization. For example:
>>>>>>>>>>>>> A partition expansion is a common operational task in Kafka: To
>>>>>>>>>> balance
>>>>>>>>>>>>> resource utilization and cost, topics are typically created
>>> with a
>>>>>>>>>>> moderate
>>>>>>>>>>>>> default partition count. However, as traffic grows over time,
>>> it is
>>>>>>>>>>> often
>>>>>>>>>>>>> necessary to increase the number of partitions to accommodate
>>> the
>>>>>>>>>> higher
>>>>>>>>>>>>> workload.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Jian
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
>>>>>>> describing
>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to
>>>>>>> describe
>>>>>>>>>> the
>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but applying
>>>>>>> these
>>>>>>>>>>> terms
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> After all, in this scenario, users don't really care whether
>>> a
>>>>>>>>>>> partition
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that they
>>> won't
>>>>>>>>>>> silently
>>>>>>>>>>>>>>> lose any live records produced to the topic during their
>>> active
>>>>>>>>>>>>>> consumption.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
>>>>>>>> section.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
>>> 凌晨1:12 寫道:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
>>> motivation for
>>>>>>>>>> the
>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
>>>>>>> without
>>>>>>>>>> a
>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded partition
>>> (hot)
>>>>>>>> is
>>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer has
>>> never
>>>>>>>>>> seen
>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
>>> partition
>>>>>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
>>>>>>>> insufficient
>>>>>>>>>>>>>>>> because:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
>>>>>>> nodes
>>>>>>>>>>> due
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
>>>>>>> duration,
>>>>>>>>>>>>>>>> causing
>>>>>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek time
>>> on
>>>>>>>>>> retry,
>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data
>>> loss."
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> However, both of these situations are rare. If these issues
>>>>>>>>>> persist,
>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations
>>> don't
>>>>>>>>>> need a
>>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>> solution. If users care about those rare situations, they
>>> can
>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>> customized logic using
>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>> always
>>>>>>>> be
>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
>>> mismatch
>>>>>>>> is
>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
>>>>>>>>>> poll().
>>>>>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
>>>>>>> poll().
>>>>>>>>>>> I’ll
>>>>>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
>>> 下午4:50 寫道:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>>>>>>> partition
>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>>>>>>>>>> broker_current_time
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
>>>>>>>>>> support
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>> always
>>>>>>>> be
>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
>>> calling
>>>>>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
>>>>>>> behavior
>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
>>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions
>>>>>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>>>>>> auto.offset.reset
>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
>>>>>>>> separate
>>>>>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
>>>>>>>>>> earliest).
>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
>>>>>>>>>>> user-facing
>>>>>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
>>> 清晨7:46
>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
>>>>>>>> below:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default
>>>>>>> value
>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
>>>>>>>>>> latest,
>>>>>>>>>>> and
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
>>> internal
>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest
>>>>>>>> already
>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
>>> other
>>>>>>>>>>> values
>>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to
>>> all
>>>>>>> for
>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a
>>> Kafka
>>>>>>>>>> topic
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
>>>>>>> offset
>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
>>>>>>> partitions
>>>>>>>>>>> before
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record
>>> loss
>>>>>>>>>> issue
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions
>>> will
>>>>>>>> take
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
>>>>>>>>>>> by_duration if
>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
>>>>>>>>>>> important
>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
>>> clarify
>>>>>>> with
>>>>>>>>>> a
>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
>>> logic to
>>>>>>>>>>> other
>>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
>>>>>>>>>> by_duration
>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
>>>>>>>>>>> partition
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
>>>>>>>>>> earliest
>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
>>> example,
>>>>>>>>>> if a
>>>>>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
>>> during
>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
>>>>>>>>>> resuming,
>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
>>>>>>>>>>> technically
>>>>>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
>>> max.age
>>>>>>>>>>> threshold.
>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
>>>>>>>> "hot/new"
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
>>> weird. It
>>>>>>>>>> only
>>>>>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that
>>> the
>>>>>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
>>> values
>>>>>>> like
>>>>>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
>>>>>>>> control
>>>>>>>>>>>>>> newly
>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
>>> starts.
>>>>>>>>>>> Have we
>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>>>>>>>> .partitions?
>>>>>>>>>>> If
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults
>>> to
>>>>>>> the
>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
>>> explicitly to
>>>>>>>>>>>>>> customize
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
>>> s7133700@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>>>>>>>>>> PartitionMetadata
>>>>>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
>>>>>>>>>>> server-side
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>>>>>> MembershipManager
>>>>>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata
>>> for
>>>>>>>> the
>>>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if
>>> the
>>>>>>>>>> topic
>>>>>>>>>>> ID
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
>>>>>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs
>>> at
>>>>>>> all
>>>>>>>>>> and
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
>>>>>>> latest
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
>>>>>>>> 下午4:04
>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
>>>>>>>>>> resolution
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
>>> control
>>>>>>>>>> plane
>>>>>>>>>>>>>>>> clean.
>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>>>>>>>> inter-broker
>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
>>> source of
>>>>>>>>>> truth
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
>>>>>>>>>>> negligible.
>>>>>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
>>>>>>>> minutes
>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
>>>>>>> won't
>>>>>>>>>>> impact
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
>>>>>>>>>> 2026年4月29日
>>>>>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
>>>>>>>>>>> conversation
>>>>>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with
>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
>>> boolean;
>>>>>>> the
>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
>>> boolean
>>>>>>> is
>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>>>>>>> partition,
>>>>>>>>>> it
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
>>> ages.
>>>>>>>> Is
>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
>>> also
>>>>>>>>>>> explain
>>>>>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
>>>>>>>>>> offsets,
>>>>>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder
>>> if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
>>> place
>>>>>>> for
>>>>>>>>>>> this
>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it
>>> could
>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
>>> cautious
>>>>>>>> about
>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
>>>>>>>> concepts.
>>>>>>>>>>> The
>>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>>>>>>>> partitions.
>>>>>>>>>>> The
>>>>>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
>>> Streams
>>>>>>> API
>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do
>>> if we
>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar
>>> Basani
>>>>>>>> via
>>>>>>>>>>> dev
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
>>> about
>>>>>>> the
>>>>>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
>>> PartitionAges on
>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
>>> recommended
>>>>>>>>>> value
>>>>>>>>>>> or
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>>>>>>>>>> metadata.max.age.ms
>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
>>>>>>>>>> Streams,
>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This
>>> is
>>>>>>> the
>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
>>> mistakenly
>>>>>>>>>>> configure
>>>>>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
>>>>>>> back
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
>>>>>>> which
>>>>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
>>> resolves
>>>>>>>> the
>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
>>> earliest was
>>>>>>>>>>> applied
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
>>>>>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>>>>>> 2026年4月28日
>>>>>>>>>>> 晚上8:03
>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
>>> creation
>>>>>>>> time
>>>>>>>>>>> via
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
>>>>>>> diagnose
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>>> KIP-1327
>>>>>>>>>>> Prevent
>>>>>>>>>>>>>> Hot
>>>>>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
>>> expanded
>>>>>>>>>> (hot)
>>>>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>>>>>>>> automatically
>>>>>>>>>>> fall
>>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a
>>> full
>>>>>>>>>>> historical
>>>>>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
Hi Jun,

Thanks for the feedback. I agree with your conclusion. It would be better to "make haste" with a new KIP to address this. We can focus the discussion on the following points:

1) Should we change the default reset policy to by_duration for all consumers?
2) What is the optimal duration to prevent data loss while avoiding large backlogs? Note that the classic consumer has a 3-second heartbeat interval, whereas the async consumer uses 5 seconds.
3) What are the potential side effects of replacing latest with by_duration as the default behavior?

Best,
Chia-Ping

On 2026/06/30 20:39:24 Jun Rao via dev wrote:
> Hi, Chia-Ping,
> 
> I agree that using by_duration with a value of 5s to 10s addresses most
> common issues with expanded partitions, given the default
> heartbeat.interval.ms is 3 seconds. We can consider setting that as the new
> default. The current proposal may address some other rarer cases, but it
> feels a bit like over engineering.
> 
> Thanks,
> 
> Jun
> 
> On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org> wrote:
> 
> > - Short durations fail during extended downtimes, leading to the exact
> > silent data
> > loss this KIP aims to prevent.
> >
> > Losing the records produced to expanded partitions during a downtime is
> > actually the expected behavior for the latest policy. This is exactly why,
> > in the current KIP proposal, we prefer by_duration over earliest for the
> > expanded partitions—we want to avoid forcing users to consume a massive
> > accumulated backlog just because a partition is technically "new".
> >
> > To quote my previous comment from May 15:
> > "...if a consumer is offline for a while and a new partition is created
> > during that downtime, the user might actually want to skip to latest when
> > resuming, rather than reading from earliest just because the partition is
> > technically 'new' to the group."
> >
> > On 2026/06/30 12:12:47 黃竣陽 wrote:
> > > Hello all,
> > >
> > > The metadata blindness and consumer downtime windows are inherently
> > > unpredictable, making a fixed by_duration approach unviable. No static
> > > duration can accommodate both extremes:
> > >
> > > - Short durations fail during extended downtimes, leading to the exact
> > silent data
> > > loss this KIP aims to prevent.
> > > - Long durations (calibrated for worst-case scenarios) cause routine
> > restarts to
> > > trigger unnecessary mass replays, undermining the latest offset reset
> > policy.
> > >
> > > This is why the proposed solution uses creation time. It provides a
> > discrete,
> > > deterministic signal—verifying whether a partition is actually newer
> > than the group,
> > > rather than relying on an arbitrary time window that has no correlation
> > with the consumer’s
> > > actual downtime.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to revisit Jun previous comment about using by_duration=5s. I
> > realize now that having a small backlog is actually an acceptable
> > trade-off, especially since the latest policy inherently brings a tiny
> > backlog to starting consumers anyway due to the time gap between fetching
> > the offset and actually consuming.
> > > >
> > > > If we bring the simple by_duration=5s solution back to the KIP, I'd
> > like to discuss a further step: Should we just change the default policy
> > from latest to by_duration=5s specifically for the new consumer and share
> > consumer?
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > > >> hi Jun
> > > >>
> > > >> The most important part of this story is how users should expect the
> > data
> > > >> they can see when using the latest or by_duration policy with expanded
> > > >> partitions.
> > > >>
> > > >> Yes, the by_duration policy can minimize data loss, but it is
> > > >> non-deterministic, which means users will either read too many
> > historical
> > > >> records from existing partitions or lose some records from expanded
> > > >> partitions.
> > > >>
> > > >> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
> > > >> understand, and that is why I preferred having a whole new policy
> > based
> > > >> entirely on group creation time (KIP-1282)
> > > >>
> > > >> Best,
> > > >> Chia-Ping
> > > >>
> > > >> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> > > >>
> > > >>> Hi, Chia-Ping and Jiunn-Yang,
> > > >>>
> > > >>> Thanks for the reply. I am still trying to understand the value of
> > the new
> > > >>> configs with the KIP.
> > > >>>
> > > >>> The motivation of the KIP is that a user doesn't want to miss the
> > data if
> > > >>> the backlog is small. The backlog of the existing partition is easy
> > to
> > > >>> understand because it relates to retention time. The backlog for the
> > new
> > > >>> partition is a bit subtle to understand since it depends on the
> > metadata
> > > >>> refresh delay. To set auto.offset.reset.max.age.ms, the user needs
> > to
> > > >>> understand the metadata refresh delay on the consumer side and use
> > it to
> > > >>> set the config.
> > > >>>
> > > >>> Now, let's consider the alternative: setting the same value for the
> > > >>> existing by_duration policy. The KIP lists three issues with this
> > approach.
> > > >>> 1. It computes the seek target client-side as now() - duration, which
> > > >>> introduces clock skew across consumers and forces operators to choose
> > > >>> overly large durations, causing unnecessary reprocessing.
> > > >>> 2. The target timestamp is recomputed on each retry, so failed
> > > >>> ListOffsetsRequest retries can shift the target forward and
> > potentially
> > > >>> miss records produced between attempts.
> > > >>> 3. It applies uniformly to all partitions without committed offsets,
> > and
> > > >>> cannot distinguish newly expanded partitions from long-existing
> > partitions
> > > >>> newly assigned to the group, leading to unnecessary replay.
> > > >>>
> > > >>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
> > buffer to
> > > >>> the metadata refresh delay. We could also consider improving the
> > > >>> implementation. For issue 3, the metadata refresh delay is typically
> > low
> > > >>> (in the order of minutes with the classic consumer and tens of
> > seconds with
> > > >>> the new consumer). If a user is ok with reading that much backlog
> > for new
> > > >>> partitions, it seems they will be ok doing the same for existing
> > > >>> partitions.
> > > >>>
> > > >>> So, instead of introducing a new config, could we just reuse the
> > existing
> > > >>> config with better documentation and/or implementation?
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>>
> > > >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>>
> > > >>>> Hello Jun,
> > > >>>>
> > > >>>> You're right that group creation time is the more intuitive answer
> > at
> > > >>>> first glance,
> > > >>>> the KIP's own motivation talks about partitions that "predate the
> > group"
> > > >>>> vs partitions
> > > >>>> "created during group runtime," which directly points to a
> > > >>> group-lifecycle
> > > >>>> classifier.
> > > >>>> I'd like to walk through why we landed on partition age, and the
> > > >>>> trade-offs we considered.
> > > >>>>
> > > >>>> We evaluated three candidate signals:
> > > >>>>
> > > >>>> 1. `by_duration:5secs`
> > > >>>>
> > > >>>> This covers the metadata blindness window, but has issues the KIP
> > > >>>> currently documents
> > > >>>> under "Why not use `by_duration`?":
> > > >>>>
> > > >>>> - Client-side `now() - duration` introduces clock skew across
> > consumers.
> > > >>>> - `ListOffsets` retries shift the target forward, potentially
> > missing
> > > >>>> records produced between
> > > >>>> attempts.
> > > >>>> - It applies uniformly to all partitions without committed offsets,
> > > >>>> including pre-existing partitions
> > > >>>> newly assigned to the group, causing unnecessary replay.
> > > >>>>
> > > >>>> 2. Group creation time as classifier
> > > >>>>
> > > >>>> This works cleanly when the consumer is actively running. Our
> > concern
> > > >>>> is the idle / late-rejoin case:
> > > >>>>
> > > >>>> T=0:         Group created.
> > > >>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > >>>> T=50:        Partition added during the idle window.
> > > >>>> T=100:       Consumer resumes.
> > > >>>>
> > > >>>> Under group creation time, the new partition is classified as new
> > > >>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
> > > >>>> But during `[T=1, T=100]`, base partitions also accumulated data
> > that
> > > >>>> the consumer accepts as lost — that is precisely the contract of
> > > >>>> `auto.offset.reset=latest`. There is no principled reason to treat
> > > >>>> the new partition differently; both contain backlog accumulated
> > during
> > > >>>> the same idle window.
> > > >>>>
> > > >>>> This aligns with the "backlog is backlog” principle you raised in
> > > >>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
> > > >>>> every other partition during the same idle period; forcing 0-backlog
> > > >>>> tolerance only on new partitions would be inconsistent with that
> > > >>>> tolerance.
> > > >>>>
> > > >>>> 3. Partition age vs threshold
> > > >>>>
> > > >>>> Partition age corresponds to the actual silent data loss window,
> > > >>>> the gap between partition creation and the consumer’s metadata
> > > >>>> refresh. Within this window, data loss is genuinely silent: the
> > > >>>> consumer had no opportunity to know about the partition. Outside
> > this
> > > >>>> window, missing data reflects either:
> > > >>>>
> > > >>>> - (a) the user’s tolerated cost of running with idle consumers, or
> > > >>>> - (b) an operational issue to surface via monitoring, not via reset
> > > >>> policy.
> > > >>>>
> > > >>>> We did not choose partition age because it is more elegant than
> > group
> > > >>>> creation time — we chose it because its failure mode (requires a
> > > >>>> threshold) is
> > > >>>> less invasive than the failure mode of group creation time
> > (overrides
> > > >>>> user-stated
> > > >>>> `latest` intent during idle periods).
> > > >>>>
> > > >>>> Best Regards,
> > > >>>> Jiunn-Yang
> > > >>>>
> > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > > >>>>>
> > > >>>>> Hi Jun,
> > > >>>>>
> > > >>>>> Relying on both creation times will create an inconsistent
> > scenario. A
> > > >>>>> consumer that lost all offsets due to a long sleep will seek to the
> > > >>>>> beginning for the partitions created later than the group.
> > > >>>>>
> > > >>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
> > > >>>> using a
> > > >>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
> > KIP-1327
> > > >>>> goes
> > > >>>>> back to using flexible configurations to prevent users from falling
> > > >>> into
> > > >>>>> that pitfall.
> > > >>>>>
> > > >>>>> Best, Chia-Ping
> > > >>>>>
> > > >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> > > >>>>>
> > > >>>>>> Hi, Jiunn-Yang,
> > > >>>>>>
> > > >>>>>> Thanks for the reply and sorry for the late reply.
> > > >>>>>>
> > > >>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels
> > weird to
> > > >>>> me.
> > > >>>>>> It
> > > >>>>>> categorizes partitions as new or existing based on the partition
> > > >>>> creation
> > > >>>>>> time. Intuitively, the categorization should be based on the group
> > > >>>> creation
> > > >>>>>> time: all partitions existing when the group is created are
> > existing
> > > >>> and
> > > >>>>>> all partitions created after the group creation are new
> > partitions.
> > > >>>>>>
> > > >>>>>> Jun
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> Hi all,
> > > >>>>>>>
> > > >>>>>>> Manually bumping this thread. If there is no further
> > > >>>>>>> discussion, I will close the vote.
> > > >>>>>>>
> > > >>>>>>> Best Regards,
> > > >>>>>>> Jiunn-Yang
> > > >>>>>>>
> > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > >>>>>>>>
> > > >>>>>>>> Hello Jian,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for your feedback,
> > > >>>>>>>>
> > > >>>>>>>> Agreed, partition expansion is a common operational task, not an
> > > >>> edge
> > > >>>>>>>> case. I've updated the Motivation section accordingly.
> > > >>>>>>>>
> > > >>>>>>>> Best Regards,
> > > >>>>>>>> Jiunn-Yang
> > > >>>>>>>>
> > > >>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > >>>>>>>>>
> > > >>>>>>>>> Hi Jiunn-Yang:
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
> > this
> > > >>>>>> is a
> > > >>>>>>>>> common scenario rather than an edge case, which further
> > > >>> demonstrates
> > > >>>>>> the
> > > >>>>>>>>> need for this optimization. For example:
> > > >>>>>>>>> A partition expansion is a common operational task in Kafka: To
> > > >>>>>> balance
> > > >>>>>>>>> resource utilization and cost, topics are typically created
> > with a
> > > >>>>>>> moderate
> > > >>>>>>>>> default partition count. However, as traffic grows over time,
> > it is
> > > >>>>>>> often
> > > >>>>>>>>> necessary to increase the number of partitions to accommodate
> > the
> > > >>>>>> higher
> > > >>>>>>>>> workload.
> > > >>>>>>>>>
> > > >>>>>>>>> Regards
> > > >>>>>>>>> Jian
> > > >>>>>>>>>
> > > >>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hello chia,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best Regards,
> > > >>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi Jiunn-Yang,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> > > >>> describing
> > > >>>>>>>>>>> partitions in the KIP? I understand you are using them to
> > > >>> describe
> > > >>>>>> the
> > > >>>>>>>>>>> "freshness" or the users' need for the records, but applying
> > > >>> these
> > > >>>>>>> terms
> > > >>>>>>>>>> to
> > > >>>>>>>>>>> the partition itself feels a bit unnatural.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> After all, in this scenario, users don't really care whether
> > a
> > > >>>>>>> partition
> > > >>>>>>>>>> is
> > > >>>>>>>>>>> newly expanded or not. Their only expectation is that they
> > won't
> > > >>>>>>> silently
> > > >>>>>>>>>>> lose any live records produced to the topic during their
> > active
> > > >>>>>>>>>> consumption.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best, Chia-Ping
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
> > > >>>> section.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
> > 凌晨1:12 寫道:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > motivation for
> > > >>>>>> the
> > > >>>>>>>>>> KIP.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
> > > >>> without
> > > >>>>>> a
> > > >>>>>>>>>>>>> committed offset are the same. A newly expanded partition
> > (hot)
> > > >>>> is
> > > >>>>>>>>>>>>> fundamentally different from a partition the consumer has
> > never
> > > >>>>>> seen
> > > >>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> > partition
> > > >>>>>>>>>>>>> fundamentally different from the cold?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> > > >>>> insufficient
> > > >>>>>>>>>>>> because:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
> > > >>> nodes
> > > >>>>>>> due
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> > > >>> duration,
> > > >>>>>>>>>>>> causing
> > > >>>>>>>>>>>>> unnecessary reprocessing.
> > > >>>>>>>>>>>>> - On network errors, the client recalculates the seek time
> > on
> > > >>>>>> retry,
> > > >>>>>>>>>>>>> shifting the target timestamp forward and risking data
> > loss."
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> However, both of these situations are rare. If these issues
> > > >>>>>> persist,
> > > >>>>>>>>>> more
> > > >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations
> > don't
> > > >>>>>> need a
> > > >>>>>>>>>>>> common
> > > >>>>>>>>>>>>> solution. If users care about those rare situations, they
> > can
> > > >>>>>>> implement
> > > >>>>>>>>>>>>> customized logic using
> > > >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
> > > >>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hello chia,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for the feedback,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> If the creation time exists, the returned value should
> > always
> > > >>>> be
> > > >>>>>>>>>>>> greater
> > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> > > >>>>>>> field
> > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> The earliest point at which we can detect the version
> > mismatch
> > > >>>> is
> > > >>>>>>>>>> during
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
> > > >>>>>> poll().
> > > >>>>>>>>>>>>>> Therefore, the
> > > >>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
> > > >>> poll().
> > > >>>>>>> I’ll
> > > >>>>>>>>>>>>>> clarify this in the KIP.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
> > 下午4:50 寫道:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> hi Jiunn
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> > > >>> partition
> > > >>>>>> in
> > > >>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> > > >>>>>>>>>> broker_current_time
> > > >>>>>>>>>>>> -
> > > >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
> > > >>>>>> support
> > > >>>>>>>>>> this
> > > >>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> If the creation time exists, the returned value should
> > always
> > > >>>> be
> > > >>>>>>>>>>>> greater
> > > >>>>>>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> > > >>>>>>> field
> > > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
> > calling
> > > >>>>>>>>>> `poll()`?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>> Hello Jun, chia,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
> > > >>>>>>> discussion
> > > >>>>>>>>>>>>>>>> feedback.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
> > > >>> behavior
> > > >>>>>>> from
> > > >>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> > > >>>>>>> auto.offset.reset
> > > >>>>>>>>>>>>>> values
> > > >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> > > >>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > > >>>>>>>>>>>>>>>> config setting
> > > >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > >>> auto.offset.reset
> > > >>>>>>>>>> policy
> > > >>>>>>>>>>>>>> continues
> > > >>>>>>>>>>>>>>>> to apply unchanged.
> > > >>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
> > > >>>> separate
> > > >>>>>>>>>>>>>> internal config
> > > >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
> > > >>>>>> earliest).
> > > >>>>>>>>>> This
> > > >>>>>>>>>>>>>> decoupled design makes
> > > >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
> > > >>>>>>> user-facing
> > > >>>>>>>>>>>>>> configuration in a future KIP.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
> > 清晨7:46
> > > >>> 寫道:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
> > > >>>> below:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default
> > > >>> value
> > > >>>>>> of
> > > >>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
> > > >>>>>> latest,
> > > >>>>>>> and
> > > >>>>>>>>>>>> it
> > > >>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
> > > >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> > internal
> > > >>>>>>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest
> > > >>>> already
> > > >>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
> > other
> > > >>>>>>> values
> > > >>>>>>>>>>>> in a
> > > >>>>>>>>>>>>>> separate KIP.
> > > >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> > > >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to
> > all
> > > >>> for
> > > >>>>>>>>>>>>>> consistency.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a
> > Kafka
> > > >>>>>> topic
> > > >>>>>>> is
> > > >>>>>>>>>>>>>> expanded
> > > >>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
> > > >>> offset
> > > >>>>>>> reset
> > > >>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> > > >>> partitions
> > > >>>>>>> before
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record
> > loss
> > > >>>>>> issue
> > > >>>>>>>>>> could
> > > >>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>> happen, right?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions
> > will
> > > >>>> take
> > > >>>>>>> the
> > > >>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
> > > >>>>>>> by_duration if
> > > >>>>>>>>>>>>>> needed.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > > >>>>>>>>>> chia7712@apache.org
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
> > > >>>>>>> important
> > > >>>>>>>>>>>> from
> > > >>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
> > clarify
> > > >>> with
> > > >>>>>> a
> > > >>>>>>> few
> > > >>>>>>>>>>>>>> questions:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
> > logic to
> > > >>>>>>> other
> > > >>>>>>>>>>>> reset
> > > >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> > > >>>>>> by_duration
> > > >>>>>>>>>>>> don't
> > > >>>>>>>>>>>>>> seem
> > > >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
> > > >>>>>>> partition
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>>>> expanded.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
> > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
> > > >>>>>> earliest
> > > >>>>>>> or
> > > >>>>>>>>>>>>>> latest,
> > > >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> > example,
> > > >>>>>> if a
> > > >>>>>>>>>>>>>> consumer is
> > > >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
> > during
> > > >>>>>> that
> > > >>>>>>>>>>>>>> downtime,
> > > >>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
> > > >>>>>> resuming,
> > > >>>>>>>>>>>> rather
> > > >>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
> > > >>>>>>> technically
> > > >>>>>>>>>>>>>> "new" to
> > > >>>>>>>>>>>>>>>>>>> the group.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
> > max.age
> > > >>>>>>> threshold.
> > > >>>>>>>>>>>> It
> > > >>>>>>>>>>>>>> gives
> > > >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
> > > >>>> "hot/new"
> > > >>>>>>> and
> > > >>>>>>>>>>>>>> what is
> > > >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
> > weird. It
> > > >>>>>> only
> > > >>>>>>>>>>>>>> applies when
> > > >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that
> > the
> > > >>>>>>>>>> motivation
> > > >>>>>>>>>>>>>>>>>>> equally
> > > >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
> > values
> > > >>> like
> > > >>>>>>>>>>>>>> by_duration.
> > > >>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
> > > >>>> control
> > > >>>>>>>>>> newly
> > > >>>>>>>>>>>>>> created
> > > >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
> > starts.
> > > >>>>>>> Have we
> > > >>>>>>>>>>>>>>>>>>> considered
> > > >>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > > >>>> .partitions?
> > > >>>>>>> If
> > > >>>>>>>>>>>> this
> > > >>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults
> > to
> > > >>> the
> > > >>>>>>>>>> policy
> > > >>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > explicitly to
> > > >>>>>>>>>> customize
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > s7133700@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > > >>>>>>> PartitionMetadata
> > > >>>>>>>>>>>>>> struct
> > > >>>>>>>>>>>>>>>>>>>>> gains a new
> > > >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> > > >>>>>>> server-side
> > > >>>>>>>>>>>> by
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> broker as
> > > >>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > >>>>>>> MembershipManager
> > > >>>>>>>>>>>>>> detects
> > > >>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>> newly assigned
> > > >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata
> > for
> > > >>>> the
> > > >>>>>>>>>>>> affected
> > > >>>>>>>>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if
> > the
> > > >>>>>> topic
> > > >>>>>>> ID
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>> in the cache.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> MB0:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> > > >>>>>>>>>>>>>> MetadataResponse
> > > >>>>>>>>>>>>>>>>>>>>> version via the
> > > >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> > > >>>>>>> negotiated
> > > >>>>>>>>>>>>>>>>>>> version is
> > > >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs
> > at
> > > >>> all
> > > >>>>>> and
> > > >>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>> throw an
> > > >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
> > > >>> latest
> > > >>>>>>> and
> > > >>>>>>>>>>>>>> risking
> > > >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
> > > >>>> 下午4:04
> > > >>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> hi David
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> > > >>>>>> resolution
> > > >>>>>>>>>> from
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
> > control
> > > >>>>>> plane
> > > >>>>>>>>>>>> clean.
> > > >>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>> main
> > > >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > > >>>> inter-broker
> > > >>>>>>>>>> clock
> > > >>>>>>>>>>>>>> skew.
> > > >>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
> > source of
> > > >>>>>> truth
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>> time.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
> > > >>>>>>> negligible.
> > > >>>>>>>>>>>>>> Given
> > > >>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
> > > >>>> minutes
> > > >>>>>>> or
> > > >>>>>>>>>>>>>> hours, a
> > > >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
> > > >>> won't
> > > >>>>>>> impact
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> > > >>>>>> 2026年4月29日
> > > >>>>>>>>>>>> 下午3:29
> > > >>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> > > >>>>>>> conversation
> > > >>>>>>>>>>>> but I
> > > >>>>>>>>>>>>>>>>>>> took a
> > > >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with
> > the
> > > >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
> > boolean;
> > > >>> the
> > > >>>>>>> KIP
> > > >>>>>>>>>>>>>> states
> > > >>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
> > boolean
> > > >>> is
> > > >>>>>>> set.
> > > >>>>>>>>>>>>>>>>>>> Implicitly,
> > > >>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> > > >>> partition,
> > > >>>>>> it
> > > >>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>> issue a
> > > >>>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
> > ages.
> > > >>>> Is
> > > >>>>>>> my
> > > >>>>>>>>>>>>>>>>>>>>> understanding
> > > >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
> > also
> > > >>>>>>> explain
> > > >>>>>>>>>>>>>> how it
> > > >>>>>>>>>>>>>>>>>>>>> fits
> > > >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> > > >>>>>> offsets,
> > > >>>>>>>>>>>> etc.).
> > > >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder
> > if
> > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
> > place
> > > >>> for
> > > >>>>>>> this
> > > >>>>>>>>>>>> given
> > > >>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> a new
> > > >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it
> > could
> > > >>>>>> simply
> > > >>>>>>>>>>>>>> include
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > cautious
> > > >>>> about
> > > >>>>>>> not
> > > >>>>>>>>>>>>>>>>>>> overloading
> > > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
> > > >>>> concepts.
> > > >>>>>>> The
> > > >>>>>>>>>>>> API
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > > >>>> partitions.
> > > >>>>>>> The
> > > >>>>>>>>>>>> fact
> > > >>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
> > Streams
> > > >>> API
> > > >>>>>>> also
> > > >>>>>>>>>>>>>> suggests
> > > >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do
> > if we
> > > >>>>>>> want to
> > > >>>>>>>>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>> David
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar
> > Basani
> > > >>>> via
> > > >>>>>>> dev
> > > >>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
> > about
> > > >>> the
> > > >>>>>>> gap.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > >>>>>>> RequestPartitionAges
> > > >>>>>>>>>>>>>> field.
> > > >>>>>>>>>>>>>>>>>>> Can a
> > > >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > PartitionAges on
> > > >>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > >>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > recommended
> > > >>>>>> value
> > > >>>>>>> or
> > > >>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>> ConfigDef
> > > >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > > >>>>>>>>>>>> metadata.max.age.ms
> > > >>>>>>>>>>>>>> ?
> > > >>>>>>>>>>>>>>>>>>>>> Sizing
> > > >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> > > >>>>>> Streams,
> > > >>>>>>>>>>>> would
> > > >>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>> better
> > > >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > >>>>>> auto.offset.reset.latest.max.age
> > > >>>>>>> to
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > >>>>>>>>>>>>>>>>>>>>> for a
> > > >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This
> > is
> > > >>> the
> > > >>>>>>> most
> > > >>>>>>>>>>>>>>>>>>> familiar
> > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > mistakenly
> > > >>>>>>> configure
> > > >>>>>>>>>>>>>> it. Or
> > > >>>>>>>>>>>>>>>>>>>>> may be
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
> > > >>> back
> > > >>>>>> to
> > > >>>>>>>>>>>>>> earliest"
> > > >>>>>>>>>>>>>>>>>>>>> reads as
> > > >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
> > > >>> which
> > > >>>>>>> isn't
> > > >>>>>>>>>>>>>>>>>>> supported.
> > > >>>>>>>>>>>>>>>>>>>>> May
> > > >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
> > resolves
> > > >>>> the
> > > >>>>>>>>>>>> initial
> > > >>>>>>>>>>>>>>>>>>>>> position to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > earliest was
> > > >>>>>>> applied
> > > >>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> > > >>>>>>> unchanged.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > >>>>>>> s7133700@gmail.com>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > >>> 2026年4月28日
> > > >>>>>>> 晚上8:03
> > > >>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
> > creation
> > > >>>> time
> > > >>>>>>> via
> > > >>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> Admin
> > > >>>>>>>>>>>>>>>>>>>>> API?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> > > >>> diagnose
> > > >>>>>> and
> > > >>>>>>>>>>>>>>>>>>> troubleshoot
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> > KIP-1327
> > > >>>>>>> Prevent
> > > >>>>>>>>>> Hot
> > > >>>>>>>>>>>>>> Data
> > > >>>>>>>>>>>>>>>>>>>>> Loss
> > > >>>>>>>>>>>>>>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > >>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
> > expanded
> > > >>>>>> (hot)
> > > >>>>>>>>>>>>>> partitions
> > > >>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > > >>>> automatically
> > > >>>>>>> fall
> > > >>>>>>>>>>>>>> back
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a
> > full
> > > >>>>>>> historical
> > > >>>>>>>>>>>>>>>>>>> reprocess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> > >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping,

I agree that using by_duration with a value of 5s to 10s addresses most
common issues with expanded partitions, given the default
heartbeat.interval.ms is 3 seconds. We can consider setting that as the new
default. The current proposal may address some other rarer cases, but it
feels a bit like over engineering.

Thanks,

Jun

On Tue, Jun 30, 2026 at 5:58 AM Chia-Ping Tsai <ch...@apache.org> wrote:

> - Short durations fail during extended downtimes, leading to the exact
> silent data
> loss this KIP aims to prevent.
>
> Losing the records produced to expanded partitions during a downtime is
> actually the expected behavior for the latest policy. This is exactly why,
> in the current KIP proposal, we prefer by_duration over earliest for the
> expanded partitions—we want to avoid forcing users to consume a massive
> accumulated backlog just because a partition is technically "new".
>
> To quote my previous comment from May 15:
> "...if a consumer is offline for a while and a new partition is created
> during that downtime, the user might actually want to skip to latest when
> resuming, rather than reading from earliest just because the partition is
> technically 'new' to the group."
>
> On 2026/06/30 12:12:47 黃竣陽 wrote:
> > Hello all,
> >
> > The metadata blindness and consumer downtime windows are inherently
> > unpredictable, making a fixed by_duration approach unviable. No static
> > duration can accommodate both extremes:
> >
> > - Short durations fail during extended downtimes, leading to the exact
> silent data
> > loss this KIP aims to prevent.
> > - Long durations (calibrated for worst-case scenarios) cause routine
> restarts to
> > trigger unnecessary mass replays, undermining the latest offset reset
> policy.
> >
> > This is why the proposed solution uses creation time. It provides a
> discrete,
> > deterministic signal—verifying whether a partition is actually newer
> than the group,
> > rather than relying on an arbitrary time window that has no correlation
> with the consumer’s
> > actual downtime.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > >
> > > Hi all,
> > >
> > > I'd like to revisit Jun previous comment about using by_duration=5s. I
> realize now that having a small backlog is actually an acceptable
> trade-off, especially since the latest policy inherently brings a tiny
> backlog to starting consumers anyway due to the time gap between fetching
> the offset and actually consuming.
> > >
> > > If we bring the simple by_duration=5s solution back to the KIP, I'd
> like to discuss a further step: Should we just change the default policy
> from latest to by_duration=5s specifically for the new consumer and share
> consumer?
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> > >> hi Jun
> > >>
> > >> The most important part of this story is how users should expect the
> data
> > >> they can see when using the latest or by_duration policy with expanded
> > >> partitions.
> > >>
> > >> Yes, the by_duration policy can minimize data loss, but it is
> > >> non-deterministic, which means users will either read too many
> historical
> > >> records from existing partitions or lose some records from expanded
> > >> partitions.
> > >>
> > >> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
> > >> understand, and that is why I preferred having a whole new policy
> based
> > >> entirely on group creation time (KIP-1282)
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> > >>
> > >>> Hi, Chia-Ping and Jiunn-Yang,
> > >>>
> > >>> Thanks for the reply. I am still trying to understand the value of
> the new
> > >>> configs with the KIP.
> > >>>
> > >>> The motivation of the KIP is that a user doesn't want to miss the
> data if
> > >>> the backlog is small. The backlog of the existing partition is easy
> to
> > >>> understand because it relates to retention time. The backlog for the
> new
> > >>> partition is a bit subtle to understand since it depends on the
> metadata
> > >>> refresh delay. To set auto.offset.reset.max.age.ms, the user needs
> to
> > >>> understand the metadata refresh delay on the consumer side and use
> it to
> > >>> set the config.
> > >>>
> > >>> Now, let's consider the alternative: setting the same value for the
> > >>> existing by_duration policy. The KIP lists three issues with this
> approach.
> > >>> 1. It computes the seek target client-side as now() - duration, which
> > >>> introduces clock skew across consumers and forces operators to choose
> > >>> overly large durations, causing unnecessary reprocessing.
> > >>> 2. The target timestamp is recomputed on each retry, so failed
> > >>> ListOffsetsRequest retries can shift the target forward and
> potentially
> > >>> miss records produced between attempts.
> > >>> 3. It applies uniformly to all partitions without committed offsets,
> and
> > >>> cannot distinguish newly expanded partitions from long-existing
> partitions
> > >>> newly assigned to the group, leading to unnecessary replay.
> > >>>
> > >>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
> buffer to
> > >>> the metadata refresh delay. We could also consider improving the
> > >>> implementation. For issue 3, the metadata refresh delay is typically
> low
> > >>> (in the order of minutes with the classic consumer and tens of
> seconds with
> > >>> the new consumer). If a user is ok with reading that much backlog
> for new
> > >>> partitions, it seems they will be ok doing the same for existing
> > >>> partitions.
> > >>>
> > >>> So, instead of introducing a new config, could we just reuse the
> existing
> > >>> config with better documentation and/or implementation?
> > >>>
> > >>> Jun
> > >>>
> > >>>
> > >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>
> > >>>> Hello Jun,
> > >>>>
> > >>>> You're right that group creation time is the more intuitive answer
> at
> > >>>> first glance,
> > >>>> the KIP's own motivation talks about partitions that "predate the
> group"
> > >>>> vs partitions
> > >>>> "created during group runtime," which directly points to a
> > >>> group-lifecycle
> > >>>> classifier.
> > >>>> I'd like to walk through why we landed on partition age, and the
> > >>>> trade-offs we considered.
> > >>>>
> > >>>> We evaluated three candidate signals:
> > >>>>
> > >>>> 1. `by_duration:5secs`
> > >>>>
> > >>>> This covers the metadata blindness window, but has issues the KIP
> > >>>> currently documents
> > >>>> under "Why not use `by_duration`?":
> > >>>>
> > >>>> - Client-side `now() - duration` introduces clock skew across
> consumers.
> > >>>> - `ListOffsets` retries shift the target forward, potentially
> missing
> > >>>> records produced between
> > >>>> attempts.
> > >>>> - It applies uniformly to all partitions without committed offsets,
> > >>>> including pre-existing partitions
> > >>>> newly assigned to the group, causing unnecessary replay.
> > >>>>
> > >>>> 2. Group creation time as classifier
> > >>>>
> > >>>> This works cleanly when the consumer is actively running. Our
> concern
> > >>>> is the idle / late-rejoin case:
> > >>>>
> > >>>> T=0:         Group created.
> > >>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > >>>> T=50:        Partition added during the idle window.
> > >>>> T=100:       Consumer resumes.
> > >>>>
> > >>>> Under group creation time, the new partition is classified as new
> > >>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
> > >>>> But during `[T=1, T=100]`, base partitions also accumulated data
> that
> > >>>> the consumer accepts as lost — that is precisely the contract of
> > >>>> `auto.offset.reset=latest`. There is no principled reason to treat
> > >>>> the new partition differently; both contain backlog accumulated
> during
> > >>>> the same idle window.
> > >>>>
> > >>>> This aligns with the "backlog is backlog” principle you raised in
> > >>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
> > >>>> every other partition during the same idle period; forcing 0-backlog
> > >>>> tolerance only on new partitions would be inconsistent with that
> > >>>> tolerance.
> > >>>>
> > >>>> 3. Partition age vs threshold
> > >>>>
> > >>>> Partition age corresponds to the actual silent data loss window,
> > >>>> the gap between partition creation and the consumer’s metadata
> > >>>> refresh. Within this window, data loss is genuinely silent: the
> > >>>> consumer had no opportunity to know about the partition. Outside
> this
> > >>>> window, missing data reflects either:
> > >>>>
> > >>>> - (a) the user’s tolerated cost of running with idle consumers, or
> > >>>> - (b) an operational issue to surface via monitoring, not via reset
> > >>> policy.
> > >>>>
> > >>>> We did not choose partition age because it is more elegant than
> group
> > >>>> creation time — we chose it because its failure mode (requires a
> > >>>> threshold) is
> > >>>> less invasive than the failure mode of group creation time
> (overrides
> > >>>> user-stated
> > >>>> `latest` intent during idle periods).
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > >>>>>
> > >>>>> Hi Jun,
> > >>>>>
> > >>>>> Relying on both creation times will create an inconsistent
> scenario. A
> > >>>>> consumer that lost all offsets due to a long sleep will seek to the
> > >>>>> beginning for the partitions created later than the group.
> > >>>>>
> > >>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
> > >>>> using a
> > >>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
> KIP-1327
> > >>>> goes
> > >>>>> back to using flexible configurations to prevent users from falling
> > >>> into
> > >>>>> that pitfall.
> > >>>>>
> > >>>>> Best, Chia-Ping
> > >>>>>
> > >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> > >>>>>
> > >>>>>> Hi, Jiunn-Yang,
> > >>>>>>
> > >>>>>> Thanks for the reply and sorry for the late reply.
> > >>>>>>
> > >>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels
> weird to
> > >>>> me.
> > >>>>>> It
> > >>>>>> categorizes partitions as new or existing based on the partition
> > >>>> creation
> > >>>>>> time. Intuitively, the categorization should be based on the group
> > >>>> creation
> > >>>>>> time: all partitions existing when the group is created are
> existing
> > >>> and
> > >>>>>> all partitions created after the group creation are new
> partitions.
> > >>>>>>
> > >>>>>> Jun
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> Manually bumping this thread. If there is no further
> > >>>>>>> discussion, I will close the vote.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>>>
> > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > >>>>>>>>
> > >>>>>>>> Hello Jian,
> > >>>>>>>>
> > >>>>>>>> Thanks for your feedback,
> > >>>>>>>>
> > >>>>>>>> Agreed, partition expansion is a common operational task, not an
> > >>> edge
> > >>>>>>>> case. I've updated the Motivation section accordingly.
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Jiunn-Yang
> > >>>>>>>>
> > >>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > >>>>>>>>>
> > >>>>>>>>> Hi Jiunn-Yang:
> > >>>>>>>>>
> > >>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
> this
> > >>>>>> is a
> > >>>>>>>>> common scenario rather than an edge case, which further
> > >>> demonstrates
> > >>>>>> the
> > >>>>>>>>> need for this optimization. For example:
> > >>>>>>>>> A partition expansion is a common operational task in Kafka: To
> > >>>>>> balance
> > >>>>>>>>> resource utilization and cost, topics are typically created
> with a
> > >>>>>>> moderate
> > >>>>>>>>> default partition count. However, as traffic grows over time,
> it is
> > >>>>>>> often
> > >>>>>>>>> necessary to increase the number of partitions to accommodate
> the
> > >>>>>> higher
> > >>>>>>>>> workload.
> > >>>>>>>>>
> > >>>>>>>>> Regards
> > >>>>>>>>> Jian
> > >>>>>>>>>
> > >>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > >>>>>>>>>
> > >>>>>>>>>> Hello chia,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > >>>>>>>>>>
> > >>>>>>>>>> Best Regards,
> > >>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>
> > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> > >>> describing
> > >>>>>>>>>>> partitions in the KIP? I understand you are using them to
> > >>> describe
> > >>>>>> the
> > >>>>>>>>>>> "freshness" or the users' need for the records, but applying
> > >>> these
> > >>>>>>> terms
> > >>>>>>>>>> to
> > >>>>>>>>>>> the partition itself feels a bit unnatural.
> > >>>>>>>>>>>
> > >>>>>>>>>>> After all, in this scenario, users don't really care whether
> a
> > >>>>>>> partition
> > >>>>>>>>>> is
> > >>>>>>>>>>> newly expanded or not. Their only expectation is that they
> won't
> > >>>>>>> silently
> > >>>>>>>>>>> lose any live records produced to the topic during their
> active
> > >>>>>>>>>> consumption.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
> > >>>> section.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
> 凌晨1:12 寫道:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> motivation for
> > >>>>>> the
> > >>>>>>>>>> KIP.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
> > >>> without
> > >>>>>> a
> > >>>>>>>>>>>>> committed offset are the same. A newly expanded partition
> (hot)
> > >>>> is
> > >>>>>>>>>>>>> fundamentally different from a partition the consumer has
> never
> > >>>>>> seen
> > >>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> partition
> > >>>>>>>>>>>>> fundamentally different from the cold?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> > >>>> insufficient
> > >>>>>>>>>>>> because:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
> > >>> nodes
> > >>>>>>> due
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> > >>> duration,
> > >>>>>>>>>>>> causing
> > >>>>>>>>>>>>> unnecessary reprocessing.
> > >>>>>>>>>>>>> - On network errors, the client recalculates the seek time
> on
> > >>>>>> retry,
> > >>>>>>>>>>>>> shifting the target timestamp forward and risking data
> loss."
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> However, both of these situations are rare. If these issues
> > >>>>>> persist,
> > >>>>>>>>>> more
> > >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations
> don't
> > >>>>>> need a
> > >>>>>>>>>>>> common
> > >>>>>>>>>>>>> solution. If users care about those rare situations, they
> can
> > >>>>>>> implement
> > >>>>>>>>>>>>> customized logic using
> > >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
> > >>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the feedback,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> If the creation time exists, the returned value should
> always
> > >>>> be
> > >>>>>>>>>>>> greater
> > >>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> > >>>>>>> field
> > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The earliest point at which we can detect the version
> mismatch
> > >>>> is
> > >>>>>>>>>> during
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
> > >>>>>> poll().
> > >>>>>>>>>>>>>> Therefore, the
> > >>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
> > >>> poll().
> > >>>>>>> I’ll
> > >>>>>>>>>>>>>> clarify this in the KIP.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
> 下午4:50 寫道:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> hi Jiunn
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> > >>> partition
> > >>>>>> in
> > >>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> > >>>>>>>>>> broker_current_time
> > >>>>>>>>>>>> -
> > >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
> > >>>>>> support
> > >>>>>>>>>> this
> > >>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> If the creation time exists, the returned value should
> always
> > >>>> be
> > >>>>>>>>>>>> greater
> > >>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> > >>>>>>> field
> > >>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
> calling
> > >>>>>>>>>> `poll()`?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
> > >>>>>>> discussion
> > >>>>>>>>>>>>>>>> feedback.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
> > >>> behavior
> > >>>>>>> from
> > >>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> > >>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> > >>>>>>>>>>>>>> auto.offset.reset.new.partitions
> > >>>>>>>>>>>>>>>> config setting
> > >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > >>> auto.offset.reset
> > >>>>>>>>>> policy
> > >>>>>>>>>>>>>> continues
> > >>>>>>>>>>>>>>>> to apply unchanged.
> > >>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
> > >>>> separate
> > >>>>>>>>>>>>>> internal config
> > >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
> > >>>>>> earliest).
> > >>>>>>>>>> This
> > >>>>>>>>>>>>>> decoupled design makes
> > >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
> > >>>>>>> user-facing
> > >>>>>>>>>>>>>> configuration in a future KIP.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
> 清晨7:46
> > >>> 寫道:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
> > >>>> below:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default
> > >>> value
> > >>>>>> of
> > >>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
> > >>>>>> latest,
> > >>>>>>> and
> > >>>>>>>>>>>> it
> > >>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
> > >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an
> internal
> > >>>>>>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest
> > >>>> already
> > >>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
> other
> > >>>>>>> values
> > >>>>>>>>>>>> in a
> > >>>>>>>>>>>>>> separate KIP.
> > >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> > >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to
> all
> > >>> for
> > >>>>>>>>>>>>>> consistency.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a
> Kafka
> > >>>>>> topic
> > >>>>>>> is
> > >>>>>>>>>>>>>> expanded
> > >>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
> > >>> offset
> > >>>>>>> reset
> > >>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> > >>> partitions
> > >>>>>>> before
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record
> loss
> > >>>>>> issue
> > >>>>>>>>>> could
> > >>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>> happen, right?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions
> will
> > >>>> take
> > >>>>>>> the
> > >>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
> > >>>>>>> by_duration if
> > >>>>>>>>>>>>>> needed.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > >>>>>>>>>> chia7712@apache.org
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
> > >>>>>>> important
> > >>>>>>>>>>>> from
> > >>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
> clarify
> > >>> with
> > >>>>>> a
> > >>>>>>> few
> > >>>>>>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
> logic to
> > >>>>>>> other
> > >>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> > >>>>>> by_duration
> > >>>>>>>>>>>> don't
> > >>>>>>>>>>>>>> seem
> > >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
> > >>>>>>> partition
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>> expanded.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
> > >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
> > >>>>>> earliest
> > >>>>>>> or
> > >>>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> example,
> > >>>>>> if a
> > >>>>>>>>>>>>>> consumer is
> > >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
> during
> > >>>>>> that
> > >>>>>>>>>>>>>> downtime,
> > >>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
> > >>>>>> resuming,
> > >>>>>>>>>>>> rather
> > >>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
> > >>>>>>> technically
> > >>>>>>>>>>>>>> "new" to
> > >>>>>>>>>>>>>>>>>>> the group.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
> max.age
> > >>>>>>> threshold.
> > >>>>>>>>>>>> It
> > >>>>>>>>>>>>>> gives
> > >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
> > >>>> "hot/new"
> > >>>>>>> and
> > >>>>>>>>>>>>>> what is
> > >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
> weird. It
> > >>>>>> only
> > >>>>>>>>>>>>>> applies when
> > >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that
> the
> > >>>>>>>>>> motivation
> > >>>>>>>>>>>>>>>>>>> equally
> > >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
> values
> > >>> like
> > >>>>>>>>>>>>>> by_duration.
> > >>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
> > >>>> control
> > >>>>>>>>>> newly
> > >>>>>>>>>>>>>> created
> > >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
> starts.
> > >>>>>>> Have we
> > >>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > >>>> .partitions?
> > >>>>>>> If
> > >>>>>>>>>>>> this
> > >>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults
> to
> > >>> the
> > >>>>>>>>>> policy
> > >>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> explicitly to
> > >>>>>>>>>> customize
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> s7133700@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > >>>>>>> PartitionMetadata
> > >>>>>>>>>>>>>> struct
> > >>>>>>>>>>>>>>>>>>>>> gains a new
> > >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> > >>>>>>> server-side
> > >>>>>>>>>>>> by
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> broker as
> > >>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > >>>>>>> MembershipManager
> > >>>>>>>>>>>>>> detects
> > >>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>> newly assigned
> > >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata
> for
> > >>>> the
> > >>>>>>>>>>>> affected
> > >>>>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if
> the
> > >>>>>> topic
> > >>>>>>> ID
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>> in the cache.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> MB0:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> > >>>>>>>>>>>>>> MetadataResponse
> > >>>>>>>>>>>>>>>>>>>>> version via the
> > >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> > >>>>>>> negotiated
> > >>>>>>>>>>>>>>>>>>> version is
> > >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs
> at
> > >>> all
> > >>>>>> and
> > >>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>> throw an
> > >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
> > >>> latest
> > >>>>>>> and
> > >>>>>>>>>>>>>> risking
> > >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
> > >>>> 下午4:04
> > >>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> hi David
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> > >>>>>> resolution
> > >>>>>>>>>> from
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
> control
> > >>>>>> plane
> > >>>>>>>>>>>> clean.
> > >>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>> main
> > >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > >>>> inter-broker
> > >>>>>>>>>> clock
> > >>>>>>>>>>>>>> skew.
> > >>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
> source of
> > >>>>>> truth
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>> time.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
> > >>>>>>> negligible.
> > >>>>>>>>>>>>>> Given
> > >>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
> > >>>> minutes
> > >>>>>>> or
> > >>>>>>>>>>>>>> hours, a
> > >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
> > >>> won't
> > >>>>>>> impact
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> fallback decision.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> > >>>>>> 2026年4月29日
> > >>>>>>>>>>>> 下午3:29
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> > >>>>>>> conversation
> > >>>>>>>>>>>> but I
> > >>>>>>>>>>>>>>>>>>> took a
> > >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with
> the
> > >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
> boolean;
> > >>> the
> > >>>>>>> KIP
> > >>>>>>>>>>>>>> states
> > >>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
> boolean
> > >>> is
> > >>>>>>> set.
> > >>>>>>>>>>>>>>>>>>> Implicitly,
> > >>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> > >>> partition,
> > >>>>>> it
> > >>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>> issue a
> > >>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
> ages.
> > >>>> Is
> > >>>>>>> my
> > >>>>>>>>>>>>>>>>>>>>> understanding
> > >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
> also
> > >>>>>>> explain
> > >>>>>>>>>>>>>> how it
> > >>>>>>>>>>>>>>>>>>>>> fits
> > >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> > >>>>>> offsets,
> > >>>>>>>>>>>> etc.).
> > >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder
> if
> > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
> place
> > >>> for
> > >>>>>>> this
> > >>>>>>>>>>>> given
> > >>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> a new
> > >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it
> could
> > >>>>>> simply
> > >>>>>>>>>>>>>> include
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> cautious
> > >>>> about
> > >>>>>>> not
> > >>>>>>>>>>>>>>>>>>> overloading
> > >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
> > >>>> concepts.
> > >>>>>>> The
> > >>>>>>>>>>>> API
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > >>>> partitions.
> > >>>>>>> The
> > >>>>>>>>>>>> fact
> > >>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
> Streams
> > >>> API
> > >>>>>>> also
> > >>>>>>>>>>>>>> suggests
> > >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do
> if we
> > >>>>>>> want to
> > >>>>>>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar
> Basani
> > >>>> via
> > >>>>>>> dev
> > >>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
> about
> > >>> the
> > >>>>>>> gap.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > >>>>>>> RequestPartitionAges
> > >>>>>>>>>>>>>> field.
> > >>>>>>>>>>>>>>>>>>> Can a
> > >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> PartitionAges on
> > >>>>>>>>>>>>>>>>>>> TopicPartitions)
> > >>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> recommended
> > >>>>>> value
> > >>>>>>> or
> > >>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>> ConfigDef
> > >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > >>>>>>>>>>>> metadata.max.age.ms
> > >>>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>>>>>>>>> Sizing
> > >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> > >>>>>> Streams,
> > >>>>>>>>>>>> would
> > >>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > >>>>>> auto.offset.reset.latest.max.age
> > >>>>>>> to
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > >>>>>>>>>>>>>>>>>>>>> for a
> > >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This
> is
> > >>> the
> > >>>>>>> most
> > >>>>>>>>>>>>>>>>>>> familiar
> > >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> mistakenly
> > >>>>>>> configure
> > >>>>>>>>>>>>>> it. Or
> > >>>>>>>>>>>>>>>>>>>>> may be
> > >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
> > >>> back
> > >>>>>> to
> > >>>>>>>>>>>>>> earliest"
> > >>>>>>>>>>>>>>>>>>>>> reads as
> > >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
> > >>> which
> > >>>>>>> isn't
> > >>>>>>>>>>>>>>>>>>> supported.
> > >>>>>>>>>>>>>>>>>>>>> May
> > >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
> resolves
> > >>>> the
> > >>>>>>>>>>>> initial
> > >>>>>>>>>>>>>>>>>>>>> position to
> > >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> earliest was
> > >>>>>>> applied
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> > >>>>>>> unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > >>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > >>> 2026年4月28日
> > >>>>>>> 晚上8:03
> > >>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
> creation
> > >>>> time
> > >>>>>>> via
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> Admin
> > >>>>>>>>>>>>>>>>>>>>> API?
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> > >>> diagnose
> > >>>>>> and
> > >>>>>>>>>>>>>>>>>>> troubleshoot
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> KIP-1327
> > >>>>>>> Prevent
> > >>>>>>>>>> Hot
> > >>>>>>>>>>>>>> Data
> > >>>>>>>>>>>>>>>>>>>>> Loss
> > >>>>>>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > >>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
> expanded
> > >>>>>> (hot)
> > >>>>>>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > >>>> automatically
> > >>>>>>> fall
> > >>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a
> full
> > >>>>>>> historical
> > >>>>>>>>>>>>>>>>>>> reprocess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
- Short durations fail during extended downtimes, leading to the exact silent data
loss this KIP aims to prevent.

Losing the records produced to expanded partitions during a downtime is actually the expected behavior for the latest policy. This is exactly why, in the current KIP proposal, we prefer by_duration over earliest for the expanded partitions—we want to avoid forcing users to consume a massive accumulated backlog just because a partition is technically "new".

To quote my previous comment from May 15:
"...if a consumer is offline for a while and a new partition is created during that downtime, the user might actually want to skip to latest when resuming, rather than reading from earliest just because the partition is technically 'new' to the group."

On 2026/06/30 12:12:47 黃竣陽 wrote:
> Hello all,
> 
> The metadata blindness and consumer downtime windows are inherently 
> unpredictable, making a fixed by_duration approach unviable. No static 
> duration can accommodate both extremes: 
> 
> - Short durations fail during extended downtimes, leading to the exact silent data 
> loss this KIP aims to prevent. 
> - Long durations (calibrated for worst-case scenarios) cause routine restarts to 
> trigger unnecessary mass replays, undermining the latest offset reset policy. 
> 
> This is why the proposed solution uses creation time. It provides a discrete, 
> deterministic signal—verifying whether a partition is actually newer than the group, 
> rather than relying on an arbitrary time window that has no correlation with the consumer’s 
> actual downtime.
> 
> Best Regards,
> Jiunn-Yang
> 
> > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> > 
> > Hi all,
> > 
> > I'd like to revisit Jun previous comment about using by_duration=5s. I realize now that having a small backlog is actually an acceptable trade-off, especially since the latest policy inherently brings a tiny backlog to starting consumers anyway due to the time gap between fetching the offset and actually consuming.
> > 
> > If we bring the simple by_duration=5s solution back to the KIP, I'd like to discuss a further step: Should we just change the default policy from latest to by_duration=5s specifically for the new consumer and share consumer? 
> > 
> > Best,
> > Chia-Ping
> > 
> > On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> >> hi Jun
> >> 
> >> The most important part of this story is how users should expect the data
> >> they can see when using the latest or by_duration policy with expanded
> >> partitions.
> >> 
> >> Yes, the by_duration policy can minimize data loss, but it is
> >> non-deterministic, which means users will either read too many historical
> >> records from existing partitions or lose some records from expanded
> >> partitions.
> >> 
> >> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
> >> understand, and that is why I preferred having a whole new policy based
> >> entirely on group creation time (KIP-1282)
> >> 
> >> Best,
> >> Chia-Ping
> >> 
> >> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> >> 
> >>> Hi, Chia-Ping and Jiunn-Yang,
> >>> 
> >>> Thanks for the reply. I am still trying to understand the value of the new
> >>> configs with the KIP.
> >>> 
> >>> The motivation of the KIP is that a user doesn't want to miss the data if
> >>> the backlog is small. The backlog of the existing partition is easy to
> >>> understand because it relates to retention time. The backlog for the new
> >>> partition is a bit subtle to understand since it depends on the metadata
> >>> refresh delay. To set auto.offset.reset.max.age.ms, the user needs to
> >>> understand the metadata refresh delay on the consumer side and use it to
> >>> set the config.
> >>> 
> >>> Now, let's consider the alternative: setting the same value for the
> >>> existing by_duration policy. The KIP lists three issues with this approach.
> >>> 1. It computes the seek target client-side as now() - duration, which
> >>> introduces clock skew across consumers and forces operators to choose
> >>> overly large durations, causing unnecessary reprocessing.
> >>> 2. The target timestamp is recomputed on each retry, so failed
> >>> ListOffsetsRequest retries can shift the target forward and potentially
> >>> miss records produced between attempts.
> >>> 3. It applies uniformly to all partitions without committed offsets, and
> >>> cannot distinguish newly expanded partitions from long-existing partitions
> >>> newly assigned to the group, leading to unnecessary replay.
> >>> 
> >>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
> >>> the metadata refresh delay. We could also consider improving the
> >>> implementation. For issue 3, the metadata refresh delay is typically low
> >>> (in the order of minutes with the classic consumer and tens of seconds with
> >>> the new consumer). If a user is ok with reading that much backlog for new
> >>> partitions, it seems they will be ok doing the same for existing
> >>> partitions.
> >>> 
> >>> So, instead of introducing a new config, could we just reuse the existing
> >>> config with better documentation and/or implementation?
> >>> 
> >>> Jun
> >>> 
> >>> 
> >>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>> 
> >>>> Hello Jun,
> >>>> 
> >>>> You're right that group creation time is the more intuitive answer at
> >>>> first glance,
> >>>> the KIP's own motivation talks about partitions that "predate the group"
> >>>> vs partitions
> >>>> "created during group runtime," which directly points to a
> >>> group-lifecycle
> >>>> classifier.
> >>>> I'd like to walk through why we landed on partition age, and the
> >>>> trade-offs we considered.
> >>>> 
> >>>> We evaluated three candidate signals:
> >>>> 
> >>>> 1. `by_duration:5secs`
> >>>> 
> >>>> This covers the metadata blindness window, but has issues the KIP
> >>>> currently documents
> >>>> under "Why not use `by_duration`?":
> >>>> 
> >>>> - Client-side `now() - duration` introduces clock skew across consumers.
> >>>> - `ListOffsets` retries shift the target forward, potentially missing
> >>>> records produced between
> >>>> attempts.
> >>>> - It applies uniformly to all partitions without committed offsets,
> >>>> including pre-existing partitions
> >>>> newly assigned to the group, causing unnecessary replay.
> >>>> 
> >>>> 2. Group creation time as classifier
> >>>> 
> >>>> This works cleanly when the consumer is actively running. Our concern
> >>>> is the idle / late-rejoin case:
> >>>> 
> >>>> T=0:         Group created.
> >>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> >>>> T=50:        Partition added during the idle window.
> >>>> T=100:       Consumer resumes.
> >>>> 
> >>>> Under group creation time, the new partition is classified as new
> >>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
> >>>> But during `[T=1, T=100]`, base partitions also accumulated data that
> >>>> the consumer accepts as lost — that is precisely the contract of
> >>>> `auto.offset.reset=latest`. There is no principled reason to treat
> >>>> the new partition differently; both contain backlog accumulated during
> >>>> the same idle window.
> >>>> 
> >>>> This aligns with the "backlog is backlog” principle you raised in
> >>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
> >>>> every other partition during the same idle period; forcing 0-backlog
> >>>> tolerance only on new partitions would be inconsistent with that
> >>>> tolerance.
> >>>> 
> >>>> 3. Partition age vs threshold
> >>>> 
> >>>> Partition age corresponds to the actual silent data loss window,
> >>>> the gap between partition creation and the consumer’s metadata
> >>>> refresh. Within this window, data loss is genuinely silent: the
> >>>> consumer had no opportunity to know about the partition. Outside this
> >>>> window, missing data reflects either:
> >>>> 
> >>>> - (a) the user’s tolerated cost of running with idle consumers, or
> >>>> - (b) an operational issue to surface via monitoring, not via reset
> >>> policy.
> >>>> 
> >>>> We did not choose partition age because it is more elegant than group
> >>>> creation time — we chose it because its failure mode (requires a
> >>>> threshold) is
> >>>> less invasive than the failure mode of group creation time (overrides
> >>>> user-stated
> >>>> `latest` intent during idle periods).
> >>>> 
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>> 
> >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> >>>>> 
> >>>>> Hi Jun,
> >>>>> 
> >>>>> Relying on both creation times will create an inconsistent scenario. A
> >>>>> consumer that lost all offsets due to a long sleep will seek to the
> >>>>> beginning for the partitions created later than the group.
> >>>>> 
> >>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
> >>>> using a
> >>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
> >>>> goes
> >>>>> back to using flexible configurations to prevent users from falling
> >>> into
> >>>>> that pitfall.
> >>>>> 
> >>>>> Best, Chia-Ping
> >>>>> 
> >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> >>>>> 
> >>>>>> Hi, Jiunn-Yang,
> >>>>>> 
> >>>>>> Thanks for the reply and sorry for the late reply.
> >>>>>> 
> >>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels weird to
> >>>> me.
> >>>>>> It
> >>>>>> categorizes partitions as new or existing based on the partition
> >>>> creation
> >>>>>> time. Intuitively, the categorization should be based on the group
> >>>> creation
> >>>>>> time: all partitions existing when the group is created are existing
> >>> and
> >>>>>> all partitions created after the group creation are new partitions.
> >>>>>> 
> >>>>>> Jun
> >>>>>> 
> >>>>>> 
> >>>>>> 
> >>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>> 
> >>>>>>> Hi all,
> >>>>>>> 
> >>>>>>> Manually bumping this thread. If there is no further
> >>>>>>> discussion, I will close the vote.
> >>>>>>> 
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>> 
> >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> >>>>>>>> 
> >>>>>>>> Hello Jian,
> >>>>>>>> 
> >>>>>>>> Thanks for your feedback,
> >>>>>>>> 
> >>>>>>>> Agreed, partition expansion is a common operational task, not an
> >>> edge
> >>>>>>>> case. I've updated the Motivation section accordingly.
> >>>>>>>> 
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>> 
> >>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> >>>>>>>>> 
> >>>>>>>>> Hi Jiunn-Yang:
> >>>>>>>>> 
> >>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that this
> >>>>>> is a
> >>>>>>>>> common scenario rather than an edge case, which further
> >>> demonstrates
> >>>>>> the
> >>>>>>>>> need for this optimization. For example:
> >>>>>>>>> A partition expansion is a common operational task in Kafka: To
> >>>>>> balance
> >>>>>>>>> resource utilization and cost, topics are typically created with a
> >>>>>>> moderate
> >>>>>>>>> default partition count. However, as traffic grows over time, it is
> >>>>>>> often
> >>>>>>>>> necessary to increase the number of partitions to accommodate the
> >>>>>> higher
> >>>>>>>>> workload.
> >>>>>>>>> 
> >>>>>>>>> Regards
> >>>>>>>>> Jian
> >>>>>>>>> 
> >>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> >>>>>>>>> 
> >>>>>>>>>> Hello chia,
> >>>>>>>>>> 
> >>>>>>>>>> Thanks for the comments, I have updated the KIP!
> >>>>>>>>>> 
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>> 
> >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> >>>>>>>>>>> 
> >>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>> 
> >>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> >>> describing
> >>>>>>>>>>> partitions in the KIP? I understand you are using them to
> >>> describe
> >>>>>> the
> >>>>>>>>>>> "freshness" or the users' need for the records, but applying
> >>> these
> >>>>>>> terms
> >>>>>>>>>> to
> >>>>>>>>>>> the partition itself feels a bit unnatural.
> >>>>>>>>>>> 
> >>>>>>>>>>> After all, in this scenario, users don't really care whether a
> >>>>>>> partition
> >>>>>>>>>> is
> >>>>>>>>>>> newly expanded or not. Their only expectation is that they won't
> >>>>>>> silently
> >>>>>>>>>>> lose any live records produced to the topic during their active
> >>>>>>>>>> consumption.
> >>>>>>>>>>> 
> >>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >>>>>>>>>>> 
> >>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
> >>>> section.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation for
> >>>>>> the
> >>>>>>>>>> KIP.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
> >>> without
> >>>>>> a
> >>>>>>>>>>>>> committed offset are the same. A newly expanded partition (hot)
> >>>> is
> >>>>>>>>>>>>> fundamentally different from a partition the consumer has never
> >>>>>> seen
> >>>>>>>>>>>>> because it predates the group (cold)." Why is the hot partition
> >>>>>>>>>>>>> fundamentally different from the cold?
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> >>>> insufficient
> >>>>>>>>>>>> because:
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
> >>> nodes
> >>>>>>> due
> >>>>>>>>>>>> to
> >>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> >>> duration,
> >>>>>>>>>>>> causing
> >>>>>>>>>>>>> unnecessary reprocessing.
> >>>>>>>>>>>>> - On network errors, the client recalculates the seek time on
> >>>>>> retry,
> >>>>>>>>>>>>> shifting the target timestamp forward and risking data loss."
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> However, both of these situations are rare. If these issues
> >>>>>> persist,
> >>>>>>>>>> more
> >>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
> >>>>>> need a
> >>>>>>>>>>>> common
> >>>>>>>>>>>>> solution. If users care about those rare situations, they can
> >>>>>>> implement
> >>>>>>>>>>>>> customized logic using
> >>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
> >>> wrote:
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Thanks for the feedback,
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> If the creation time exists, the returned value should always
> >>>> be
> >>>>>>>>>>>> greater
> >>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >>>>>>> field
> >>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> The earliest point at which we can detect the version mismatch
> >>>> is
> >>>>>>>>>> during
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
> >>>>>> poll().
> >>>>>>>>>>>>>> Therefore, the
> >>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
> >>> poll().
> >>>>>>> I’ll
> >>>>>>>>>>>>>> clarify this in the KIP.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> >>> partition
> >>>>>> in
> >>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> >>>>>>>>>> broker_current_time
> >>>>>>>>>>>> -
> >>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
> >>>>>> support
> >>>>>>>>>> this
> >>>>>>>>>>>>>> feature or the partition creation time is unknown.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> If the creation time exists, the returned value should always
> >>>> be
> >>>>>>>>>>>> greater
> >>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >>>>>>> field
> >>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
> >>>>>>>>>> `poll()`?
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>>>>>>>>>>>> Hello Jun, chia,
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
> >>>>>>> discussion
> >>>>>>>>>>>>>>>> feedback.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
> >>> behavior
> >>>>>>> from
> >>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> >>>>>>> auto.offset.reset
> >>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> >>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> >>>>>>>>>>>>>> auto.offset.reset.new.partitions
> >>>>>>>>>>>>>>>> config setting
> >>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> >>> auto.offset.reset
> >>>>>>>>>> policy
> >>>>>>>>>>>>>> continues
> >>>>>>>>>>>>>>>> to apply unchanged.
> >>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
> >>>> separate
> >>>>>>>>>>>>>> internal config
> >>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
> >>>>>> earliest).
> >>>>>>>>>> This
> >>>>>>>>>>>>>> decoupled design makes
> >>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
> >>>>>>> user-facing
> >>>>>>>>>>>>>> configuration in a future KIP.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
> >>> 寫道:
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
> >>>> below:
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default
> >>> value
> >>>>>> of
> >>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
> >>>>>> latest,
> >>>>>>> and
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
> >>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
> >>>>>>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest
> >>>> already
> >>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of other
> >>>>>>> values
> >>>>>>>>>>>> in a
> >>>>>>>>>>>>>> separate KIP.
> >>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> >>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all
> >>> for
> >>>>>>>>>>>>>> consistency.
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
> >>>>>> topic
> >>>>>>> is
> >>>>>>>>>>>>>> expanded
> >>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
> >>> offset
> >>>>>>> reset
> >>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> >>> partitions
> >>>>>>> before
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
> >>>>>> issue
> >>>>>>>>>> could
> >>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>> happen, right?
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will
> >>>> take
> >>>>>>> the
> >>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
> >>>>>>> by_duration if
> >>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> >>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
> >>>>>>> important
> >>>>>>>>>>>> from
> >>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
> >>> with
> >>>>>> a
> >>>>>>> few
> >>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic to
> >>>>>>> other
> >>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> >>>>>> by_duration
> >>>>>>>>>>>> don't
> >>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
> >>>>>>> partition
> >>>>>>>>>> is
> >>>>>>>>>>>>>> expanded.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
> >>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
> >>>>>> earliest
> >>>>>>> or
> >>>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For example,
> >>>>>> if a
> >>>>>>>>>>>>>> consumer is
> >>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created during
> >>>>>> that
> >>>>>>>>>>>>>> downtime,
> >>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
> >>>>>> resuming,
> >>>>>>>>>>>> rather
> >>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
> >>>>>>> technically
> >>>>>>>>>>>>>> "new" to
> >>>>>>>>>>>>>>>>>>> the group.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
> >>>>>>> threshold.
> >>>>>>>>>>>> It
> >>>>>>>>>>>>>> gives
> >>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
> >>>> "hot/new"
> >>>>>>> and
> >>>>>>>>>>>>>> what is
> >>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
> >>>>>> only
> >>>>>>>>>>>>>> applies when
> >>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> >>>>>>>>>> motivation
> >>>>>>>>>>>>>>>>>>> equally
> >>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
> >>> like
> >>>>>>>>>>>>>> by_duration.
> >>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
> >>>> control
> >>>>>>>>>> newly
> >>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group starts.
> >>>>>>> Have we
> >>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> >>>> .partitions?
> >>>>>>> If
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
> >>> the
> >>>>>>>>>> policy
> >>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly to
> >>>>>>>>>> customize
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> >>>>>>> PartitionMetadata
> >>>>>>>>>>>>>> struct
> >>>>>>>>>>>>>>>>>>>>> gains a new
> >>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> >>>>>>> server-side
> >>>>>>>>>>>> by
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> broker as
> >>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> >>>>>>> MembershipManager
> >>>>>>>>>>>>>> detects
> >>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>> newly assigned
> >>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
> >>>> the
> >>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
> >>>>>> topic
> >>>>>>> ID
> >>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>> in the cache.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> MB0:
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> >>>>>>>>>>>>>> MetadataResponse
> >>>>>>>>>>>>>>>>>>>>> version via the
> >>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> >>>>>>> negotiated
> >>>>>>>>>>>>>>>>>>> version is
> >>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
> >>> all
> >>>>>> and
> >>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> throw an
> >>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
> >>> latest
> >>>>>>> and
> >>>>>>>>>>>>>> risking
> >>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
> >>>> 下午4:04
> >>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> >>>>>> resolution
> >>>>>>>>>> from
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
> >>>>>> plane
> >>>>>>>>>>>> clean.
> >>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> >>>> inter-broker
> >>>>>>>>>> clock
> >>>>>>>>>>>>>> skew.
> >>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
> >>>>>> truth
> >>>>>>>>>> for
> >>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
> >>>>>>> negligible.
> >>>>>>>>>>>>>> Given
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
> >>>> minutes
> >>>>>>> or
> >>>>>>>>>>>>>> hours, a
> >>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
> >>> won't
> >>>>>>> impact
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> >>>>>> 2026年4月29日
> >>>>>>>>>>>> 下午3:29
> >>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> >>>>>>> conversation
> >>>>>>>>>>>> but I
> >>>>>>>>>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
> >>> the
> >>>>>>> KIP
> >>>>>>>>>>>>>> states
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
> >>> is
> >>>>>>> set.
> >>>>>>>>>>>>>>>>>>> Implicitly,
> >>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> >>> partition,
> >>>>>> it
> >>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>> issue a
> >>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages.
> >>>> Is
> >>>>>>> my
> >>>>>>>>>>>>>>>>>>>>> understanding
> >>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also
> >>>>>>> explain
> >>>>>>>>>>>>>> how it
> >>>>>>>>>>>>>>>>>>>>> fits
> >>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> >>>>>> offsets,
> >>>>>>>>>>>> etc.).
> >>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
> >>> for
> >>>>>>> this
> >>>>>>>>>>>> given
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> a new
> >>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
> >>>>>> simply
> >>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
> >>>> about
> >>>>>>> not
> >>>>>>>>>>>>>>>>>>> overloading
> >>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
> >>>> concepts.
> >>>>>>> The
> >>>>>>>>>>>> API
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> >>>> partitions.
> >>>>>>> The
> >>>>>>>>>>>> fact
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
> >>> API
> >>>>>>> also
> >>>>>>>>>>>>>> suggests
> >>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if we
> >>>>>>> want to
> >>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
> >>>> via
> >>>>>>> dev
> >>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
> >>> the
> >>>>>>> gap.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> >>>>>>> RequestPartitionAges
> >>>>>>>>>>>>>> field.
> >>>>>>>>>>>>>>>>>>> Can a
> >>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> >>>>>>>>>>>>>>>>>>> TopicPartitions)
> >>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
> >>>>>> value
> >>>>>>> or
> >>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> >>>>>>>>>>>> metadata.max.age.ms
> >>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> >>>>>> Streams,
> >>>>>>>>>>>> would
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> >>>>>> auto.offset.reset.latest.max.age
> >>>>>>> to
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
> >>> the
> >>>>>>> most
> >>>>>>>>>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
> >>>>>>> configure
> >>>>>>>>>>>>>> it. Or
> >>>>>>>>>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
> >>> back
> >>>>>> to
> >>>>>>>>>>>>>> earliest"
> >>>>>>>>>>>>>>>>>>>>> reads as
> >>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
> >>> which
> >>>>>>> isn't
> >>>>>>>>>>>>>>>>>>> supported.
> >>>>>>>>>>>>>>>>>>>>> May
> >>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
> >>>> the
> >>>>>>>>>>>> initial
> >>>>>>>>>>>>>>>>>>>>> position to
> >>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was
> >>>>>>> applied
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> >>>>>>> unchanged.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> >>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> >>> 2026年4月28日
> >>>>>>> 晚上8:03
> >>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
> >>>> time
> >>>>>>> via
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> Admin
> >>>>>>>>>>>>>>>>>>>>> API?
> >>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> >>> diagnose
> >>>>>> and
> >>>>>>>>>>>>>>>>>>> troubleshoot
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
> >>>>>>> Prevent
> >>>>>>>>>> Hot
> >>>>>>>>>>>>>> Data
> >>>>>>>>>>>>>>>>>>>>> Loss
> >>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>> 
> >>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
> >>>>>> (hot)
> >>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> >>>> automatically
> >>>>>>> fall
> >>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
> >>>>>>> historical
> >>>>>>>>>>>>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>> 
> >>>> 
> >>> 
> >> 
> 
> 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello all,

The metadata blindness and consumer downtime windows are inherently 
unpredictable, making a fixed by_duration approach unviable. No static 
duration can accommodate both extremes: 

- Short durations fail during extended downtimes, leading to the exact silent data 
loss this KIP aims to prevent. 
- Long durations (calibrated for worst-case scenarios) cause routine restarts to 
trigger unnecessary mass replays, undermining the latest offset reset policy. 

This is why the proposed solution uses creation time. It provides a discrete, 
deterministic signal—verifying whether a partition is actually newer than the group, 
rather than relying on an arbitrary time window that has no correlation with the consumer’s 
actual downtime.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月30日 下午3:05 寫道:
> 
> Hi all,
> 
> I'd like to revisit Jun previous comment about using by_duration=5s. I realize now that having a small backlog is actually an acceptable trade-off, especially since the latest policy inherently brings a tiny backlog to starting consumers anyway due to the time gap between fetching the offset and actually consuming.
> 
> If we bring the simple by_duration=5s solution back to the KIP, I'd like to discuss a further step: Should we just change the default policy from latest to by_duration=5s specifically for the new consumer and share consumer? 
> 
> Best,
> Chia-Ping
> 
> On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
>> hi Jun
>> 
>> The most important part of this story is how users should expect the data
>> they can see when using the latest or by_duration policy with expanded
>> partitions.
>> 
>> Yes, the by_duration policy can minimize data loss, but it is
>> non-deterministic, which means users will either read too many historical
>> records from existing partitions or lose some records from expanded
>> partitions.
>> 
>> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
>> understand, and that is why I preferred having a whole new policy based
>> entirely on group creation time (KIP-1282)
>> 
>> Best,
>> Chia-Ping
>> 
>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>> 
>>> Hi, Chia-Ping and Jiunn-Yang,
>>> 
>>> Thanks for the reply. I am still trying to understand the value of the new
>>> configs with the KIP.
>>> 
>>> The motivation of the KIP is that a user doesn't want to miss the data if
>>> the backlog is small. The backlog of the existing partition is easy to
>>> understand because it relates to retention time. The backlog for the new
>>> partition is a bit subtle to understand since it depends on the metadata
>>> refresh delay. To set auto.offset.reset.max.age.ms, the user needs to
>>> understand the metadata refresh delay on the consumer side and use it to
>>> set the config.
>>> 
>>> Now, let's consider the alternative: setting the same value for the
>>> existing by_duration policy. The KIP lists three issues with this approach.
>>> 1. It computes the seek target client-side as now() - duration, which
>>> introduces clock skew across consumers and forces operators to choose
>>> overly large durations, causing unnecessary reprocessing.
>>> 2. The target timestamp is recomputed on each retry, so failed
>>> ListOffsetsRequest retries can shift the target forward and potentially
>>> miss records produced between attempts.
>>> 3. It applies uniformly to all partitions without committed offsets, and
>>> cannot distinguish newly expanded partitions from long-existing partitions
>>> newly assigned to the group, leading to unnecessary replay.
>>> 
>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
>>> the metadata refresh delay. We could also consider improving the
>>> implementation. For issue 3, the metadata refresh delay is typically low
>>> (in the order of minutes with the classic consumer and tens of seconds with
>>> the new consumer). If a user is ok with reading that much backlog for new
>>> partitions, it seems they will be ok doing the same for existing
>>> partitions.
>>> 
>>> So, instead of introducing a new config, could we just reuse the existing
>>> config with better documentation and/or implementation?
>>> 
>>> Jun
>>> 
>>> 
>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hello Jun,
>>>> 
>>>> You're right that group creation time is the more intuitive answer at
>>>> first glance,
>>>> the KIP's own motivation talks about partitions that "predate the group"
>>>> vs partitions
>>>> "created during group runtime," which directly points to a
>>> group-lifecycle
>>>> classifier.
>>>> I'd like to walk through why we landed on partition age, and the
>>>> trade-offs we considered.
>>>> 
>>>> We evaluated three candidate signals:
>>>> 
>>>> 1. `by_duration:5secs`
>>>> 
>>>> This covers the metadata blindness window, but has issues the KIP
>>>> currently documents
>>>> under "Why not use `by_duration`?":
>>>> 
>>>> - Client-side `now() - duration` introduces clock skew across consumers.
>>>> - `ListOffsets` retries shift the target forward, potentially missing
>>>> records produced between
>>>> attempts.
>>>> - It applies uniformly to all partitions without committed offsets,
>>>> including pre-existing partitions
>>>> newly assigned to the group, causing unnecessary replay.
>>>> 
>>>> 2. Group creation time as classifier
>>>> 
>>>> This works cleanly when the consumer is actively running. Our concern
>>>> is the idle / late-rejoin case:
>>>> 
>>>> T=0:         Group created.
>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>> T=50:        Partition added during the idle window.
>>>> T=100:       Consumer resumes.
>>>> 
>>>> Under group creation time, the new partition is classified as new
>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
>>>> But during `[T=1, T=100]`, base partitions also accumulated data that
>>>> the consumer accepts as lost — that is precisely the contract of
>>>> `auto.offset.reset=latest`. There is no principled reason to treat
>>>> the new partition differently; both contain backlog accumulated during
>>>> the same idle window.
>>>> 
>>>> This aligns with the "backlog is backlog” principle you raised in
>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
>>>> every other partition during the same idle period; forcing 0-backlog
>>>> tolerance only on new partitions would be inconsistent with that
>>>> tolerance.
>>>> 
>>>> 3. Partition age vs threshold
>>>> 
>>>> Partition age corresponds to the actual silent data loss window,
>>>> the gap between partition creation and the consumer’s metadata
>>>> refresh. Within this window, data loss is genuinely silent: the
>>>> consumer had no opportunity to know about the partition. Outside this
>>>> window, missing data reflects either:
>>>> 
>>>> - (a) the user’s tolerated cost of running with idle consumers, or
>>>> - (b) an operational issue to surface via monitoring, not via reset
>>> policy.
>>>> 
>>>> We did not choose partition age because it is more elegant than group
>>>> creation time — we chose it because its failure mode (requires a
>>>> threshold) is
>>>> less invasive than the failure mode of group creation time (overrides
>>>> user-stated
>>>> `latest` intent during idle periods).
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>>>>> 
>>>>> Hi Jun,
>>>>> 
>>>>> Relying on both creation times will create an inconsistent scenario. A
>>>>> consumer that lost all offsets due to a long sleep will seek to the
>>>>> beginning for the partitions created later than the group.
>>>>> 
>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
>>>> using a
>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
>>>> goes
>>>>> back to using flexible configurations to prevent users from falling
>>> into
>>>>> that pitfall.
>>>>> 
>>>>> Best, Chia-Ping
>>>>> 
>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>> 
>>>>>> JR1. The design of auto.offset.reset.max.age.ms still feels weird to
>>>> me.
>>>>>> It
>>>>>> categorizes partitions as new or existing based on the partition
>>>> creation
>>>>>> time. Intuitively, the categorization should be based on the group
>>>> creation
>>>>>> time: all partitions existing when the group is created are existing
>>> and
>>>>>> all partitions created after the group creation are new partitions.
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> Manually bumping this thread. If there is no further
>>>>>>> discussion, I will close the vote.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>> 
>>>>>>>> Hello Jian,
>>>>>>>> 
>>>>>>>> Thanks for your feedback,
>>>>>>>> 
>>>>>>>> Agreed, partition expansion is a common operational task, not an
>>> edge
>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>> 
>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>> 
>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that this
>>>>>> is a
>>>>>>>>> common scenario rather than an edge case, which further
>>> demonstrates
>>>>>> the
>>>>>>>>> need for this optimization. For example:
>>>>>>>>> A partition expansion is a common operational task in Kafka: To
>>>>>> balance
>>>>>>>>> resource utilization and cost, topics are typically created with a
>>>>>>> moderate
>>>>>>>>> default partition count. However, as traffic grows over time, it is
>>>>>>> often
>>>>>>>>> necessary to increase the number of partitions to accommodate the
>>>>>> higher
>>>>>>>>> workload.
>>>>>>>>> 
>>>>>>>>> Regards
>>>>>>>>> Jian
>>>>>>>>> 
>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>> 
>>>>>>>>>> Hello chia,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
>>> describing
>>>>>>>>>>> partitions in the KIP? I understand you are using them to
>>> describe
>>>>>> the
>>>>>>>>>>> "freshness" or the users' need for the records, but applying
>>> these
>>>>>>> terms
>>>>>>>>>> to
>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>> 
>>>>>>>>>>> After all, in this scenario, users don't really care whether a
>>>>>>> partition
>>>>>>>>>> is
>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't
>>>>>>> silently
>>>>>>>>>>> lose any live records produced to the topic during their active
>>>>>>>>>> consumption.
>>>>>>>>>>> 
>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>> 
>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
>>>> section.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation for
>>>>>> the
>>>>>>>>>> KIP.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
>>> without
>>>>>> a
>>>>>>>>>>>>> committed offset are the same. A newly expanded partition (hot)
>>>> is
>>>>>>>>>>>>> fundamentally different from a partition the consumer has never
>>>>>> seen
>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot partition
>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
>>>> insufficient
>>>>>>>>>>>> because:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
>>> nodes
>>>>>>> due
>>>>>>>>>>>> to
>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
>>> duration,
>>>>>>>>>>>> causing
>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on
>>>>>> retry,
>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss."
>>>>>>>>>>>>> 
>>>>>>>>>>>>> However, both of these situations are rare. If these issues
>>>>>> persist,
>>>>>>>>>> more
>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
>>>>>> need a
>>>>>>>>>>>> common
>>>>>>>>>>>>> solution. If users care about those rare situations, they can
>>>>>>> implement
>>>>>>>>>>>>> customized logic using
>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If the creation time exists, the returned value should always
>>>> be
>>>>>>>>>>>> greater
>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>> field
>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The earliest point at which we can detect the version mismatch
>>>> is
>>>>>>>>>> during
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
>>>>>> poll().
>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
>>> poll().
>>>>>>> I’ll
>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>>> partition
>>>>>> in
>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>>>>>> broker_current_time
>>>>>>>>>>>> -
>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
>>>>>> support
>>>>>>>>>> this
>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If the creation time exists, the returned value should always
>>>> be
>>>>>>>>>>>> greater
>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>> field
>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
>>>>>>> discussion
>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
>>> behavior
>>>>>>> from
>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
>>>>>>> auto.offset.reset
>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>>>>>>>>> auto.offset.reset.new.partitions
>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>> auto.offset.reset
>>>>>>>>>> policy
>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
>>>> separate
>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
>>>>>> earliest).
>>>>>>>>>> This
>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
>>>>>>> user-facing
>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
>>> 寫道:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
>>>> below:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default
>>> value
>>>>>> of
>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
>>>>>> latest,
>>>>>>> and
>>>>>>>>>>>> it
>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
>>>>>>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest
>>>> already
>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of other
>>>>>>> values
>>>>>>>>>>>> in a
>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all
>>> for
>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
>>>>>> topic
>>>>>>> is
>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
>>> offset
>>>>>>> reset
>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
>>> partitions
>>>>>>> before
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
>>>>>> issue
>>>>>>>>>> could
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will
>>>> take
>>>>>>> the
>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
>>>>>>> by_duration if
>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
>>>>>>> important
>>>>>>>>>>>> from
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
>>> with
>>>>>> a
>>>>>>> few
>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic to
>>>>>>> other
>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
>>>>>> by_duration
>>>>>>>>>>>> don't
>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
>>>>>>> partition
>>>>>>>>>> is
>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
>>>>>> earliest
>>>>>>> or
>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For example,
>>>>>> if a
>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created during
>>>>>> that
>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
>>>>>> resuming,
>>>>>>>>>>>> rather
>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
>>>>>>> technically
>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
>>>>>>> threshold.
>>>>>>>>>>>> It
>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
>>>> "hot/new"
>>>>>>> and
>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
>>>>>> only
>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
>>> like
>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
>>>> control
>>>>>>>>>> newly
>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group starts.
>>>>>>> Have we
>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>>>> .partitions?
>>>>>>> If
>>>>>>>>>>>> this
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
>>> the
>>>>>>>>>> policy
>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly to
>>>>>>>>>> customize
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>>>>>> PartitionMetadata
>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
>>>>>>> server-side
>>>>>>>>>>>> by
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>> MembershipManager
>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
>>>> the
>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
>>>>>> topic
>>>>>>> ID
>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
>>> all
>>>>>> and
>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
>>> latest
>>>>>>> and
>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
>>>> 下午4:04
>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
>>>>>> resolution
>>>>>>>>>> from
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
>>>>>> plane
>>>>>>>>>>>> clean.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>>>> inter-broker
>>>>>>>>>> clock
>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
>>>>>> truth
>>>>>>>>>> for
>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
>>>>>>> negligible.
>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
>>>> minutes
>>>>>>> or
>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
>>> won't
>>>>>>> impact
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
>>>>>> 2026年4月29日
>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
>>>>>>> conversation
>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
>>> the
>>>>>>> KIP
>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
>>> is
>>>>>>> set.
>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>>> partition,
>>>>>> it
>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages.
>>>> Is
>>>>>>> my
>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also
>>>>>>> explain
>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
>>>>>> offsets,
>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
>>> for
>>>>>>> this
>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
>>>>>> simply
>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
>>>> about
>>>>>>> not
>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
>>>> concepts.
>>>>>>> The
>>>>>>>>>>>> API
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>>>> partitions.
>>>>>>> The
>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
>>> API
>>>>>>> also
>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if we
>>>>>>> want to
>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
>>>> via
>>>>>>> dev
>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
>>> the
>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
>>>>>> value
>>>>>>> or
>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>>>>>> metadata.max.age.ms
>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
>>>>>> Streams,
>>>>>>>>>>>> would
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>> auto.offset.reset.latest.max.age
>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
>>> the
>>>>>>> most
>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
>>>>>>> configure
>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
>>> back
>>>>>> to
>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
>>> which
>>>>>>> isn't
>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
>>>> the
>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was
>>>>>>> applied
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>> 2026年4月28日
>>>>>>> 晚上8:03
>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
>>>> time
>>>>>>> via
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
>>> diagnose
>>>>>> and
>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
>>>>>>> Prevent
>>>>>>>>>> Hot
>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
>>>>>> (hot)
>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>>>> automatically
>>>>>>> fall
>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
>>>>>>> historical
>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>> 



Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
Hi all,

I'd like to revisit Jun previous comment about using by_duration=5s. I realize now that having a small backlog is actually an acceptable trade-off, especially since the latest policy inherently brings a tiny backlog to starting consumers anyway due to the time gap between fetching the offset and actually consuming.

If we bring the simple by_duration=5s solution back to the KIP, I'd like to discuss a further step: Should we just change the default policy from latest to by_duration=5s specifically for the new consumer and share consumer? 

Best,
Chia-Ping

On 2026/06/15 17:39:19 Chia-Ping Tsai wrote:
> hi Jun
> 
> The most important part of this story is how users should expect the data
> they can see when using the latest or by_duration policy with expanded
> partitions.
> 
> Yes, the by_duration policy can minimize data loss, but it is
> non-deterministic, which means users will either read too many historical
> records from existing partitions or lose some records from expanded
> partitions.
> 
> Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
> understand, and that is why I preferred having a whole new policy based
> entirely on group creation time (KIP-1282)
> 
> Best,
> Chia-Ping
> 
> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> 
> > Hi, Chia-Ping and Jiunn-Yang,
> >
> > Thanks for the reply. I am still trying to understand the value of the new
> > configs with the KIP.
> >
> > The motivation of the KIP is that a user doesn't want to miss the data if
> > the backlog is small. The backlog of the existing partition is easy to
> > understand because it relates to retention time. The backlog for the new
> > partition is a bit subtle to understand since it depends on the metadata
> > refresh delay. To set auto.offset.reset.max.age.ms, the user needs to
> > understand the metadata refresh delay on the consumer side and use it to
> > set the config.
> >
> > Now, let's consider the alternative: setting the same value for the
> > existing by_duration policy. The KIP lists three issues with this approach.
> > 1. It computes the seek target client-side as now() - duration, which
> > introduces clock skew across consumers and forces operators to choose
> > overly large durations, causing unnecessary reprocessing.
> > 2. The target timestamp is recomputed on each retry, so failed
> > ListOffsetsRequest retries can shift the target forward and potentially
> > miss records produced between attempts.
> > 3. It applies uniformly to all partitions without committed offsets, and
> > cannot distinguish newly expanded partitions from long-existing partitions
> > newly assigned to the group, leading to unnecessary replay.
> >
> > Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
> > the metadata refresh delay. We could also consider improving the
> > implementation. For issue 3, the metadata refresh delay is typically low
> > (in the order of minutes with the classic consumer and tens of seconds with
> > the new consumer). If a user is ok with reading that much backlog for new
> > partitions, it seems they will be ok doing the same for existing
> > partitions.
> >
> > So, instead of introducing a new config, could we just reuse the existing
> > config with better documentation and/or implementation?
> >
> > Jun
> >
> >
> > On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> > > Hello Jun,
> > >
> > > You're right that group creation time is the more intuitive answer at
> > > first glance,
> > > the KIP's own motivation talks about partitions that "predate the group"
> > > vs partitions
> > > "created during group runtime," which directly points to a
> > group-lifecycle
> > > classifier.
> > > I'd like to walk through why we landed on partition age, and the
> > > trade-offs we considered.
> > >
> > > We evaluated three candidate signals:
> > >
> > > 1. `by_duration:5secs`
> > >
> > > This covers the metadata blindness window, but has issues the KIP
> > > currently documents
> > > under "Why not use `by_duration`?":
> > >
> > > - Client-side `now() - duration` introduces clock skew across consumers.
> > > - `ListOffsets` retries shift the target forward, potentially missing
> > > records produced between
> > > attempts.
> > > - It applies uniformly to all partitions without committed offsets,
> > > including pre-existing partitions
> > > newly assigned to the group, causing unnecessary replay.
> > >
> > > 2. Group creation time as classifier
> > >
> > > This works cleanly when the consumer is actively running. Our concern
> > > is the idle / late-rejoin case:
> > >
> > > T=0:         Group created.
> > > T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > T=50:        Partition added during the idle window.
> > > T=100:       Consumer resumes.
> > >
> > > Under group creation time, the new partition is classified as new
> > > (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
> > > But during `[T=1, T=100]`, base partitions also accumulated data that
> > > the consumer accepts as lost — that is precisely the contract of
> > > `auto.offset.reset=latest`. There is no principled reason to treat
> > > the new partition differently; both contain backlog accumulated during
> > > the same idle window.
> > >
> > > This aligns with the "backlog is backlog” principle you raised in
> > > the KIP-1282 thread: a `latest` user has tolerated some backlog on
> > > every other partition during the same idle period; forcing 0-backlog
> > > tolerance only on new partitions would be inconsistent with that
> > > tolerance.
> > >
> > > 3. Partition age vs threshold
> > >
> > > Partition age corresponds to the actual silent data loss window,
> > > the gap between partition creation and the consumer’s metadata
> > > refresh. Within this window, data loss is genuinely silent: the
> > > consumer had no opportunity to know about the partition. Outside this
> > > window, missing data reflects either:
> > >
> > > - (a) the user’s tolerated cost of running with idle consumers, or
> > > - (b) an operational issue to surface via monitoring, not via reset
> > policy.
> > >
> > > We did not choose partition age because it is more elegant than group
> > > creation time — we chose it because its failure mode (requires a
> > > threshold) is
> > > less invasive than the failure mode of group creation time (overrides
> > > user-stated
> > > `latest` intent during idle periods).
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > > > Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > > >
> > > > Hi Jun,
> > > >
> > > > Relying on both creation times will create an inconsistent scenario. A
> > > > consumer that lost all offsets due to a long sleep will seek to the
> > > > beginning for the partitions created later than the group.
> > > >
> > > > That is why we initially proposed KIP-1282 to fix the inconsistency
> > > using a
> > > > whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
> > > goes
> > > > back to using flexible configurations to prevent users from falling
> > into
> > > > that pitfall.
> > > >
> > > > Best, Chia-Ping
> > > >
> > > > Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> > > >
> > > >> Hi, Jiunn-Yang,
> > > >>
> > > >> Thanks for the reply and sorry for the late reply.
> > > >>
> > > >> JR1. The design of auto.offset.reset.max.age.ms still feels weird to
> > > me.
> > > >> It
> > > >> categorizes partitions as new or existing based on the partition
> > > creation
> > > >> time. Intuitively, the categorization should be based on the group
> > > creation
> > > >> time: all partitions existing when the group is created are existing
> > and
> > > >> all partitions created after the group creation are new partitions.
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> Manually bumping this thread. If there is no further
> > > >>> discussion, I will close the vote.
> > > >>>
> > > >>> Best Regards,
> > > >>> Jiunn-Yang
> > > >>>
> > > >>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > >>>>
> > > >>>> Hello Jian,
> > > >>>>
> > > >>>> Thanks for your feedback,
> > > >>>>
> > > >>>> Agreed, partition expansion is a common operational task, not an
> > edge
> > > >>>> case. I've updated the Motivation section accordingly.
> > > >>>>
> > > >>>> Best Regards,
> > > >>>> Jiunn-Yang
> > > >>>>
> > > >>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > >>>>>
> > > >>>>> Hi Jiunn-Yang:
> > > >>>>>
> > > >>>>> Thanks for the KIP. I think it would be useful to clarify that this
> > > >> is a
> > > >>>>> common scenario rather than an edge case, which further
> > demonstrates
> > > >> the
> > > >>>>> need for this optimization. For example:
> > > >>>>> A partition expansion is a common operational task in Kafka: To
> > > >> balance
> > > >>>>> resource utilization and cost, topics are typically created with a
> > > >>> moderate
> > > >>>>> default partition count. However, as traffic grows over time, it is
> > > >>> often
> > > >>>>> necessary to increase the number of partitions to accommodate the
> > > >> higher
> > > >>>>> workload.
> > > >>>>>
> > > >>>>> Regards
> > > >>>>> Jian
> > > >>>>>
> > > >>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > >>>>>
> > > >>>>>> Hello chia,
> > > >>>>>>
> > > >>>>>> Thanks for the comments, I have updated the KIP!
> > > >>>>>>
> > > >>>>>> Best Regards,
> > > >>>>>> Jiunn-Yang
> > > >>>>>>
> > > >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> > > >>>>>>>
> > > >>>>>>> Hi Jiunn-Yang,
> > > >>>>>>>
> > > >>>>>>> Would you mind removing the terms "hot" and "cold" when
> > describing
> > > >>>>>>> partitions in the KIP? I understand you are using them to
> > describe
> > > >> the
> > > >>>>>>> "freshness" or the users' need for the records, but applying
> > these
> > > >>> terms
> > > >>>>>> to
> > > >>>>>>> the partition itself feels a bit unnatural.
> > > >>>>>>>
> > > >>>>>>> After all, in this scenario, users don't really care whether a
> > > >>> partition
> > > >>>>>> is
> > > >>>>>>> newly expanded or not. Their only expectation is that they won't
> > > >>> silently
> > > >>>>>>> lose any live records produced to the topic during their active
> > > >>>>>> consumption.
> > > >>>>>>>
> > > >>>>>>> Best, Chia-Ping
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > >>>>>>>
> > > >>>>>>>> Hello Jun,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the feedback, I have updated the KIP motivation
> > > section.
> > > >>>>>>>>
> > > >>>>>>>> Best Regards,
> > > >>>>>>>> Jiunn-Yang
> > > >>>>>>>>
> > > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> > > >>>>>>>>>
> > > >>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks for the reply. I think we need a stronger motivation for
> > > >> the
> > > >>>>>> KIP.
> > > >>>>>>>>>
> > > >>>>>>>>> The KIP says "The core insight is that not all partitions
> > without
> > > >> a
> > > >>>>>>>>> committed offset are the same. A newly expanded partition (hot)
> > > is
> > > >>>>>>>>> fundamentally different from a partition the consumer has never
> > > >> seen
> > > >>>>>>>>> because it predates the group (cold)." Why is the hot partition
> > > >>>>>>>>> fundamentally different from the cold?
> > > >>>>>>>>>
> > > >>>>>>>>> The KIP says "The existing by_duration policy is also
> > > insufficient
> > > >>>>>>>> because:
> > > >>>>>>>>>
> > > >>>>>>>>> - The calculated seek time (now() - duration) varies across
> > nodes
> > > >>> due
> > > >>>>>>>> to
> > > >>>>>>>>> clock skew. To be safe, users must set an overly large
> > duration,
> > > >>>>>>>> causing
> > > >>>>>>>>> unnecessary reprocessing.
> > > >>>>>>>>> - On network errors, the client recalculates the seek time on
> > > >> retry,
> > > >>>>>>>>> shifting the target timestamp forward and risking data loss."
> > > >>>>>>>>>
> > > >>>>>>>>> However, both of these situations are rare. If these issues
> > > >> persist,
> > > >>>>>> more
> > > >>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
> > > >> need a
> > > >>>>>>>> common
> > > >>>>>>>>> solution. If users care about those rare situations, they can
> > > >>> implement
> > > >>>>>>>>> customized logic using
> > > >>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > >>>>>>>>>
> > > >>>>>>>>> Jun
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hello chia,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for the feedback,
> > > >>>>>>>>>>
> > > >>>>>>>>>>> If the creation time exists, the returned value should always
> > > be
> > > >>>>>>>> greater
> > > >>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > >>>>>>>>>>
> > > >>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> > > >>> field
> > > >>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>
> > > >>>>>>>>>> The earliest point at which we can detect the version mismatch
> > > is
> > > >>>>>> during
> > > >>>>>>>>>> the
> > > >>>>>>>>>> first metadata fetch after assignment, which occurs inside
> > > >> poll().
> > > >>>>>>>>>> Therefore, the
> > > >>>>>>>>>> user would encounter an UnsupportedVersionException from
> > poll().
> > > >>> I’ll
> > > >>>>>>>>>> clarify this in the KIP.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best Regards,
> > > >>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> hi Jiunn
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> > partition
> > > >> in
> > > >>>>>>>>>> milliseconds, computed server-side by the broker as
> > > >>>>>> broker_current_time
> > > >>>>>>>> -
> > > >>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
> > > >> support
> > > >>>>>> this
> > > >>>>>>>>>> feature or the partition creation time is unknown.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> If the creation time exists, the returned value should always
> > > be
> > > >>>>>>>> greater
> > > >>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> > > >>> field
> > > >>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
> > > >>>>>> `poll()`?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best,
> > > >>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > >>>>>>>>>>>> Hello Jun, chia,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
> > > >>> discussion
> > > >>>>>>>>>>>> feedback.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> The updated design decouples the new-partition reset
> > behavior
> > > >>> from
> > > >>>>>>>>>>>> the base auto.offset.reset policy:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> > > >>> auto.offset.reset
> > > >>>>>>>>>> values
> > > >>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > >>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> > > >>>>>>>>>> auto.offset.reset.new.partitions
> > > >>>>>>>>>>>> config setting
> > > >>>>>>>>>>>> - For existing ("cold") partitions, the base
> > auto.offset.reset
> > > >>>>>> policy
> > > >>>>>>>>>> continues
> > > >>>>>>>>>>>> to apply unchanged.
> > > >>>>>>>>>>>> - The new-partition reset behavior is represented by a
> > > separate
> > > >>>>>>>>>> internal config
> > > >>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
> > > >> earliest).
> > > >>>>>> This
> > > >>>>>>>>>> decoupled design makes
> > > >>>>>>>>>>>> it straightforward to promote the behavior to a public
> > > >>> user-facing
> > > >>>>>>>>>> configuration in a future KIP.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
> > 寫道:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
> > > below:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default
> > value
> > > >> of
> > > >>>>>>>>>> earliest. It fixes the data loss from both by_duration and
> > > >> latest,
> > > >>> and
> > > >>>>>>>> it
> > > >>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
> > > >>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
> > > >>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest
> > > already
> > > >>>>>>>>>> addresses the issue, and we can discuss the use cases of other
> > > >>> values
> > > >>>>>>>> in a
> > > >>>>>>>>>> separate KIP.
> > > >>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> > > >>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all
> > for
> > > >>>>>>>>>> consistency.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>> Hi, Chia-Ping,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
> > > >> topic
> > > >>> is
> > > >>>>>>>>>> expanded
> > > >>>>>>>>>>>>>> with new partitions, consumers using the latest auto
> > offset
> > > >>> reset
> > > >>>>>>>>>> policy
> > > >>>>>>>>>>>>>> will silently miss all records produced to those
> > partitions
> > > >>> before
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > >>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
> > > >> issue
> > > >>>>>> could
> > > >>>>>>>>>> also
> > > >>>>>>>>>>>>>> happen, right?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will
> > > take
> > > >>> the
> > > >>>>>>>> same
> > > >>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
> > > >>> by_duration if
> > > >>>>>>>>>> needed.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > > >>>>>> chia7712@apache.org
> > > >>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
> > > >>> important
> > > >>>>>>>> from
> > > >>>>>>>>>> your
> > > >>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
> > with
> > > >> a
> > > >>> few
> > > >>>>>>>>>> questions:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic to
> > > >>> other
> > > >>>>>>>> reset
> > > >>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> > > >> by_duration
> > > >>>>>>>> don't
> > > >>>>>>>>>> seem
> > > >>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
> > > >>> partition
> > > >>>>>> is
> > > >>>>>>>>>> expanded.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 2. What values would we expect users to configure for
> > > >>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
> > > >> earliest
> > > >>> or
> > > >>>>>>>>>> latest,
> > > >>>>>>>>>>>>>>> we might run into the exact same edge cases. For example,
> > > >> if a
> > > >>>>>>>>>> consumer is
> > > >>>>>>>>>>>>>>> offline for a while and a new partition is created during
> > > >> that
> > > >>>>>>>>>> downtime,
> > > >>>>>>>>>>>>>>> the user might actually want to skip to latest when
> > > >> resuming,
> > > >>>>>>>> rather
> > > >>>>>>>>>> than
> > > >>>>>>>>>>>>>>> reading from earliest just because the partition is
> > > >>> technically
> > > >>>>>>>>>> "new" to
> > > >>>>>>>>>>>>>>> the group.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
> > > >>> threshold.
> > > >>>>>>>> It
> > > >>>>>>>>>> gives
> > > >>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
> > > "hot/new"
> > > >>> and
> > > >>>>>>>>>> what is
> > > >>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks for the KIP.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
> > > >> only
> > > >>>>>>>>>> applies when
> > > >>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> > > >>>>>> motivation
> > > >>>>>>>>>>>>>>> equally
> > > >>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
> > like
> > > >>>>>>>>>> by_duration.
> > > >>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>> intention is that we want to have a separate way to
> > > control
> > > >>>>>> newly
> > > >>>>>>>>>> created
> > > >>>>>>>>>>>>>>>> partitions vs existing partitions when the group starts.
> > > >>> Have we
> > > >>>>>>>>>>>>>>> considered
> > > >>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > > .partitions?
> > > >>> If
> > > >>>>>>>> this
> > > >>>>>>>>>> new
> > > >>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
> > the
> > > >>>>>> policy
> > > >>>>>>>>>> used
> > > >>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly to
> > > >>>>>> customize
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> behavior for new partitions.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com>
> > > >>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hello all,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > > >>> PartitionMetadata
> > > >>>>>>>>>> struct
> > > >>>>>>>>>>>>>>>>> gains a new
> > > >>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> > > >>> server-side
> > > >>>>>>>> by
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> broker as
> > > >>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > >>> MembershipManager
> > > >>>>>>>>>> detects
> > > >>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>> newly assigned
> > > >>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
> > > the
> > > >>>>>>>> affected
> > > >>>>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > >>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
> > > >> topic
> > > >>> ID
> > > >>>>>> is
> > > >>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>> in the cache.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> MB0:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> > > >>>>>>>>>> MetadataResponse
> > > >>>>>>>>>>>>>>>>> version via the
> > > >>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> > > >>> negotiated
> > > >>>>>>>>>>>>>>> version is
> > > >>>>>>>>>>>>>>>>> unsupported, the consumer
> > > >>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
> > all
> > > >> and
> > > >>>>>> can
> > > >>>>>>>>>>>>>>> throw an
> > > >>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
> > latest
> > > >>> and
> > > >>>>>>>>>> risking
> > > >>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
> > > 下午4:04
> > > >>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> hi David
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> > > >> resolution
> > > >>>>>> from
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
> > > >> plane
> > > >>>>>>>> clean.
> > > >>>>>>>>>> The
> > > >>>>>>>>>>>>>>> main
> > > >>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > > inter-broker
> > > >>>>>> clock
> > > >>>>>>>>>> skew.
> > > >>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
> > > >> truth
> > > >>>>>> for
> > > >>>>>>>>>> time.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
> > > >>> negligible.
> > > >>>>>>>>>> Given
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
> > > minutes
> > > >>> or
> > > >>>>>>>>>> hours, a
> > > >>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
> > won't
> > > >>> impact
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>>>>> fallback decision.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> > > >> 2026年4月29日
> > > >>>>>>>> 下午3:29
> > > >>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> > > >>> conversation
> > > >>>>>>>> but I
> > > >>>>>>>>>>>>>>> took a
> > > >>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> > > >>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > >>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
> > the
> > > >>> KIP
> > > >>>>>>>>>> states
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
> > is
> > > >>> set.
> > > >>>>>>>>>>>>>>> Implicitly,
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> > partition,
> > > >> it
> > > >>>>>> will
> > > >>>>>>>>>>>>>>> issue a
> > > >>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages.
> > > Is
> > > >>> my
> > > >>>>>>>>>>>>>>>>> understanding
> > > >>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also
> > > >>> explain
> > > >>>>>>>>>> how it
> > > >>>>>>>>>>>>>>>>> fits
> > > >>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> > > >> offsets,
> > > >>>>>>>> etc.).
> > > >>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> > > >>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
> > for
> > > >>> this
> > > >>>>>>>> given
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> a new
> > > >>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
> > > >> simply
> > > >>>>>>>>>> include
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
> > > about
> > > >>> not
> > > >>>>>>>>>>>>>>> overloading
> > > >>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
> > > concepts.
> > > >>> The
> > > >>>>>>>> API
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > > partitions.
> > > >>> The
> > > >>>>>>>> fact
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
> > API
> > > >>> also
> > > >>>>>>>>>> suggests
> > > >>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if we
> > > >>> want to
> > > >>>>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>> David
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
> > > via
> > > >>> dev
> > > >>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
> > the
> > > >>> gap.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > >>> RequestPartitionAges
> > > >>>>>>>>>> field.
> > > >>>>>>>>>>>>>>> Can a
> > > >>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> > > >>>>>>>>>>>>>>> TopicPartitions)
> > > >>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
> > > >> value
> > > >>> or
> > > >>>>>> a
> > > >>>>>>>>>>>>>>> ConfigDef
> > > >>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > > >>>>>>>> metadata.max.age.ms
> > > >>>>>>>>>> ?
> > > >>>>>>>>>>>>>>>>> Sizing
> > > >>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> > > >> Streams,
> > > >>>>>>>> would
> > > >>>>>>>>>> it
> > > >>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>> better
> > > >>>>>>>>>>>>>>>>>>>>> to add this new config
> > > >> auto.offset.reset.latest.max.age
> > > >>> to
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > >>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > >>>>>>>>>>>>>>>>> for a
> > > >>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
> > the
> > > >>> most
> > > >>>>>>>>>>>>>>> familiar
> > > >>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
> > > >>> configure
> > > >>>>>>>>>> it. Or
> > > >>>>>>>>>>>>>>>>> may be
> > > >>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
> > back
> > > >> to
> > > >>>>>>>>>> earliest"
> > > >>>>>>>>>>>>>>>>> reads as
> > > >>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
> > which
> > > >>> isn't
> > > >>>>>>>>>>>>>>> supported.
> > > >>>>>>>>>>>>>>>>> May
> > > >>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
> > > the
> > > >>>>>>>> initial
> > > >>>>>>>>>>>>>>>>> position to
> > > >>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was
> > > >>> applied
> > > >>>>>>>> to
> > > >>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> > > >>> unchanged.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>> Murali
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > >>> s7133700@gmail.com>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2026年4月28日
> > > >>> 晚上8:03
> > > >>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
> > > time
> > > >>> via
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>>> Admin
> > > >>>>>>>>>>>>>>>>> API?
> > > >>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> > diagnose
> > > >> and
> > > >>>>>>>>>>>>>>> troubleshoot
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
> > > >>> Prevent
> > > >>>>>> Hot
> > > >>>>>>>>>> Data
> > > >>>>>>>>>>>>>>>>> Loss
> > > >>>>>>>>>>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > >>>>>>>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>
> > > >>
> > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > >>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > >>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > >>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
> > > >> (hot)
> > > >>>>>>>>>> partitions
> > > >>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > > automatically
> > > >>> fall
> > > >>>>>>>>>> back
> > > >>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > >>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
> > > >>> historical
> > > >>>>>>>>>>>>>>> reprocess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Andrew Schofield <sc...@apache.org>.
Hi Chia-Ping,
For (1), we do that already, by starting at offset zero. For (2), we will follow the existing auto.offset.reset policy.

But, I know that you folks have more closely researched this area and you've dug into some nasty corner cases for consumer groups. Maybe there are some of those for all group types.

I'm happy for share groups to be added to this KIP using a similar technique as for consumer groups. I think it's less critical for share groups, but I am generally in favour of consistency.

Thanks,
Andrew

On 2026/06/29 14:57:43 Chia-Ping Tsai wrote:
> Hi Andrew,
> 
> Yes, we can figure out the expanded partitions using ShareGroupStatePartitionMetadataInfo, but the true question is - should a share group need a different policy for expanded partitions?
> 
> The challenge encountered by normal consumers with the latest policy is that sometimes users want to reset to 0, and sometimes it is required to set to latest.
> 
> 1) The former happens when an expanded partition is dynamically added to an active group. Users often expect to consume from the very beginning of this new partition to avoid missing data, even if the overall group policy is latest.
> 2) The latter happens when a group's state is completely wiped (e.g., when offsets expire for normal consumers, or when a share group is manually deleted and recreated). In this scenario, they want a consistent latest offset behavior across both old and newly expanded partitions.
> 
> Given these distinct user expectations, does it make sense to consider a more granular offset reset strategy specifically for partition expansion in share groups?
> 
> Best,
> Chia-Ping
> 
> On 2026/06/29 07:20:40 Andrew Schofield wrote:
> > Hi Jun,
> > This is handled because share groups do not support regex subscriptions :) I think this KIP is a nice improvement, but I also think that we don't need it for share groups with their current behaviour. I am perfectly amenable to having it extended to cover share groups too, but I don't think it will make a practical difference.
> > 
> > Thanks,
> > Andrew
> > 
> > On 2026/06/26 18:57:54 Jun Rao via dev wrote:
> > > Hi, Andrew,
> > > 
> > > There is another case where a share consumer subscribes to a regex and a
> > > new topic is later created that matches that regex. Intuitively, they
> > > should use the earliest offset for consumption. But how does the share
> > > coordinator distinguish these from existing topics explicitly added by the
> > > user without the creation timestamp?
> > > 
> > > Thanks,
> > > 
> > > Jun
> > > 
> > > On Fri, Jun 26, 2026 at 11:00 AM Andrew Schofield <sc...@apache.org>
> > > wrote:
> > > 
> > > > Hi Jun,
> > > > When a user adds an existing topic to a share group subscription, we do
> > > > use the auto.offset.reset policy because those partitions already exist and
> > > > may well have data on them. We only use 0 in the case where a topic already
> > > > being consumed gained some additional new partitions. These partitions will
> > > > be empty at the time of creation, so zero seems like a reasonable starting
> > > > point. Partitions being consumed in a share group always have a known
> > > > starting position once assigned. We don't have the equivalent of a consumer
> > > > group not having a committed offset.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On 2026/06/26 17:36:31 Jun Rao via dev wrote:
> > > > > Hi, Andrew,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > I am not sure it's always a good idea to start new partitions with
> > > > offset 0
> > > > > for the share group. For example, once a share group starts, a user could
> > > > > add a subscription to an existing topic. That existing topic could have a
> > > > > large backlog and starting it with offset 0 forces the user to consume
> > > > that
> > > > > entire backlog. In KIP-1327, such a partition will be treated as an
> > > > > existing partition and will follow the auto.offset.reset policy. This
> > > > seems
> > > > > more intuitive. It would be useful to consider if the share group should
> > > > > adopt the same policy.
> > > > >
> > > > > Jun
> > > > >
> > > > > On Fri, Jun 26, 2026 at 3:42 AM Andrew Schofield <sc...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Jiunn-Yang et al.,
> > > > > > Some background on the problem I hinted at for the KIP-932
> > > > implementation.
> > > > > > KIP-932 states "If the number of partitions is increased for a topic
> > > > with a
> > > > > > subscription in a share group, the SPSO for the newly created
> > > > > > share-partitions is initialized to 0 (which is of course both the
> > > > earliest
> > > > > > and latest offset for an empty topic-partition). This means there is no
> > > > > > doubt about what happens when the number of partitions is increased."
> > > > > >
> > > > > > The problem was that we had not implemented the use of 0, but instead
> > > > were
> > > > > > using the share.auto.offset.reset config to choose the initial offset
> > > > in
> > > > > > this case. We write a ShareGroupStatePartitionMetadata record when a
> > > > topic
> > > > > > is first being initialized for consumption by a share group, and this
> > > > > > contains a list of the partitions which we are initializing. As a
> > > > result, I
> > > > > > believe we can already tell when partitions are added, and we do know
> > > > when
> > > > > > to initialize to 0 as it says in the KIP. We will put in a patch that
> > > > uses
> > > > > > 0 as intended.
> > > > > >
> > > > > > I do not mind whether we want to incorporate share groups into
> > > > KIP-1327,
> > > > > > but I do not think that it is necessary to do so.
> > > > > >
> > > > > > Thanks,
> > > > > > Andrew
> > > > > >
> > > > > > On 2026/06/26 04:31:28 Chia-Ping Tsai wrote:
> > > > > > > > JR5. Since this changes the on-disk data format for
> > > > PartitionRecord, we
> > > > > > > > need to gate this by a new MV. It also changes the format of
> > > > > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not
> > > > sure if
> > > > > > this
> > > > > > > > requires a new MV or a new GV.
> > > > > > >
> > > > > > > That is a good point. If we want to maintain consistency for the
> > > > > > NewPartitions field when the group coordinator migrates between new
> > > > and old
> > > > > > binaries, a version gate is definitely required. For example:
> > > > > > >
> > > > > > > GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())
> > > > > > >
> > > > > > > The group coordinator should NOT set the NewPartitions field when
> > > > the gv
> > > > > > is smaller than gv_2
> > > > > > >
> > > > > > > Best,
> > > > > > > Chia-Ping
> > > > > > >
> > > > > > > On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> > > > > > > > Hi, Jiunn-Yang,
> > > > > > > >
> > > > > > > > JR2. We should try to maintain consistency across our APIs. So, it
> > > > > > would be
> > > > > > > > useful to understand if the user-facing part of this KIP is what
> > > > share
> > > > > > > > consumer wants too. It would be problematic if we proceed with this
> > > > > > > > approach only for the share consumer to choose a different one
> > > > later.
> > > > > > > >
> > > > > > > > JR3. I was thinking that if we add the NewPartitions field in
> > > > > > > > GroupHeartbeatResponse, there is no need to expose the
> > > > group/partition
> > > > > > > > creation timestamp to other RPCs and the Cli.
> > > > > > > >
> > > > > > > > JR5. Since this changes the on-disk data format for
> > > > PartitionRecord, we
> > > > > > > > need to gate this by a new MV. It also changes the format of
> > > > > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not
> > > > sure if
> > > > > > this
> > > > > > > > requires a new MV or a new GV.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > > On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hello Jun, chia,
> > > > > > > > >
> > > > > > > > > Thanks for your feedback.
> > > > > > > > >
> > > > > > > > > JR3 JR4 I have updated the KIP.
> > > > > > > > >
> > > > > > > > > JR2 The new-partition detection mechanism is reusable. The
> > > > > > coordinator
> > > > > > > > > already has
> > > > > > > > > access to both timestamps required for classification, so the
> > > > > > > > > determination can be made
> > > > > > > > > server-side without introducing any new RPCs. The main
> > > > difference is
> > > > > > where
> > > > > > > > > the policy is
> > > > > > > > > enforced: modern consumer resolves it on the client, whereas a
> > > > share
> > > > > > group
> > > > > > > > > resolves the
> > > > > > > > > starting offset on the partition leader. As a result, the
> > > > > > classification
> > > > > > > > > decision would need to
> > > > > > > > > be propagated through the existing share-state initialization
> > > > path.
> > > > > > We
> > > > > > > > > would also need to persist
> > > > > > > > > the group's creation time as part of the share-group state, since
> > > > > > that
> > > > > > > > > information is not stored today.
> > > > > > > > >
> > > > > > > > > None of this is a blocker, but it does require changes to the
> > > > > > share-state
> > > > > > > > > format and introduces
> > > > > > > > > a share-specific configuration. Given that additional scope, I
> > > > would
> > > > > > > > > prefer to keep share groups out
> > > > > > > > > of scope for this KIP and address them in a follow-up effort,
> > > > while
> > > > > > > > > documenting the relevant reuse
> > > > > > > > > opportunities here.
> > > > > > > > >
> > > > > > > > > Best Regards,
> > > > > > > > > Jiunn-Yang
> > > > > > > > >
> > > > > > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > > > > > > > > >
> > > > > > > > > >> It inspires me to think about mirror clusters. What if the
> > > > mirror
> > > > > > group
> > > > > > > > > is created 'after' the expanded partitions? In this scenario,
> > > > the new
> > > > > > > > > configuration 'max.age.ms' could handle it if users are aware
> > > > of the
> > > > > > > > > migration and set a suitable time interval. Of course, this would
> > > > > > require
> > > > > > > > > both creation times to be exposed to clients.
> > > > > > > > > >
> > > > > > > > > > Thinking more about the mirror cluster scenario we discussed, I
> > > > > > realize
> > > > > > > > > it introduces significant complexity. Handling it properly would
> > > > > > require a
> > > > > > > > > tolerance time window rather than a strict time interval to
> > > > account
> > > > > > for
> > > > > > > > > replication lag. To prevent scope creep and keep KIP-1327
> > > > focused on
> > > > > > its
> > > > > > > > > primary goal, I think it would be best to defer the
> > > > > > multi-cluster/mirroring
> > > > > > > > > scenario and explore those approaches in a separate KIP
> > > > > > > > > >
> > > > > > > > > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > > > > > > > > >> hi all,
> > > > > > > > > >>
> > > > > > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > > > > > client.
> > > > > > > > > I am
> > > > > > > > > >>> not sure if there are other use cases for those timestamps.
> > > > An
> > > > > > > > > alternative
> > > > > > > > > >>> is to avoid exposing those timestamps to the client. The
> > > > group
> > > > > > > > > coordinator
> > > > > > > > > >>> will categorize the partitions using those timestamps and
> > > > > > include the
> > > > > > > > > >>> categorization in the assigned partitions in
> > > > > > GroupHeartbeatResponse.
> > > > > > > > > >>
> > > > > > > > > >> It inspires me to think about mirror clusters. What if the
> > > > mirror
> > > > > > group
> > > > > > > > > is created 'after' the expanded partitions? In this scenario,
> > > > the new
> > > > > > > > > configuration 'max.age.ms' could handle it if users are aware
> > > > of the
> > > > > > > > > migration and set a suitable time interval. Of course, this would
> > > > > > require
> > > > > > > > > both creation times to be exposed to clients.
> > > > > > > > > >>
> > > > > > > > > >> Best,
> > > > > > > > > >> Chia-Ping
> > > > > > > > > >>
> > > > > > > > > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > > > > > > > > >>> Hi, Jiunn-Yang,
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks for the updated KIP.
> > > > > > > > > >>>
> > > > > > > > > >>> JR2. Share consumer (KIP-932) states: "If the number of
> > > > > > partitions is
> > > > > > > > > >>> increased for a topic with a subscription in a share group,
> > > > the
> > > > > > SPSO
> > > > > > > > > for
> > > > > > > > > >>> the newly created share-partitions is initialized to 0 ".
> > > > > > However, this
> > > > > > > > > >>> part has not been implemented yet. Currently, all partitions
> > > > > > without
> > > > > > > > > offset
> > > > > > > > > >>> follows the strategy given by share.auto.offset.reset. So, we
> > > > > > have an
> > > > > > > > > >>> opportunity to see if it's better to align this KIP's changes
> > > > > > with the
> > > > > > > > > >>> share consumer.
> > > > > > > > > >>> JR2.1 The first part is whether customizing the offset for
> > > > new
> > > > > > > > > partitions
> > > > > > > > > >>> to something other than 0 is useful in the share consumer. It
> > > > > > seems
> > > > > > > > > that
> > > > > > > > > >>> the motivation in this KIP applies to share consumer too.
> > > > > > > > > >>> JR2.2 The second part is how new partitions are detected. The
> > > > > > mechanism
> > > > > > > > > >>> introduced in this KIP seems quite effective. A group
> > > > > > coordinator could
> > > > > > > > > >>> pick up newly created partitions for an existing topic or
> > > > newly
> > > > > > created
> > > > > > > > > >>> topics matching the regex. Those partitions are categorized
> > > > as
> > > > > > new
> > > > > > > > > >>> partitions and will follow the strategy given by
> > > > > > > > > >>> auto.offset.reset.new.partitions.
> > > > > > > > > >>> This makes sense since those partitions typically don't have
> > > > a
> > > > > > large
> > > > > > > > > >>> backlog. A consumer could also additionally subscribe to an
> > > > > > existing
> > > > > > > > > topic
> > > > > > > > > >>> in an existing group. Those partitions will be categorized as
> > > > > > existing
> > > > > > > > > >>> partitions. This also makes sense since those partitions
> > > > could
> > > > > > have a
> > > > > > > > > large
> > > > > > > > > >>> backlog and it is better to follow the strategy given by
> > > > > > > > > auto.offset.reset. It
> > > > > > > > > >>> will be useful to think through whether the logic for
> > > > > > categorizing new
> > > > > > > > > >>> partitions should also be reused in the share consumer.
> > > > > > > > > >>>
> > > > > > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > > > > > client.
> > > > > > > > > I am
> > > > > > > > > >>> not sure if there are other use cases for those timestamps.
> > > > An
> > > > > > > > > alternative
> > > > > > > > > >>> is to avoid exposing those timestamps to the client. The
> > > > group
> > > > > > > > > coordinator
> > > > > > > > > >>> will categorize the partitions using those timestamps and
> > > > > > include the
> > > > > > > > > >>> categorization in the assigned partitions in
> > > > > > GroupHeartbeatResponse.
> > > > > > > > > >>>
> > > > > > > > > >>> JR4. It would be useful to support this feature on existing
> > > > > > groups as
> > > > > > > > > well.
> > > > > > > > > >>> For example, any partition without a creation timestamp can
> > > > > > always be
> > > > > > > > > >>> categorized as an existing partition.
> > > > > > > > > >>>
> > > > > > > > > >>> Jun
> > > > > > > > > >>>
> > > > > > > > > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com>
> > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hello Andrew,
> > > > > > > > > >>>>
> > > > > > > > > >>>> Thanks for your feedback,
> > > > > > > > > >>>>>> I've just been re-reading this KIP to ensure that share
> > > > group
> > > > > > > > > support is
> > > > > > > > > >>>>>> not needed. I've discovered that we made a mistake
> > > > > > implementing
> > > > > > > > > KIP-932
> > > > > > > > > >>>> in
> > > > > > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > > > > > really tight
> > > > > > > > > >>>> enough
> > > > > > > > > >>>>>> not to require KIP-1327.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Could you walk us through the specific area where the
> > > > mistake
> > > > > > was
> > > > > > > > > found?
> > > > > > > > > >>>> That would
> > > > > > > > > >>>> help us assess whether it naturally fits within the scope of
> > > > > > KIP-1327
> > > > > > > > > or
> > > > > > > > > >>>> should be addressed
> > > > > > > > > >>>> separately.
> > > > > > > > > >>>>
> > > > > > > > > >>>> AS1 I have updated the KIP
> > > > > > > > > >>>>
> > > > > > > > > >>>> Best Regards,
> > > > > > > > > >>>> Jiunn-Yang
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49
> > > > 寫道:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> hi Andrew
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> I've discovered that we made a mistake implementing
> > > > KIP-932
> > > > > > in this
> > > > > > > > > >>>> area,
> > > > > > > > > >>>>> so we'll rectify that and check whether it's really tight
> > > > > > enough not
> > > > > > > > > to
> > > > > > > > > >>>>> require KIP-1327.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Would you mind sharing more details about that? If it's not
> > > > > > anything
> > > > > > > > > >>>> close
> > > > > > > > > >>>>> to NP-hard, we could definitely look into covering it
> > > > within
> > > > > > KIP-1327
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Best,
> > > > > > > > > >>>>> Chia-Ping
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三
> > > > > > 下午7:49寫道:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Hi Jiunn-Yang,
> > > > > > > > > >>>>>> I've just been re-reading this KIP to ensure that share
> > > > group
> > > > > > > > > support is
> > > > > > > > > >>>>>> not needed. I've discovered that we made a mistake
> > > > > > implementing
> > > > > > > > > KIP-932
> > > > > > > > > >>>> in
> > > > > > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > > > > > really tight
> > > > > > > > > >>>> enough
> > > > > > > > > >>>>>> not to require KIP-1327.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> One trivial comments for consistency.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > > > > > > > > represented by
> > > > > > > > > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Thanks,
> > > > > > > > > >>>>>> Andrew
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > > > > > > > > >>>>>>> Hi all,
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Manually bumping this thread.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Best Regards,
> > > > > > > > > >>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Hello chia,
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日
> > > > 凌晨12:47
> > > > > > 寫道:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> hi Jiunn-Yang
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> When the config is set on a cluster that has not yet
> > > > been
> > > > > > > > > >>>>>> upgraded... classification cannot occur... the consumer
> > > > falls
> > > > > > back
> > > > > > > > > to
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> base auto.offset.reset for the affected partitions. No
> > > > > > exception is
> > > > > > > > > >>>> thrown,
> > > > > > > > > >>>>>> and no operational disruption results.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Existing group can't take advantage of this excellent
> > > > new
> > > > > > > > > >>>>>> configuration. Allowing users to modify the group creation
> > > > > > time
> > > > > > > > > might be
> > > > > > > > > >>>>>> overkill. Instead, we could print a useful warning
> > > > message to
> > > > > > guide
> > > > > > > > > >>>> users.
> > > > > > > > > >>>>>> For example, we can suggest that they re-create the group
> > > > > > with their
> > > > > > > > > >>>>>> existing committed offsets
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> Protocol changes
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Would you mind listing those RPC changes in a table
> > > > format?
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> The full interaction matrix between the base policy
> > > > and
> > > > > > the
> > > > > > > > > >>>>>> new-partition policy is:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Please add a filed to describe the target scenario when
> > > > > > using
> > > > > > > > > these
> > > > > > > > > >>>>>> policies
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Best,
> > > > > > > > > >>>>>>>>> Chia-Ping
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > > > > > > > > >>>>>>>>>> Hello Jun, chia,
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for
> > > > the
> > > > > > new
> > > > > > > > > >>>>>>>>>> approach, PTAL
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日
> > > > 上午8:23
> > > > > > 寫道:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> hi Jun
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Yes, your approach is great. I think the combination
> > > > of
> > > > > > latest
> > > > > > > > > (for
> > > > > > > > > >>>>>> existing partitions) and by_duration (for new partitions)
> > > > can
> > > > > > > > > address
> > > > > > > > > >>>> 99%
> > > > > > > > > >>>>>> of the complaints I have heard regarding this issue.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Also, leveraging the group creation time here opens
> > > > the
> > > > > > door to
> > > > > > > > > >>>>>> implementing a new policy based on timestamp seek in the
> > > > > > future,
> > > > > > > > > should
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>> community want to pursue that.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Thanks for your patience and constructive feedback.
> > > > We
> > > > > > will
> > > > > > > > > update
> > > > > > > > > >>>>>> the KIP accordingly.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> Best, Chia-Ping
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日
> > > > > > 清晨5:11
> > > > > > > > > 寫道:
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Thanks for the reply.
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > > > > > > > > configure a
> > > > > > > > > >>>>>> different
> > > > > > > > > >>>>>>>>>>>> offset policy for existing partitions vs new
> > > > partitions.
> > > > > > > > > However,
> > > > > > > > > >>>>>> using
> > > > > > > > > >>>>>>>>>>>> group creation time to capture that seems more
> > > > > > intuitive.
> > > > > > > > > Here is
> > > > > > > > > >>>>>> another
> > > > > > > > > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and
> > > > > > categorize
> > > > > > > > > new
> > > > > > > > > >>>>>> partitions
> > > > > > > > > >>>>>>>>>>>> based on group creation time. Introduce
> > > > > > > > > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose
> > > > > > values
> > > > > > > > > can be
> > > > > > > > > >>>>>> earliest,
> > > > > > > > > >>>>>>>>>>>> latest and by_duration, the same as
> > > > auto.offset.reset.
> > > > > > Users
> > > > > > > > > can
> > > > > > > > > >>>>>> set
> > > > > > > > > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if
> > > > > > they
> > > > > > > > > want to
> > > > > > > > > >>>>>> guarantee
> > > > > > > > > >>>>>>>>>>>> no data loss on new partitions. They can also use
> > > > > > by_duration
> > > > > > > > > to
> > > > > > > > > >>>>>> set an
> > > > > > > > > >>>>>>>>>>>> upper bound on the backlog replayed, which can be
> > > > > > different
> > > > > > > > > from
> > > > > > > > > >>>>>> that of
> > > > > > > > > >>>>>>>>>>>> the existing partitions. This will address your
> > > > concern
> > > > > > about
> > > > > > > > > too
> > > > > > > > > >>>>>> much
> > > > > > > > > >>>>>>>>>>>> backlog being replayed when the offsets are lost.
> > > > What
> > > > > > do you
> > > > > > > > > >>>>>> think?
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > > > > > > > > >>>>>> chia7712@gmail.com> wrote:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> hi Jun
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> The most important part of this story is how users
> > > > > > should
> > > > > > > > > expect
> > > > > > > > > >>>>>> the data
> > > > > > > > > >>>>>>>>>>>>> they can see when using the latest or by_duration
> > > > > > policy with
> > > > > > > > > >>>>>> expanded
> > > > > > > > > >>>>>>>>>>>>> partitions.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss,
> > > > > > but it is
> > > > > > > > > >>>>>>>>>>>>> non-deterministic, which means users will either
> > > > read
> > > > > > too
> > > > > > > > > many
> > > > > > > > > >>>>>> historical
> > > > > > > > > >>>>>>>>>>>>> records from existing partitions or lose some
> > > > records
> > > > > > from
> > > > > > > > > >>>>>> expanded
> > > > > > > > > >>>>>>>>>>>>> partitions.
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > > > > > > > > >>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>>>>>>> is a bit hard to understand, and that is why I
> > > > > > preferred
> > > > > > > > > having a
> > > > > > > > > >>>>>> whole new
> > > > > > > > > >>>>>>>>>>>>> policy based entirely on group creation time
> > > > (KIP-1282)
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>> Chia-Ping
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > > 2026年6月16日週二
> > > > > > > > > 上午1:08寫道:
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to
> > > > understand
> > > > > > the
> > > > > > > > > value
> > > > > > > > > >>>>>> of the new
> > > > > > > > > >>>>>>>>>>>>>> configs with the KIP.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't
> > > > want
> > > > > > to
> > > > > > > > > miss
> > > > > > > > > >>>>>> the data if
> > > > > > > > > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing
> > > > > > partition
> > > > > > > > > is
> > > > > > > > > >>>>>> easy to
> > > > > > > > > >>>>>>>>>>>>>> understand because it relates to retention time.
> > > > The
> > > > > > > > > backlog for
> > > > > > > > > >>>>>> the new
> > > > > > > > > >>>>>>>>>>>>>> partition is a bit subtle to understand since it
> > > > > > depends on
> > > > > > > > > the
> > > > > > > > > >>>>>> metadata
> > > > > > > > > >>>>>>>>>>>>>> refresh delay. To set
> > > > auto.offset.reset.max.age.ms
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > > > >>>>>>> ,
> > > > > > > > > >>>>>>>>>>>>>> the user needs to
> > > > > > > > > >>>>>>>>>>>>>> understand the metadata refresh delay on the
> > > > consumer
> > > > > > side
> > > > > > > > > and
> > > > > > > > > >>>>>> use it to
> > > > > > > > > >>>>>>>>>>>>>> set the config.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the
> > > > same
> > > > > > value
> > > > > > > > > for
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three
> > > > > > issues with
> > > > > > > > > >>>> this
> > > > > > > > > >>>>>>>>>>>>>> approach.
> > > > > > > > > >>>>>>>>>>>>>> 1. It computes the seek target client-side as
> > > > now() -
> > > > > > > > > duration,
> > > > > > > > > >>>>>> which
> > > > > > > > > >>>>>>>>>>>>>> introduces clock skew across consumers and forces
> > > > > > operators
> > > > > > > > > to
> > > > > > > > > >>>>>> choose
> > > > > > > > > >>>>>>>>>>>>>> overly large durations, causing unnecessary
> > > > > > reprocessing.
> > > > > > > > > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each
> > > > retry,
> > > > > > so
> > > > > > > > > failed
> > > > > > > > > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target
> > > > > > forward and
> > > > > > > > > >>>>>> potentially
> > > > > > > > > >>>>>>>>>>>>>> miss records produced between attempts.
> > > > > > > > > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without
> > > > > > committed
> > > > > > > > > >>>>>> offsets, and
> > > > > > > > > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > > > > > > > > long-existing
> > > > > > > > > >>>>>> partitions
> > > > > > > > > >>>>>>>>>>>>>> newly assigned to the group, leading to
> > > > unnecessary
> > > > > > replay.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated
> > > > by
> > > > > > adding
> > > > > > > > > a bit
> > > > > > > > > >>>>>> buffer to
> > > > > > > > > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > > > > > > > > improving the
> > > > > > > > > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh
> > > > > > delay is
> > > > > > > > > >>>>>> typically low
> > > > > > > > > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer
> > > > > > and tens
> > > > > > > > > of
> > > > > > > > > >>>>>> seconds
> > > > > > > > > >>>>>>>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading
> > > > that
> > > > > > much
> > > > > > > > > >>>>>> backlog for new
> > > > > > > > > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the
> > > > same
> > > > > > for
> > > > > > > > > existing
> > > > > > > > > >>>>>>>>>>>>>> partitions.
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> So, instead of introducing a new config, could we
> > > > just
> > > > > > > > > reuse the
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>>>>>>>> config with better documentation and/or
> > > > > > implementation?
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <
> > > > > > s7133700@gmail.com>
> > > > > > > > > >>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Hello Jun,
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> You're right that group creation time is the more
> > > > > > intuitive
> > > > > > > > > >>>>>> answer at
> > > > > > > > > >>>>>>>>>>>>>>> first glance,
> > > > > > > > > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions
> > > > that
> > > > > > > > > "predate
> > > > > > > > > >>>>>> the group"
> > > > > > > > > >>>>>>>>>>>>>>> vs partitions
> > > > > > > > > >>>>>>>>>>>>>>> "created during group runtime," which directly
> > > > > > points to a
> > > > > > > > > >>>>>>>>>>>>>> group-lifecycle
> > > > > > > > > >>>>>>>>>>>>>>> classifier.
> > > > > > > > > >>>>>>>>>>>>>>> I'd like to walk through why we landed on
> > > > partition
> > > > > > age,
> > > > > > > > > and
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>>>>>>>>> trade-offs we considered.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> This covers the metadata blindness window, but
> > > > has
> > > > > > issues
> > > > > > > > > the
> > > > > > > > > >>>>>> KIP
> > > > > > > > > >>>>>>>>>>>>>>> currently documents
> > > > > > > > > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock
> > > > > > skew
> > > > > > > > > across
> > > > > > > > > >>>>>> consumers.
> > > > > > > > > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > > > > > > > potentially
> > > > > > > > > >>>>>> missing
> > > > > > > > > >>>>>>>>>>>>>>> records produced between
> > > > > > > > > >>>>>>>>>>>>>>> attempts.
> > > > > > > > > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without
> > > > > > committed
> > > > > > > > > >>>>>> offsets,
> > > > > > > > > >>>>>>>>>>>>>>> including pre-existing partitions
> > > > > > > > > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary
> > > > > > replay.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively
> > > > > > running.
> > > > > > > > > Our
> > > > > > > > > >>>>>> concern
> > > > > > > > > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> T=0:         Group created.
> > > > > > > > > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected,
> > > > > > etc.).
> > > > > > > > > >>>>>>>>>>>>>>> T=50:        Partition added during the idle
> > > > window.
> > > > > > > > > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Under group creation time, the new partition is
> > > > > > classified
> > > > > > > > > as
> > > > > > > > > >>>>>> new
> > > > > > > > > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying
> > > > > > everything
> > > > > > > > > from
> > > > > > > > > >>>>>> T=50.
> > > > > > > > > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also
> > > > > > accumulated
> > > > > > > > > >>>>>> data that
> > > > > > > > > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely
> > > > the
> > > > > > > > > contract
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no
> > > > principled
> > > > > > reason
> > > > > > > > > to
> > > > > > > > > >>>>>> treat
> > > > > > > > > >>>>>>>>>>>>>>> the new partition differently; both contain
> > > > backlog
> > > > > > > > > accumulated
> > > > > > > > > >>>>>> during
> > > > > > > > > >>>>>>>>>>>>>>> the same idle window.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog”
> > > > principle
> > > > > > you
> > > > > > > > > raised
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has
> > > > tolerated
> > > > > > some
> > > > > > > > > backlog
> > > > > > > > > >>>>>> on
> > > > > > > > > >>>>>>>>>>>>>>> every other partition during the same idle
> > > > period;
> > > > > > forcing
> > > > > > > > > >>>>>> 0-backlog
> > > > > > > > > >>>>>>>>>>>>>>> tolerance only on new partitions would be
> > > > > > inconsistent with
> > > > > > > > > >>>> that
> > > > > > > > > >>>>>>>>>>>>>>> tolerance.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent
> > > > data
> > > > > > loss
> > > > > > > > > >>>> window,
> > > > > > > > > >>>>>>>>>>>>>>> the gap between partition creation and the
> > > > consumer’s
> > > > > > > > > metadata
> > > > > > > > > >>>>>>>>>>>>>>> refresh. Within this window, data loss is
> > > > genuinely
> > > > > > > > > silent: the
> > > > > > > > > >>>>>>>>>>>>>>> consumer had no opportunity to know about the
> > > > > > partition.
> > > > > > > > > >>>>>> Outside this
> > > > > > > > > >>>>>>>>>>>>>>> window, missing data reflects either:
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with
> > > > idle
> > > > > > > > > consumers,
> > > > > > > > > >>>>>> or
> > > > > > > > > >>>>>>>>>>>>>>> - (b) an operational issue to surface via
> > > > > > monitoring, not
> > > > > > > > > via
> > > > > > > > > >>>>>> reset
> > > > > > > > > >>>>>>>>>>>>>> policy.
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> We did not choose partition age because it is
> > > > more
> > > > > > elegant
> > > > > > > > > than
> > > > > > > > > >>>>>> group
> > > > > > > > > >>>>>>>>>>>>>>> creation time — we chose it because its failure
> > > > mode
> > > > > > > > > (requires
> > > > > > > > > >>>> a
> > > > > > > > > >>>>>>>>>>>>>>> threshold) is
> > > > > > > > > >>>>>>>>>>>>>>> less invasive than the failure mode of group
> > > > > > creation time
> > > > > > > > > >>>>>> (overrides
> > > > > > > > > >>>>>>>>>>>>>>> user-stated
> > > > > > > > > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > > 2026年6月13日
> > > > > > 上午11:52
> > > > > > > > > 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Hi Jun,
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Relying on both creation times will create an
> > > > > > inconsistent
> > > > > > > > > >>>>>> scenario. A
> > > > > > > > > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long
> > > > sleep
> > > > > > will
> > > > > > > > > seek
> > > > > > > > > >>>>>> to the
> > > > > > > > > >>>>>>>>>>>>>>>> beginning for the partitions created later than
> > > > the
> > > > > > group.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to
> > > > fix
> > > > > > the
> > > > > > > > > >>>>>> inconsistency
> > > > > > > > > >>>>>>>>>>>>>>> using a
> > > > > > > > > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach
> > > > a
> > > > > > > > > consensus,
> > > > > > > > > >>>>>> KIP-1327
> > > > > > > > > >>>>>>>>>>>>>>> goes
> > > > > > > > > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent
> > > > > > users
> > > > > > > > > from
> > > > > > > > > >>>>>> falling
> > > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > > >>>>>>>>>>>>>>>> that pitfall.
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > > > > 2026年6月13日週六
> > > > > > > > > >>>>>> 上午6:49寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late
> > > > reply.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> JR1. The design of
> > > > auto.offset.reset.max.age.ms
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> still feels weird to
> > > > > > > > > >>>>>>>>>>>>>>> me.
> > > > > > > > > >>>>>>>>>>>>>>>>> It
> > > > > > > > > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing
> > > > based on
> > > > > > the
> > > > > > > > > >>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>> creation
> > > > > > > > > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be
> > > > > > based on
> > > > > > > > > the
> > > > > > > > > >>>>>> group
> > > > > > > > > >>>>>>>>>>>>>>> creation
> > > > > > > > > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is
> > > > > > created
> > > > > > > > > are
> > > > > > > > > >>>>>> existing
> > > > > > > > > >>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>> all partitions created after the group creation
> > > > > > are new
> > > > > > > > > >>>>>> partitions.
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <
> > > > > > s7133700@gmail.com>
> > > > > > > > > >>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no
> > > > > > further
> > > > > > > > > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16
> > > > 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common
> > > > > > operational
> > > > > > > > > task,
> > > > > > > > > >>>>>> not an
> > > > > > > > > >>>>>>>>>>>>>> edge
> > > > > > > > > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section
> > > > > > accordingly.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日
> > > > > > 下午5:49 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be
> > > > useful
> > > > > > to
> > > > > > > > > clarify
> > > > > > > > > >>>>>> that
> > > > > > > > > >>>>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>>>> is a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case,
> > > > which
> > > > > > > > > further
> > > > > > > > > >>>>>>>>>>>>>> demonstrates
> > > > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common
> > > > operational
> > > > > > task in
> > > > > > > > > >>>>>> Kafka: To
> > > > > > > > > >>>>>>>>>>>>>>>>> balance
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are
> > > > > > typically
> > > > > > > > > >>>>>> created with a
> > > > > > > > > >>>>>>>>>>>>>>>>>> moderate
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic
> > > > > > grows
> > > > > > > > > over
> > > > > > > > > >>>>>> time, it
> > > > > > > > > >>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>> often
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of
> > > > partitions
> > > > > > to
> > > > > > > > > >>>>>> accommodate the
> > > > > > > > > >>>>>>>>>>>>>>>>> higher
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> workload.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Regards
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> Jian
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六
> > > > 22:31写道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the
> > > > > > KIP!
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > > > > 2026年5月30日
> > > > > > > > > 晚上8:29
> > > > > > > > > >>>>>> 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot"
> > > > and
> > > > > > "cold"
> > > > > > > > > when
> > > > > > > > > >>>>>>>>>>>>>> describing
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you
> > > > are
> > > > > > using
> > > > > > > > > them
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>>>>>>>>>> describe
> > > > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the
> > > > > > records, but
> > > > > > > > > >>>>>> applying
> > > > > > > > > >>>>>>>>>>>>>> these
> > > > > > > > > >>>>>>>>>>>>>>>>>> terms
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit
> > > > unnatural.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't
> > > > > > really care
> > > > > > > > > >>>>>> whether a
> > > > > > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only
> > > > expectation
> > > > > > is
> > > > > > > > > that
> > > > > > > > > >>>>>> they won't
> > > > > > > > > >>>>>>>>>>>>>>>>>> silently
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the
> > > > topic
> > > > > > during
> > > > > > > > > their
> > > > > > > > > >>>>>> active
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> consumption.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六
> > > > > > 下午12:30寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated
> > > > the
> > > > > > KIP
> > > > > > > > > >>>>>> motivation
> > > > > > > > > >>>>>>>>>>>>>>> section.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org>
> > > > 於
> > > > > > > > > 2026年5月30日
> > > > > > > > > >>>>>> 凌晨1:12
> > > > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a
> > > > > > stronger
> > > > > > > > > >>>>>> motivation
> > > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that
> > > > not
> > > > > > all
> > > > > > > > > >>>>>> partitions
> > > > > > > > > >>>>>>>>>>>>>> without
> > > > > > > > > >>>>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly
> > > > > > expanded
> > > > > > > > > >>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>> (hot)
> > > > > > > > > >>>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a
> > > > partition the
> > > > > > > > > consumer
> > > > > > > > > >>>>>> has
> > > > > > > > > >>>>>>>>>>>>>> never
> > > > > > > > > >>>>>>>>>>>>>>>>> seen
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)."
> > > > Why
> > > > > > is the
> > > > > > > > > hot
> > > > > > > > > >>>>>>>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration
> > > > > > policy is
> > > > > > > > > also
> > > > > > > > > >>>>>>>>>>>>>>> insufficient
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> because:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() -
> > > > > > duration)
> > > > > > > > > varies
> > > > > > > > > >>>>>> across
> > > > > > > > > >>>>>>>>>>>>>> nodes
> > > > > > > > > >>>>>>>>>>>>>>>>>> due
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set
> > > > an
> > > > > > overly
> > > > > > > > > large
> > > > > > > > > >>>>>>>>>>>>>> duration,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> causing
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client
> > > > > > recalculates the
> > > > > > > > > seek
> > > > > > > > > >>>>>> time on
> > > > > > > > > >>>>>>>>>>>>>>>>> retry,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward
> > > > and
> > > > > > risking
> > > > > > > > > data
> > > > > > > > > >>>>>> loss."
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are
> > > > rare.
> > > > > > If
> > > > > > > > > these
> > > > > > > > > >>>>>> issues
> > > > > > > > > >>>>>>>>>>>>>>>>> persist,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> more
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere.
> > > > Rare
> > > > > > > > > >>>>>> situations don't
> > > > > > > > > >>>>>>>>>>>>>>>>> need a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> common
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > > > > > > > situations,
> > > > > > > > > >>>>>> they can
> > > > > > > > > >>>>>>>>>>>>>>>>>> implement
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > ConsumerRebalanceListener.onPartitionsAssigned().
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > > > > > > > >>>>>> s7133700@gmail.com>
> > > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the
> > > > returned
> > > > > > value
> > > > > > > > > >>>>>> should
> > > > > > > > > >>>>>>>>>>>>>> always
> > > > > > > > > >>>>>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the
> > > > > > KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > > > > > positive
> > > > > > > > > >>>>>> any
> > > > > > > > > >>>>>>>>>>>>>>>>>> field
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can
> > > > detect
> > > > > > the
> > > > > > > > > version
> > > > > > > > > >>>>>>>>>>>>>> mismatch
> > > > > > > > > >>>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> during
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment,
> > > > > > which
> > > > > > > > > occurs
> > > > > > > > > >>>>>> inside
> > > > > > > > > >>>>>>>>>>>>>>>>> poll().
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > > > > > > > UnsupportedVersionException
> > > > > > > > > >>>>>> from
> > > > > > > > > >>>>>>>>>>>>>> poll().
> > > > > > > > > >>>>>>>>>>>>>>>>>> I’ll
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > > > 於
> > > > > > > > > 2026年5月17日
> > > > > > > > > >>>>>> 下午4:50
> > > > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1):
> > > > The
> > > > > > age of
> > > > > > > > > this
> > > > > > > > > >>>>>>>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>>>> in
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by
> > > > the
> > > > > > broker
> > > > > > > > > as
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> -
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if
> > > > the
> > > > > > broker
> > > > > > > > > >>>>>> does not
> > > > > > > > > >>>>>>>>>>>>>>>>> support
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time
> > > > is
> > > > > > > > > unknown.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the
> > > > returned
> > > > > > value
> > > > > > > > > >>>>>> should
> > > > > > > > > >>>>>>>>>>>>>> always
> > > > > > > > > >>>>>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > > > > > positive
> > > > > > > > > >>>>>> any
> > > > > > > > > >>>>>>>>>>>>>>>>>> field
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter
> > > > > > UnsupportedVersionException
> > > > > > > > > when
> > > > > > > > > >>>>>> calling
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design
> > > > > > change
> > > > > > > > > based on
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>> discussion
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the
> > > > > > new-partition
> > > > > > > > > >>>> reset
> > > > > > > > > >>>>>>>>>>>>>> behavior
> > > > > > > > > >>>>>>>>>>>>>>>>>> from
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> now applies to all
> > > > > > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration,
> > > > none).
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the
> > > > > > consumer
> > > > > > > > > resets
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> .partitions
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions,
> > > > the
> > > > > > base
> > > > > > > > > >>>>>>>>>>>>>> auto.offset.reset
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > > > > > > > represented
> > > > > > > > > >>>>>> by a
> > > > > > > > > >>>>>>>>>>>>>>> separate
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>> .partitions,
> > > > > > > > > >>>>>>>>>>>>>> currently fixed to
> > > > > > > > > >>>>>>>>>>>>>>>>> earliest).
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> This
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the
> > > > > > behavior to a
> > > > > > > > > >>>>>> public
> > > > > > > > > >>>>>>>>>>>>>>>>>> user-facing
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > > > 於
> > > > > > > > > 2026年5月16日
> > > > > > > > > >>>>>> 清晨7:46
> > > > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The
> > > > proposal
> > > > > > from me
> > > > > > > > > is
> > > > > > > > > >>>>>> listed
> > > > > > > > > >>>>>>>>>>>>>>> below:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>> .partitions
> > > > > > > > > >>>>>>>>>>>>>> with a default value
> > > > > > > > > >>>>>>>>>>>>>>>>> of
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from
> > > > both
> > > > > > > > > >>>>>> by_duration and
> > > > > > > > > >>>>>>>>>>>>>>>>> latest,
> > > > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> it
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > > > > > > > >>>>>> auto.offset.reset=earliest.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>> .partitions
> > > > > > > > > >>>>>>>>>>>>>> as an internal
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>>>>>>>> .partitions=earliest
> > > > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss
> > > > > > the use
> > > > > > > > > cases
> > > > > > > > > >>>>>> of
> > > > > > > > > >>>>>>>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>>>>>> values
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> in a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs,
> > > > auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>> .partitions
> > > > > > > > > >>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > > > > > > > > >>>>>>> ,
> > > > > > > > > >>>>>>>>>>>>>> will be applied to all for
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via
> > > > dev
> > > > > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the
> > > > KIP
> > > > > > says
> > > > > > > > > "When
> > > > > > > > > >>>>>> a Kafka
> > > > > > > > > >>>>>>>>>>>>>>>>> topic
> > > > > > > > > >>>>>>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers
> > > > using
> > > > > > the
> > > > > > > > > latest
> > > > > > > > > >>>>>> auto
> > > > > > > > > >>>>>>>>>>>>>> offset
> > > > > > > > > >>>>>>>>>>>>>>>>>> reset
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records
> > > > > > produced to
> > > > > > > > > those
> > > > > > > > > >>>>>>>>>>>>>> partitions
> > > > > > > > > >>>>>>>>>>>>>>>>>> before
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a
> > > > user
> > > > > > sets
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec,
> > > > > > the same
> > > > > > > > > >>>>>> record loss
> > > > > > > > > >>>>>>>>>>>>>>>>> issue
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> could
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking
> > > > > > auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>> .partitions
> > > > > > > > > >>>>>>>>>>>>>> will
> > > > > > > > > >>>>>>>>>>>>>>> take
> > > > > > > > > >>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> same
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a
> > > > user
> > > > > > could
> > > > > > > > > set
> > > > > > > > > >>>>>> it
> > > > > > > > > >>>>>>>>>>>>>>>>>> by_duration if
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM
> > > > > > Chia-Ping
> > > > > > > > > Tsai <
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might
> > > > be
> > > > > > missing
> > > > > > > > > >>>>>> something
> > > > > > > > > >>>>>>>>>>>>>>>>>> important
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> from
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with
> > > > me as
> > > > > > I try
> > > > > > > > > to
> > > > > > > > > >>>>>> clarify
> > > > > > > > > >>>>>>>>>>>>>> with
> > > > > > > > > >>>>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>> few
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for
> > > > > > extending
> > > > > > > > > this
> > > > > > > > > >>>>>> logic
> > > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>> other
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> reset
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies
> > > > like
> > > > > > > > > earliest
> > > > > > > > > >>>>>> or
> > > > > > > > > >>>>>>>>>>>>>>>>> by_duration
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> don't
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent
> > > > data
> > > > > > loss
> > > > > > > > > issue
> > > > > > > > > >>>>>> when a
> > > > > > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect
> > > > users
> > > > > > to
> > > > > > > > > >>>>>> configure for
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>> .partitions?
> > > > > > > > > >>>>>>>>>>>>>> If they set it to
> > > > > > > > > >>>>>>>>>>>>>>>>> earliest
> > > > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same
> > > > edge
> > > > > > > > > cases. For
> > > > > > > > > >>>>>>>>>>>>>> example,
> > > > > > > > > >>>>>>>>>>>>>>>>> if a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new
> > > > > > partition is
> > > > > > > > > >>>>>> created
> > > > > > > > > >>>>>>>>>>>>>> during
> > > > > > > > > >>>>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to
> > > > skip
> > > > > > to
> > > > > > > > > latest
> > > > > > > > > >>>>>> when
> > > > > > > > > >>>>>>>>>>>>>>>>> resuming,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> rather
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just
> > > > because the
> > > > > > > > > partition
> > > > > > > > > >>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>> technically
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > > > > > > > > introducing a
> > > > > > > > > >>>>>> max.age
> > > > > > > > > >>>>>>>>>>>>>>>>>> threshold.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> It
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define
> > > > what
> > > > > > is
> > > > > > > > > >>>>>> genuinely
> > > > > > > > > >>>>>>>>>>>>>>> "hot/new"
> > > > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't
> > > > > > seen yet.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao
> > > > via dev
> > > > > > > > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find
> > > > > > auto.offset.reset.latest.max.age a
> > > > > > > > > bit
> > > > > > > > > >>>>>> weird. It
> > > > > > > > > >>>>>>>>>>>>>>>>> only
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest.
> > > > > > However, it
> > > > > > > > > seems
> > > > > > > > > >>>>>> that the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> motivation
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is
> > > > > > set to
> > > > > > > > > other
> > > > > > > > > >>>>>> values
> > > > > > > > > >>>>>>>>>>>>>> like
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to
> > > > have a
> > > > > > > > > separate
> > > > > > > > > >>>>>> way to
> > > > > > > > > >>>>>>>>>>>>>>> control
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> newly
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions
> > > > > > when the
> > > > > > > > > >>>> group
> > > > > > > > > >>>>>>>>>>>>>> starts.
> > > > > > > > > >>>>>>>>>>>>>>>>>> Have we
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > > > > > > > auto.offset.reset.new
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>> .partitions?
> > > > > > > > > >>>>>>>>>>>>>>>>>> If
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset
> > > > reset
> > > > > > policy
> > > > > > > > > >>>>>> defaults to
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user
> > > > could
> > > > > > set it
> > > > > > > > > >>>>>> explicitly
> > > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> customize
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM
> > > > 黃竣陽 <
> > > > > > > > > >>>>>> s7133700@gmail.com
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this
> > > > > > thread.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於
> > > > > > 2026年5月1日
> > > > > > > > > 晚上10:37
> > > > > > > > > >>>>>> 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from
> > > > v13 to
> > > > > > v14.
> > > > > > > > > The
> > > > > > > > > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64,
> > > > > > default -1),
> > > > > > > > > >>>>>> computed
> > > > > > > > > >>>>>>>>>>>>>>>>>> server-side
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> by
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > > > > > > > >>>> partition_creation_time.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat
> > > > > > flow.
> > > > > > > > > when
> > > > > > > > > >>>>>>>>>>>>>>>>>> MembershipManager
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly
> > > > > > invalidates the
> > > > > > > > > >>>>>> metadata for
> > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> affected
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh
> > > > MetadataRequest
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset
> > > > > > decision,
> > > > > > > > > even
> > > > > > > > > >>>>>> if the
> > > > > > > > > >>>>>>>>>>>>>>>>> topic
> > > > > > > > > >>>>>>>>>>>>>>>>>> ID
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the
> > > > broker's
> > > > > > maximum
> > > > > > > > > >>>>>> supported
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at
> > > > > > connection
> > > > > > > > > time.
> > > > > > > > > >>>>>> If the
> > > > > > > > > >>>>>>>>>>>>>>>>>> negotiated
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not
> > > > support
> > > > > > > > > >>>>>> PartitionAgeMs at
> > > > > > > > > >>>>>>>>>>>>>> all
> > > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> can
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than
> > > > silently
> > > > > > falling
> > > > > > > > > >>>>>> back to
> > > > > > > > > >>>>>>>>>>>>>> latest
> > > > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any
> > > > > > operator-visible
> > > > > > > > > signal.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes
> > > > in
> > > > > > the
> > > > > > > > > KIP.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > > > > > chia7712@gmail.com> 於
> > > > > > > > > >>>>>> 2026年4月29日
> > > > > > > > > >>>>>>>>>>>>>>> 下午4:04
> > > > > > > > > >>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of
> > > > > > moving the
> > > > > > > > > >>>>>> 'age'
> > > > > > > > > >>>>>>>>>>>>>>>>> resolution
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata
> > > > API
> > > > > > to keep
> > > > > > > > > the
> > > > > > > > > >>>>>> control
> > > > > > > > > >>>>>>>>>>>>>>>>> plane
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before,
> > > > is
> > > > > > > > > introducing
> > > > > > > > > >>>>>>>>>>>>>>> inter-broker
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> clock
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach
> > > > provided
> > > > > > a
> > > > > > > > > single
> > > > > > > > > >>>>>> source of
> > > > > > > > > >>>>>>>>>>>>>>>>> truth
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this
> > > > time
> > > > > > skew
> > > > > > > > > >>>>>> should be
> > > > > > > > > >>>>>>>>>>>>>>>>>> negligible.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will
> > > > likely be
> > > > > > > > > >>>>>> configured in
> > > > > > > > > >>>>>>>>>>>>>>> minutes
> > > > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in
> > > > milliseconds)
> > > > > > between
> > > > > > > > > >>>>>> brokers
> > > > > > > > > >>>>>>>>>>>>>> won't
> > > > > > > > > >>>>>>>>>>>>>>>>>> impact
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > > > > > > > dev@kafka.apache.org>
> > > > > > > > > >>>>>> 於
> > > > > > > > > >>>>>>>>>>>>>>>>> 2026年4月29日
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really
> > > > followed
> > > > > > the
> > > > > > > > > >>>>>> previous
> > > > > > > > > >>>>>>>>>>>>>>>>>> conversation
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but I
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly
> > > > understand
> > > > > > the
> > > > > > > > > flow
> > > > > > > > > >>>>>> with the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP.
> > > > There
> > > > > > is a
> > > > > > > > > new
> > > > > > > > > >>>>>> boolean;
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>> KIP
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned
> > > > only
> > > > > > when
> > > > > > > > > this
> > > > > > > > > >>>>>> boolean
> > > > > > > > > >>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>> set.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer
> > > > > > receives
> > > > > > > > > a new
> > > > > > > > > >>>>>>>>>>>>>> partition,
> > > > > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> will
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean
> > > > set
> > > > > > to
> > > > > > > > > receive
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>>>>>>>> ages.
> > > > > > > > > >>>>>>>>>>>>>>> Is
> > > > > > > > > >>>>>>>>>>>>>>>>>> my
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps
> > > > > > clarify the
> > > > > > > > > flow
> > > > > > > > > >>>>>> and
> > > > > > > > > >>>>>>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>>>>>>>> explain
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g.
> > > > list
> > > > > > > > > offsets,
> > > > > > > > > >>>>>> fetch
> > > > > > > > > >>>>>>>>>>>>>>>>> offsets,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is
> > > > > > correct, I
> > > > > > > > > >>>>>> wonder if
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat
> > > > API
> > > > > > is the
> > > > > > > > > right
> > > > > > > > > >>>>>> place
> > > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > > >>>>>>>>>>>>>>>>>> this
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> given
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > > > > > > > Alternatively,
> > > > > > > > > >>>>>> it could
> > > > > > > > > >>>>>>>>>>>>>>>>> simply
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we
> > > > should be
> > > > > > > > > rather
> > > > > > > > > >>>>>> cautious
> > > > > > > > > >>>>>>>>>>>>>>> about
> > > > > > > > > >>>>>>>>>>>>>>>>>> not
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat
> > > > API
> > > > > > with
> > > > > > > > > >>>>>> unrelated
> > > > > > > > > >>>>>>>>>>>>>>> concepts.
> > > > > > > > > >>>>>>>>>>>>>>>>>> The
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> API
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for
> > > > assigning
> > > > > > or
> > > > > > > > > revoking
> > > > > > > > > >>>>>>>>>>>>>>> partitions.
> > > > > > > > > >>>>>>>>>>>>>>>>>> The
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> fact
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > > > > > > > corresponding
> > > > > > > > > >>>>>> Streams
> > > > > > > > > >>>>>>>>>>>>>> API
> > > > > > > > > >>>>>>>>>>>>>>>>>> also
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right.
> > > > > > What
> > > > > > > > > would we
> > > > > > > > > >>>>>> do if
> > > > > > > > > >>>>>>>>>>>>>> we
> > > > > > > > > >>>>>>>>>>>>>>>>>> want to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at
> > > > 12:28 AM
> > > > > > > > > >>>>>> Muralidhar Basani
> > > > > > > > > >>>>>>>>>>>>>>> via
> > > > > > > > > >>>>>>>>>>>>>>>>>> dev
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org>
> > > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great
> > > > kip.
> > > > > > Good to
> > > > > > > > > know
> > > > > > > > > >>>>>> about
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>> gap.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version
> > > > > > bump for
> > > > > > > > > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on
> > > > > > response,
> > > > > > > > > >>>>>> PartitionAges
> > > > > > > > > >>>>>>>>>>>>>> on
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version
> > > > > > bump?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config,
> > > > is
> > > > > > there a
> > > > > > > > > >>>>>> recommended
> > > > > > > > > >>>>>>>>>>>>>>>>> value
> > > > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it
> > > > should
> > > > > > based
> > > > > > > > > on the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of
> > > > > > javadocs I
> > > > > > > > > >>>>>> guess.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are
> > > > no
> > > > > > > > > changes to
> > > > > > > > > >>>>>> Kafka
> > > > > > > > > >>>>>>>>>>>>>>>>> Streams,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> would
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > > > > > > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > > > > > > > >>>>>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users
> > > > > > configure
> > > > > > > > > it?
> > > > > > > > > >>>>>> This is
> > > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>> most
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users
> > > > might
> > > > > > easily
> > > > > > > > > >>>>>> mistakenly
> > > > > > > > > >>>>>>>>>>>>>>>>>> configure
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing
> > > > "the
> > > > > > > > > consumer
> > > > > > > > > >>>>>> falls
> > > > > > > > > >>>>>>>>>>>>>> back
> > > > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being
> > > > changed
> > > > > > > > > >>>>>> per-partition
> > > > > > > > > >>>>>>>>>>>>>> which
> > > > > > > > > >>>>>>>>>>>>>>>>>> isn't
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something
> > > > like
> > > > > > > > > "consumer
> > > > > > > > > >>>>>> resolves
> > > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> initial
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that
> > > > > > partition" as if
> > > > > > > > > >>>>>> earliest
> > > > > > > > > >>>>>>>>>>>>>> was
> > > > > > > > > >>>>>>>>>>>>>>>>>> applied
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and
> > > > > > auto.offset.reset
> > > > > > > > > >>>>>> config is
> > > > > > > > > >>>>>>>>>>>>>>>>>> unchanged.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at
> > > > > > 2:48 PM 黃竣陽 <
> > > > > > > > > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to
> > > > > > include
> > > > > > > > > this
> > > > > > > > > >>>>>> change.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > > > > > chia7712@apache.org>
> > > > > > > > > 於
> > > > > > > > > >>>>>>>>>>>>>> 2026年4月28日
> > > > > > > > > >>>>>>>>>>>>>>>>>> 晚上8:03
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose
> > > > the
> > > > > > > > > partition
> > > > > > > > > >>>>>> creation
> > > > > > > > > >>>>>>>>>>>>>>> time
> > > > > > > > > >>>>>>>>>>>>>>>>>> via
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be
> > > > valuable
> > > > > > for
> > > > > > > > > users
> > > > > > > > > >>>> to
> > > > > > > > > >>>>>>>>>>>>>> diagnose
> > > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > > > > > > > >>>>>> auto.offset.reset.latest.max.age
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽
> > > > > > wrote:
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> > > > > > discussion
> > > > > > > > > on
> > > > > > > > > >>>>>> KIP-1327
> > > > > > > > > >>>>>>>>>>>>>>>>>> Prevent
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>> Hot
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for
> > > > Latest
> > > > > > Policy
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > >
> > > > > >
> > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
> > > > > > introduces
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets
> > > > the
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy
> > > > > > distinguish
> > > > > > > > > newly
> > > > > > > > > >>>>>> expanded
> > > > > > > > > >>>>>>>>>>>>>>>>> (hot)
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones.
> > > > > > Partitions
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the
> > > > configured
> > > > > > > > > threshold
> > > > > > > > > >>>>>>>>>>>>>>> automatically
> > > > > > > > > >>>>>>>>>>>>>>>>>> fall
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent
> > > > > > data loss
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion
> > > > > > without
> > > > > > > > > forcing
> > > > > > > > > >>>>>> a full
> > > > > > > > > >>>>>>>>>>>>>>>>>> historical
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > 
> > 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
Hi Andrew,

Yes, we can figure out the expanded partitions using ShareGroupStatePartitionMetadataInfo, but the true question is - should a share group need a different policy for expanded partitions?

The challenge encountered by normal consumers with the latest policy is that sometimes users want to reset to 0, and sometimes it is required to set to latest.

1) The former happens when an expanded partition is dynamically added to an active group. Users often expect to consume from the very beginning of this new partition to avoid missing data, even if the overall group policy is latest.
2) The latter happens when a group's state is completely wiped (e.g., when offsets expire for normal consumers, or when a share group is manually deleted and recreated). In this scenario, they want a consistent latest offset behavior across both old and newly expanded partitions.

Given these distinct user expectations, does it make sense to consider a more granular offset reset strategy specifically for partition expansion in share groups?

Best,
Chia-Ping

On 2026/06/29 07:20:40 Andrew Schofield wrote:
> Hi Jun,
> This is handled because share groups do not support regex subscriptions :) I think this KIP is a nice improvement, but I also think that we don't need it for share groups with their current behaviour. I am perfectly amenable to having it extended to cover share groups too, but I don't think it will make a practical difference.
> 
> Thanks,
> Andrew
> 
> On 2026/06/26 18:57:54 Jun Rao via dev wrote:
> > Hi, Andrew,
> > 
> > There is another case where a share consumer subscribes to a regex and a
> > new topic is later created that matches that regex. Intuitively, they
> > should use the earliest offset for consumption. But how does the share
> > coordinator distinguish these from existing topics explicitly added by the
> > user without the creation timestamp?
> > 
> > Thanks,
> > 
> > Jun
> > 
> > On Fri, Jun 26, 2026 at 11:00 AM Andrew Schofield <sc...@apache.org>
> > wrote:
> > 
> > > Hi Jun,
> > > When a user adds an existing topic to a share group subscription, we do
> > > use the auto.offset.reset policy because those partitions already exist and
> > > may well have data on them. We only use 0 in the case where a topic already
> > > being consumed gained some additional new partitions. These partitions will
> > > be empty at the time of creation, so zero seems like a reasonable starting
> > > point. Partitions being consumed in a share group always have a known
> > > starting position once assigned. We don't have the equivalent of a consumer
> > > group not having a committed offset.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On 2026/06/26 17:36:31 Jun Rao via dev wrote:
> > > > Hi, Andrew,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > I am not sure it's always a good idea to start new partitions with
> > > offset 0
> > > > for the share group. For example, once a share group starts, a user could
> > > > add a subscription to an existing topic. That existing topic could have a
> > > > large backlog and starting it with offset 0 forces the user to consume
> > > that
> > > > entire backlog. In KIP-1327, such a partition will be treated as an
> > > > existing partition and will follow the auto.offset.reset policy. This
> > > seems
> > > > more intuitive. It would be useful to consider if the share group should
> > > > adopt the same policy.
> > > >
> > > > Jun
> > > >
> > > > On Fri, Jun 26, 2026 at 3:42 AM Andrew Schofield <sc...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Jiunn-Yang et al.,
> > > > > Some background on the problem I hinted at for the KIP-932
> > > implementation.
> > > > > KIP-932 states "If the number of partitions is increased for a topic
> > > with a
> > > > > subscription in a share group, the SPSO for the newly created
> > > > > share-partitions is initialized to 0 (which is of course both the
> > > earliest
> > > > > and latest offset for an empty topic-partition). This means there is no
> > > > > doubt about what happens when the number of partitions is increased."
> > > > >
> > > > > The problem was that we had not implemented the use of 0, but instead
> > > were
> > > > > using the share.auto.offset.reset config to choose the initial offset
> > > in
> > > > > this case. We write a ShareGroupStatePartitionMetadata record when a
> > > topic
> > > > > is first being initialized for consumption by a share group, and this
> > > > > contains a list of the partitions which we are initializing. As a
> > > result, I
> > > > > believe we can already tell when partitions are added, and we do know
> > > when
> > > > > to initialize to 0 as it says in the KIP. We will put in a patch that
> > > uses
> > > > > 0 as intended.
> > > > >
> > > > > I do not mind whether we want to incorporate share groups into
> > > KIP-1327,
> > > > > but I do not think that it is necessary to do so.
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > > >
> > > > > On 2026/06/26 04:31:28 Chia-Ping Tsai wrote:
> > > > > > > JR5. Since this changes the on-disk data format for
> > > PartitionRecord, we
> > > > > > > need to gate this by a new MV. It also changes the format of
> > > > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not
> > > sure if
> > > > > this
> > > > > > > requires a new MV or a new GV.
> > > > > >
> > > > > > That is a good point. If we want to maintain consistency for the
> > > > > NewPartitions field when the group coordinator migrates between new
> > > and old
> > > > > binaries, a version gate is definitely required. For example:
> > > > > >
> > > > > > GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())
> > > > > >
> > > > > > The group coordinator should NOT set the NewPartitions field when
> > > the gv
> > > > > is smaller than gv_2
> > > > > >
> > > > > > Best,
> > > > > > Chia-Ping
> > > > > >
> > > > > > On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> > > > > > > Hi, Jiunn-Yang,
> > > > > > >
> > > > > > > JR2. We should try to maintain consistency across our APIs. So, it
> > > > > would be
> > > > > > > useful to understand if the user-facing part of this KIP is what
> > > share
> > > > > > > consumer wants too. It would be problematic if we proceed with this
> > > > > > > approach only for the share consumer to choose a different one
> > > later.
> > > > > > >
> > > > > > > JR3. I was thinking that if we add the NewPartitions field in
> > > > > > > GroupHeartbeatResponse, there is no need to expose the
> > > group/partition
> > > > > > > creation timestamp to other RPCs and the Cli.
> > > > > > >
> > > > > > > JR5. Since this changes the on-disk data format for
> > > PartitionRecord, we
> > > > > > > need to gate this by a new MV. It also changes the format of
> > > > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not
> > > sure if
> > > > > this
> > > > > > > requires a new MV or a new GV.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hello Jun, chia,
> > > > > > > >
> > > > > > > > Thanks for your feedback.
> > > > > > > >
> > > > > > > > JR3 JR4 I have updated the KIP.
> > > > > > > >
> > > > > > > > JR2 The new-partition detection mechanism is reusable. The
> > > > > coordinator
> > > > > > > > already has
> > > > > > > > access to both timestamps required for classification, so the
> > > > > > > > determination can be made
> > > > > > > > server-side without introducing any new RPCs. The main
> > > difference is
> > > > > where
> > > > > > > > the policy is
> > > > > > > > enforced: modern consumer resolves it on the client, whereas a
> > > share
> > > > > group
> > > > > > > > resolves the
> > > > > > > > starting offset on the partition leader. As a result, the
> > > > > classification
> > > > > > > > decision would need to
> > > > > > > > be propagated through the existing share-state initialization
> > > path.
> > > > > We
> > > > > > > > would also need to persist
> > > > > > > > the group's creation time as part of the share-group state, since
> > > > > that
> > > > > > > > information is not stored today.
> > > > > > > >
> > > > > > > > None of this is a blocker, but it does require changes to the
> > > > > share-state
> > > > > > > > format and introduces
> > > > > > > > a share-specific configuration. Given that additional scope, I
> > > would
> > > > > > > > prefer to keep share groups out
> > > > > > > > of scope for this KIP and address them in a follow-up effort,
> > > while
> > > > > > > > documenting the relevant reuse
> > > > > > > > opportunities here.
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > > Jiunn-Yang
> > > > > > > >
> > > > > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > > > > > > > >
> > > > > > > > >> It inspires me to think about mirror clusters. What if the
> > > mirror
> > > > > group
> > > > > > > > is created 'after' the expanded partitions? In this scenario,
> > > the new
> > > > > > > > configuration 'max.age.ms' could handle it if users are aware
> > > of the
> > > > > > > > migration and set a suitable time interval. Of course, this would
> > > > > require
> > > > > > > > both creation times to be exposed to clients.
> > > > > > > > >
> > > > > > > > > Thinking more about the mirror cluster scenario we discussed, I
> > > > > realize
> > > > > > > > it introduces significant complexity. Handling it properly would
> > > > > require a
> > > > > > > > tolerance time window rather than a strict time interval to
> > > account
> > > > > for
> > > > > > > > replication lag. To prevent scope creep and keep KIP-1327
> > > focused on
> > > > > its
> > > > > > > > primary goal, I think it would be best to defer the
> > > > > multi-cluster/mirroring
> > > > > > > > scenario and explore those approaches in a separate KIP
> > > > > > > > >
> > > > > > > > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > > > > > > > >> hi all,
> > > > > > > > >>
> > > > > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > > > > client.
> > > > > > > > I am
> > > > > > > > >>> not sure if there are other use cases for those timestamps.
> > > An
> > > > > > > > alternative
> > > > > > > > >>> is to avoid exposing those timestamps to the client. The
> > > group
> > > > > > > > coordinator
> > > > > > > > >>> will categorize the partitions using those timestamps and
> > > > > include the
> > > > > > > > >>> categorization in the assigned partitions in
> > > > > GroupHeartbeatResponse.
> > > > > > > > >>
> > > > > > > > >> It inspires me to think about mirror clusters. What if the
> > > mirror
> > > > > group
> > > > > > > > is created 'after' the expanded partitions? In this scenario,
> > > the new
> > > > > > > > configuration 'max.age.ms' could handle it if users are aware
> > > of the
> > > > > > > > migration and set a suitable time interval. Of course, this would
> > > > > require
> > > > > > > > both creation times to be exposed to clients.
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Chia-Ping
> > > > > > > > >>
> > > > > > > > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > > > > > > > >>> Hi, Jiunn-Yang,
> > > > > > > > >>>
> > > > > > > > >>> Thanks for the updated KIP.
> > > > > > > > >>>
> > > > > > > > >>> JR2. Share consumer (KIP-932) states: "If the number of
> > > > > partitions is
> > > > > > > > >>> increased for a topic with a subscription in a share group,
> > > the
> > > > > SPSO
> > > > > > > > for
> > > > > > > > >>> the newly created share-partitions is initialized to 0 ".
> > > > > However, this
> > > > > > > > >>> part has not been implemented yet. Currently, all partitions
> > > > > without
> > > > > > > > offset
> > > > > > > > >>> follows the strategy given by share.auto.offset.reset. So, we
> > > > > have an
> > > > > > > > >>> opportunity to see if it's better to align this KIP's changes
> > > > > with the
> > > > > > > > >>> share consumer.
> > > > > > > > >>> JR2.1 The first part is whether customizing the offset for
> > > new
> > > > > > > > partitions
> > > > > > > > >>> to something other than 0 is useful in the share consumer. It
> > > > > seems
> > > > > > > > that
> > > > > > > > >>> the motivation in this KIP applies to share consumer too.
> > > > > > > > >>> JR2.2 The second part is how new partitions are detected. The
> > > > > mechanism
> > > > > > > > >>> introduced in this KIP seems quite effective. A group
> > > > > coordinator could
> > > > > > > > >>> pick up newly created partitions for an existing topic or
> > > newly
> > > > > created
> > > > > > > > >>> topics matching the regex. Those partitions are categorized
> > > as
> > > > > new
> > > > > > > > >>> partitions and will follow the strategy given by
> > > > > > > > >>> auto.offset.reset.new.partitions.
> > > > > > > > >>> This makes sense since those partitions typically don't have
> > > a
> > > > > large
> > > > > > > > >>> backlog. A consumer could also additionally subscribe to an
> > > > > existing
> > > > > > > > topic
> > > > > > > > >>> in an existing group. Those partitions will be categorized as
> > > > > existing
> > > > > > > > >>> partitions. This also makes sense since those partitions
> > > could
> > > > > have a
> > > > > > > > large
> > > > > > > > >>> backlog and it is better to follow the strategy given by
> > > > > > > > auto.offset.reset. It
> > > > > > > > >>> will be useful to think through whether the logic for
> > > > > categorizing new
> > > > > > > > >>> partitions should also be reused in the share consumer.
> > > > > > > > >>>
> > > > > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > > > > client.
> > > > > > > > I am
> > > > > > > > >>> not sure if there are other use cases for those timestamps.
> > > An
> > > > > > > > alternative
> > > > > > > > >>> is to avoid exposing those timestamps to the client. The
> > > group
> > > > > > > > coordinator
> > > > > > > > >>> will categorize the partitions using those timestamps and
> > > > > include the
> > > > > > > > >>> categorization in the assigned partitions in
> > > > > GroupHeartbeatResponse.
> > > > > > > > >>>
> > > > > > > > >>> JR4. It would be useful to support this feature on existing
> > > > > groups as
> > > > > > > > well.
> > > > > > > > >>> For example, any partition without a creation timestamp can
> > > > > always be
> > > > > > > > >>> categorized as an existing partition.
> > > > > > > > >>>
> > > > > > > > >>> Jun
> > > > > > > > >>>
> > > > > > > > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com>
> > > wrote:
> > > > > > > > >>>
> > > > > > > > >>>> Hello Andrew,
> > > > > > > > >>>>
> > > > > > > > >>>> Thanks for your feedback,
> > > > > > > > >>>>>> I've just been re-reading this KIP to ensure that share
> > > group
> > > > > > > > support is
> > > > > > > > >>>>>> not needed. I've discovered that we made a mistake
> > > > > implementing
> > > > > > > > KIP-932
> > > > > > > > >>>> in
> > > > > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > > > > really tight
> > > > > > > > >>>> enough
> > > > > > > > >>>>>> not to require KIP-1327.
> > > > > > > > >>>>
> > > > > > > > >>>> Could you walk us through the specific area where the
> > > mistake
> > > > > was
> > > > > > > > found?
> > > > > > > > >>>> That would
> > > > > > > > >>>> help us assess whether it naturally fits within the scope of
> > > > > KIP-1327
> > > > > > > > or
> > > > > > > > >>>> should be addressed
> > > > > > > > >>>> separately.
> > > > > > > > >>>>
> > > > > > > > >>>> AS1 I have updated the KIP
> > > > > > > > >>>>
> > > > > > > > >>>> Best Regards,
> > > > > > > > >>>> Jiunn-Yang
> > > > > > > > >>>>
> > > > > > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49
> > > 寫道:
> > > > > > > > >>>>>
> > > > > > > > >>>>> hi Andrew
> > > > > > > > >>>>>
> > > > > > > > >>>>>> I've discovered that we made a mistake implementing
> > > KIP-932
> > > > > in this
> > > > > > > > >>>> area,
> > > > > > > > >>>>> so we'll rectify that and check whether it's really tight
> > > > > enough not
> > > > > > > > to
> > > > > > > > >>>>> require KIP-1327.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Would you mind sharing more details about that? If it's not
> > > > > anything
> > > > > > > > >>>> close
> > > > > > > > >>>>> to NP-hard, we could definitely look into covering it
> > > within
> > > > > KIP-1327
> > > > > > > > >>>>>
> > > > > > > > >>>>> Best,
> > > > > > > > >>>>> Chia-Ping
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三
> > > > > 下午7:49寫道:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Hi Jiunn-Yang,
> > > > > > > > >>>>>> I've just been re-reading this KIP to ensure that share
> > > group
> > > > > > > > support is
> > > > > > > > >>>>>> not needed. I've discovered that we made a mistake
> > > > > implementing
> > > > > > > > KIP-932
> > > > > > > > >>>> in
> > > > > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > > > > really tight
> > > > > > > > >>>> enough
> > > > > > > > >>>>>> not to require KIP-1327.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> One trivial comments for consistency.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > > > > > > > represented by
> > > > > > > > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Thanks,
> > > > > > > > >>>>>> Andrew
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > > > > > > > >>>>>>> Hi all,
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Manually bumping this thread.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Best Regards,
> > > > > > > > >>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Hello chia,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Best Regards,
> > > > > > > > >>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日
> > > 凌晨12:47
> > > > > 寫道:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> hi Jiunn-Yang
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> When the config is set on a cluster that has not yet
> > > been
> > > > > > > > >>>>>> upgraded... classification cannot occur... the consumer
> > > falls
> > > > > back
> > > > > > > > to
> > > > > > > > >>>> the
> > > > > > > > >>>>>> base auto.offset.reset for the affected partitions. No
> > > > > exception is
> > > > > > > > >>>> thrown,
> > > > > > > > >>>>>> and no operational disruption results.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Existing group can't take advantage of this excellent
> > > new
> > > > > > > > >>>>>> configuration. Allowing users to modify the group creation
> > > > > time
> > > > > > > > might be
> > > > > > > > >>>>>> overkill. Instead, we could print a useful warning
> > > message to
> > > > > guide
> > > > > > > > >>>> users.
> > > > > > > > >>>>>> For example, we can suggest that they re-create the group
> > > > > with their
> > > > > > > > >>>>>> existing committed offsets
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> Protocol changes
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Would you mind listing those RPC changes in a table
> > > format?
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> The full interaction matrix between the base policy
> > > and
> > > > > the
> > > > > > > > >>>>>> new-partition policy is:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Please add a filed to describe the target scenario when
> > > > > using
> > > > > > > > these
> > > > > > > > >>>>>> policies
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Best,
> > > > > > > > >>>>>>>>> Chia-Ping
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > > > > > > > >>>>>>>>>> Hello Jun, chia,
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for
> > > the
> > > > > new
> > > > > > > > >>>>>>>>>> approach, PTAL
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日
> > > 上午8:23
> > > > > 寫道:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> hi Jun
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Yes, your approach is great. I think the combination
> > > of
> > > > > latest
> > > > > > > > (for
> > > > > > > > >>>>>> existing partitions) and by_duration (for new partitions)
> > > can
> > > > > > > > address
> > > > > > > > >>>> 99%
> > > > > > > > >>>>>> of the complaints I have heard regarding this issue.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Also, leveraging the group creation time here opens
> > > the
> > > > > door to
> > > > > > > > >>>>>> implementing a new policy based on timestamp seek in the
> > > > > future,
> > > > > > > > should
> > > > > > > > >>>> the
> > > > > > > > >>>>>> community want to pursue that.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks for your patience and constructive feedback.
> > > We
> > > > > will
> > > > > > > > update
> > > > > > > > >>>>>> the KIP accordingly.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Best, Chia-Ping
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日
> > > > > 清晨5:11
> > > > > > > > 寫道:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Thanks for the reply.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > > > > > > > configure a
> > > > > > > > >>>>>> different
> > > > > > > > >>>>>>>>>>>> offset policy for existing partitions vs new
> > > partitions.
> > > > > > > > However,
> > > > > > > > >>>>>> using
> > > > > > > > >>>>>>>>>>>> group creation time to capture that seems more
> > > > > intuitive.
> > > > > > > > Here is
> > > > > > > > >>>>>> another
> > > > > > > > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and
> > > > > categorize
> > > > > > > > new
> > > > > > > > >>>>>> partitions
> > > > > > > > >>>>>>>>>>>> based on group creation time. Introduce
> > > > > > > > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose
> > > > > values
> > > > > > > > can be
> > > > > > > > >>>>>> earliest,
> > > > > > > > >>>>>>>>>>>> latest and by_duration, the same as
> > > auto.offset.reset.
> > > > > Users
> > > > > > > > can
> > > > > > > > >>>>>> set
> > > > > > > > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if
> > > > > they
> > > > > > > > want to
> > > > > > > > >>>>>> guarantee
> > > > > > > > >>>>>>>>>>>> no data loss on new partitions. They can also use
> > > > > by_duration
> > > > > > > > to
> > > > > > > > >>>>>> set an
> > > > > > > > >>>>>>>>>>>> upper bound on the backlog replayed, which can be
> > > > > different
> > > > > > > > from
> > > > > > > > >>>>>> that of
> > > > > > > > >>>>>>>>>>>> the existing partitions. This will address your
> > > concern
> > > > > about
> > > > > > > > too
> > > > > > > > >>>>>> much
> > > > > > > > >>>>>>>>>>>> backlog being replayed when the offsets are lost.
> > > What
> > > > > do you
> > > > > > > > >>>>>> think?
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > > > > > > > >>>>>> chia7712@gmail.com> wrote:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> hi Jun
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> The most important part of this story is how users
> > > > > should
> > > > > > > > expect
> > > > > > > > >>>>>> the data
> > > > > > > > >>>>>>>>>>>>> they can see when using the latest or by_duration
> > > > > policy with
> > > > > > > > >>>>>> expanded
> > > > > > > > >>>>>>>>>>>>> partitions.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss,
> > > > > but it is
> > > > > > > > >>>>>>>>>>>>> non-deterministic, which means users will either
> > > read
> > > > > too
> > > > > > > > many
> > > > > > > > >>>>>> historical
> > > > > > > > >>>>>>>>>>>>> records from existing partitions or lose some
> > > records
> > > > > from
> > > > > > > > >>>>>> expanded
> > > > > > > > >>>>>>>>>>>>> partitions.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > > > > > > > >>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>>>>>>> is a bit hard to understand, and that is why I
> > > > > preferred
> > > > > > > > having a
> > > > > > > > >>>>>> whole new
> > > > > > > > >>>>>>>>>>>>> policy based entirely on group creation time
> > > (KIP-1282)
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>> Chia-Ping
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > 2026年6月16日週二
> > > > > > > > 上午1:08寫道:
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to
> > > understand
> > > > > the
> > > > > > > > value
> > > > > > > > >>>>>> of the new
> > > > > > > > >>>>>>>>>>>>>> configs with the KIP.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't
> > > want
> > > > > to
> > > > > > > > miss
> > > > > > > > >>>>>> the data if
> > > > > > > > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing
> > > > > partition
> > > > > > > > is
> > > > > > > > >>>>>> easy to
> > > > > > > > >>>>>>>>>>>>>> understand because it relates to retention time.
> > > The
> > > > > > > > backlog for
> > > > > > > > >>>>>> the new
> > > > > > > > >>>>>>>>>>>>>> partition is a bit subtle to understand since it
> > > > > depends on
> > > > > > > > the
> > > > > > > > >>>>>> metadata
> > > > > > > > >>>>>>>>>>>>>> refresh delay. To set
> > > auto.offset.reset.max.age.ms
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > > >>>>>>> ,
> > > > > > > > >>>>>>>>>>>>>> the user needs to
> > > > > > > > >>>>>>>>>>>>>> understand the metadata refresh delay on the
> > > consumer
> > > > > side
> > > > > > > > and
> > > > > > > > >>>>>> use it to
> > > > > > > > >>>>>>>>>>>>>> set the config.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the
> > > same
> > > > > value
> > > > > > > > for
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three
> > > > > issues with
> > > > > > > > >>>> this
> > > > > > > > >>>>>>>>>>>>>> approach.
> > > > > > > > >>>>>>>>>>>>>> 1. It computes the seek target client-side as
> > > now() -
> > > > > > > > duration,
> > > > > > > > >>>>>> which
> > > > > > > > >>>>>>>>>>>>>> introduces clock skew across consumers and forces
> > > > > operators
> > > > > > > > to
> > > > > > > > >>>>>> choose
> > > > > > > > >>>>>>>>>>>>>> overly large durations, causing unnecessary
> > > > > reprocessing.
> > > > > > > > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each
> > > retry,
> > > > > so
> > > > > > > > failed
> > > > > > > > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target
> > > > > forward and
> > > > > > > > >>>>>> potentially
> > > > > > > > >>>>>>>>>>>>>> miss records produced between attempts.
> > > > > > > > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without
> > > > > committed
> > > > > > > > >>>>>> offsets, and
> > > > > > > > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > > > > > > > long-existing
> > > > > > > > >>>>>> partitions
> > > > > > > > >>>>>>>>>>>>>> newly assigned to the group, leading to
> > > unnecessary
> > > > > replay.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated
> > > by
> > > > > adding
> > > > > > > > a bit
> > > > > > > > >>>>>> buffer to
> > > > > > > > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > > > > > > > improving the
> > > > > > > > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh
> > > > > delay is
> > > > > > > > >>>>>> typically low
> > > > > > > > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer
> > > > > and tens
> > > > > > > > of
> > > > > > > > >>>>>> seconds
> > > > > > > > >>>>>>>>>>>>>> with
> > > > > > > > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading
> > > that
> > > > > much
> > > > > > > > >>>>>> backlog for new
> > > > > > > > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the
> > > same
> > > > > for
> > > > > > > > existing
> > > > > > > > >>>>>>>>>>>>>> partitions.
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> So, instead of introducing a new config, could we
> > > just
> > > > > > > > reuse the
> > > > > > > > >>>>>> existing
> > > > > > > > >>>>>>>>>>>>>> config with better documentation and/or
> > > > > implementation?
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <
> > > > > s7133700@gmail.com>
> > > > > > > > >>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Hello Jun,
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> You're right that group creation time is the more
> > > > > intuitive
> > > > > > > > >>>>>> answer at
> > > > > > > > >>>>>>>>>>>>>>> first glance,
> > > > > > > > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions
> > > that
> > > > > > > > "predate
> > > > > > > > >>>>>> the group"
> > > > > > > > >>>>>>>>>>>>>>> vs partitions
> > > > > > > > >>>>>>>>>>>>>>> "created during group runtime," which directly
> > > > > points to a
> > > > > > > > >>>>>>>>>>>>>> group-lifecycle
> > > > > > > > >>>>>>>>>>>>>>> classifier.
> > > > > > > > >>>>>>>>>>>>>>> I'd like to walk through why we landed on
> > > partition
> > > > > age,
> > > > > > > > and
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>>>>>>>> trade-offs we considered.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> This covers the metadata blindness window, but
> > > has
> > > > > issues
> > > > > > > > the
> > > > > > > > >>>>>> KIP
> > > > > > > > >>>>>>>>>>>>>>> currently documents
> > > > > > > > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock
> > > > > skew
> > > > > > > > across
> > > > > > > > >>>>>> consumers.
> > > > > > > > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > > > > > > potentially
> > > > > > > > >>>>>> missing
> > > > > > > > >>>>>>>>>>>>>>> records produced between
> > > > > > > > >>>>>>>>>>>>>>> attempts.
> > > > > > > > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without
> > > > > committed
> > > > > > > > >>>>>> offsets,
> > > > > > > > >>>>>>>>>>>>>>> including pre-existing partitions
> > > > > > > > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary
> > > > > replay.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively
> > > > > running.
> > > > > > > > Our
> > > > > > > > >>>>>> concern
> > > > > > > > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> T=0:         Group created.
> > > > > > > > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected,
> > > > > etc.).
> > > > > > > > >>>>>>>>>>>>>>> T=50:        Partition added during the idle
> > > window.
> > > > > > > > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Under group creation time, the new partition is
> > > > > classified
> > > > > > > > as
> > > > > > > > >>>>>> new
> > > > > > > > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying
> > > > > everything
> > > > > > > > from
> > > > > > > > >>>>>> T=50.
> > > > > > > > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also
> > > > > accumulated
> > > > > > > > >>>>>> data that
> > > > > > > > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely
> > > the
> > > > > > > > contract
> > > > > > > > >>>> of
> > > > > > > > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no
> > > principled
> > > > > reason
> > > > > > > > to
> > > > > > > > >>>>>> treat
> > > > > > > > >>>>>>>>>>>>>>> the new partition differently; both contain
> > > backlog
> > > > > > > > accumulated
> > > > > > > > >>>>>> during
> > > > > > > > >>>>>>>>>>>>>>> the same idle window.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog”
> > > principle
> > > > > you
> > > > > > > > raised
> > > > > > > > >>>>>> in
> > > > > > > > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has
> > > tolerated
> > > > > some
> > > > > > > > backlog
> > > > > > > > >>>>>> on
> > > > > > > > >>>>>>>>>>>>>>> every other partition during the same idle
> > > period;
> > > > > forcing
> > > > > > > > >>>>>> 0-backlog
> > > > > > > > >>>>>>>>>>>>>>> tolerance only on new partitions would be
> > > > > inconsistent with
> > > > > > > > >>>> that
> > > > > > > > >>>>>>>>>>>>>>> tolerance.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent
> > > data
> > > > > loss
> > > > > > > > >>>> window,
> > > > > > > > >>>>>>>>>>>>>>> the gap between partition creation and the
> > > consumer’s
> > > > > > > > metadata
> > > > > > > > >>>>>>>>>>>>>>> refresh. Within this window, data loss is
> > > genuinely
> > > > > > > > silent: the
> > > > > > > > >>>>>>>>>>>>>>> consumer had no opportunity to know about the
> > > > > partition.
> > > > > > > > >>>>>> Outside this
> > > > > > > > >>>>>>>>>>>>>>> window, missing data reflects either:
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with
> > > idle
> > > > > > > > consumers,
> > > > > > > > >>>>>> or
> > > > > > > > >>>>>>>>>>>>>>> - (b) an operational issue to surface via
> > > > > monitoring, not
> > > > > > > > via
> > > > > > > > >>>>>> reset
> > > > > > > > >>>>>>>>>>>>>> policy.
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> We did not choose partition age because it is
> > > more
> > > > > elegant
> > > > > > > > than
> > > > > > > > >>>>>> group
> > > > > > > > >>>>>>>>>>>>>>> creation time — we chose it because its failure
> > > mode
> > > > > > > > (requires
> > > > > > > > >>>> a
> > > > > > > > >>>>>>>>>>>>>>> threshold) is
> > > > > > > > >>>>>>>>>>>>>>> less invasive than the failure mode of group
> > > > > creation time
> > > > > > > > >>>>>> (overrides
> > > > > > > > >>>>>>>>>>>>>>> user-stated
> > > > > > > > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > 2026年6月13日
> > > > > 上午11:52
> > > > > > > > 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Hi Jun,
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Relying on both creation times will create an
> > > > > inconsistent
> > > > > > > > >>>>>> scenario. A
> > > > > > > > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long
> > > sleep
> > > > > will
> > > > > > > > seek
> > > > > > > > >>>>>> to the
> > > > > > > > >>>>>>>>>>>>>>>> beginning for the partitions created later than
> > > the
> > > > > group.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to
> > > fix
> > > > > the
> > > > > > > > >>>>>> inconsistency
> > > > > > > > >>>>>>>>>>>>>>> using a
> > > > > > > > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach
> > > a
> > > > > > > > consensus,
> > > > > > > > >>>>>> KIP-1327
> > > > > > > > >>>>>>>>>>>>>>> goes
> > > > > > > > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent
> > > > > users
> > > > > > > > from
> > > > > > > > >>>>>> falling
> > > > > > > > >>>>>>>>>>>>>> into
> > > > > > > > >>>>>>>>>>>>>>>> that pitfall.
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > > > 2026年6月13日週六
> > > > > > > > >>>>>> 上午6:49寫道:
> > > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late
> > > reply.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> JR1. The design of
> > > auto.offset.reset.max.age.ms
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>>>>>>>> still feels weird to
> > > > > > > > >>>>>>>>>>>>>>> me.
> > > > > > > > >>>>>>>>>>>>>>>>> It
> > > > > > > > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing
> > > based on
> > > > > the
> > > > > > > > >>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>> creation
> > > > > > > > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be
> > > > > based on
> > > > > > > > the
> > > > > > > > >>>>>> group
> > > > > > > > >>>>>>>>>>>>>>> creation
> > > > > > > > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is
> > > > > created
> > > > > > > > are
> > > > > > > > >>>>>> existing
> > > > > > > > >>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>> all partitions created after the group creation
> > > > > are new
> > > > > > > > >>>>>> partitions.
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <
> > > > > s7133700@gmail.com>
> > > > > > > > >>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no
> > > > > further
> > > > > > > > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16
> > > 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common
> > > > > operational
> > > > > > > > task,
> > > > > > > > >>>>>> not an
> > > > > > > > >>>>>>>>>>>>>> edge
> > > > > > > > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section
> > > > > accordingly.
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日
> > > > > 下午5:49 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be
> > > useful
> > > > > to
> > > > > > > > clarify
> > > > > > > > >>>>>> that
> > > > > > > > >>>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>>>> is a
> > > > > > > > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case,
> > > which
> > > > > > > > further
> > > > > > > > >>>>>>>>>>>>>> demonstrates
> > > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > > > > > > > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common
> > > operational
> > > > > task in
> > > > > > > > >>>>>> Kafka: To
> > > > > > > > >>>>>>>>>>>>>>>>> balance
> > > > > > > > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are
> > > > > typically
> > > > > > > > >>>>>> created with a
> > > > > > > > >>>>>>>>>>>>>>>>>> moderate
> > > > > > > > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic
> > > > > grows
> > > > > > > > over
> > > > > > > > >>>>>> time, it
> > > > > > > > >>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>> often
> > > > > > > > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of
> > > partitions
> > > > > to
> > > > > > > > >>>>>> accommodate the
> > > > > > > > >>>>>>>>>>>>>>>>> higher
> > > > > > > > >>>>>>>>>>>>>>>>>>>> workload.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Regards
> > > > > > > > >>>>>>>>>>>>>>>>>>>> Jian
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六
> > > 22:31写道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the
> > > > > KIP!
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > > > 2026年5月30日
> > > > > > > > 晚上8:29
> > > > > > > > >>>>>> 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot"
> > > and
> > > > > "cold"
> > > > > > > > when
> > > > > > > > >>>>>>>>>>>>>> describing
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you
> > > are
> > > > > using
> > > > > > > > them
> > > > > > > > >>>> to
> > > > > > > > >>>>>>>>>>>>>> describe
> > > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the
> > > > > records, but
> > > > > > > > >>>>>> applying
> > > > > > > > >>>>>>>>>>>>>> these
> > > > > > > > >>>>>>>>>>>>>>>>>> terms
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit
> > > unnatural.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't
> > > > > really care
> > > > > > > > >>>>>> whether a
> > > > > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only
> > > expectation
> > > > > is
> > > > > > > > that
> > > > > > > > >>>>>> they won't
> > > > > > > > >>>>>>>>>>>>>>>>>> silently
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the
> > > topic
> > > > > during
> > > > > > > > their
> > > > > > > > >>>>>> active
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> consumption.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六
> > > > > 下午12:30寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated
> > > the
> > > > > KIP
> > > > > > > > >>>>>> motivation
> > > > > > > > >>>>>>>>>>>>>>> section.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org>
> > > 於
> > > > > > > > 2026年5月30日
> > > > > > > > >>>>>> 凌晨1:12
> > > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a
> > > > > stronger
> > > > > > > > >>>>>> motivation
> > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that
> > > not
> > > > > all
> > > > > > > > >>>>>> partitions
> > > > > > > > >>>>>>>>>>>>>> without
> > > > > > > > >>>>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly
> > > > > expanded
> > > > > > > > >>>>>> partition
> > > > > > > > >>>>>>>>>>>>>> (hot)
> > > > > > > > >>>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a
> > > partition the
> > > > > > > > consumer
> > > > > > > > >>>>>> has
> > > > > > > > >>>>>>>>>>>>>> never
> > > > > > > > >>>>>>>>>>>>>>>>> seen
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)."
> > > Why
> > > > > is the
> > > > > > > > hot
> > > > > > > > >>>>>>>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration
> > > > > policy is
> > > > > > > > also
> > > > > > > > >>>>>>>>>>>>>>> insufficient
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> because:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() -
> > > > > duration)
> > > > > > > > varies
> > > > > > > > >>>>>> across
> > > > > > > > >>>>>>>>>>>>>> nodes
> > > > > > > > >>>>>>>>>>>>>>>>>> due
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set
> > > an
> > > > > overly
> > > > > > > > large
> > > > > > > > >>>>>>>>>>>>>> duration,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> causing
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client
> > > > > recalculates the
> > > > > > > > seek
> > > > > > > > >>>>>> time on
> > > > > > > > >>>>>>>>>>>>>>>>> retry,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward
> > > and
> > > > > risking
> > > > > > > > data
> > > > > > > > >>>>>> loss."
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are
> > > rare.
> > > > > If
> > > > > > > > these
> > > > > > > > >>>>>> issues
> > > > > > > > >>>>>>>>>>>>>>>>> persist,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> more
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere.
> > > Rare
> > > > > > > > >>>>>> situations don't
> > > > > > > > >>>>>>>>>>>>>>>>> need a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> common
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > > > > > > situations,
> > > > > > > > >>>>>> they can
> > > > > > > > >>>>>>>>>>>>>>>>>> implement
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > ConsumerRebalanceListener.onPartitionsAssigned().
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > > > > > > >>>>>> s7133700@gmail.com>
> > > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the
> > > returned
> > > > > value
> > > > > > > > >>>>>> should
> > > > > > > > >>>>>>>>>>>>>> always
> > > > > > > > >>>>>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the
> > > > > KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > > > > positive
> > > > > > > > >>>>>> any
> > > > > > > > >>>>>>>>>>>>>>>>>> field
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can
> > > detect
> > > > > the
> > > > > > > > version
> > > > > > > > >>>>>>>>>>>>>> mismatch
> > > > > > > > >>>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> during
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment,
> > > > > which
> > > > > > > > occurs
> > > > > > > > >>>>>> inside
> > > > > > > > >>>>>>>>>>>>>>>>> poll().
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > > > > > > UnsupportedVersionException
> > > > > > > > >>>>>> from
> > > > > > > > >>>>>>>>>>>>>> poll().
> > > > > > > > >>>>>>>>>>>>>>>>>> I’ll
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > > 於
> > > > > > > > 2026年5月17日
> > > > > > > > >>>>>> 下午4:50
> > > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1):
> > > The
> > > > > age of
> > > > > > > > this
> > > > > > > > >>>>>>>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>>>> in
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by
> > > the
> > > > > broker
> > > > > > > > as
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> -
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if
> > > the
> > > > > broker
> > > > > > > > >>>>>> does not
> > > > > > > > >>>>>>>>>>>>>>>>> support
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time
> > > is
> > > > > > > > unknown.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the
> > > returned
> > > > > value
> > > > > > > > >>>>>> should
> > > > > > > > >>>>>>>>>>>>>> always
> > > > > > > > >>>>>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > > > > positive
> > > > > > > > >>>>>> any
> > > > > > > > >>>>>>>>>>>>>>>>>> field
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter
> > > > > UnsupportedVersionException
> > > > > > > > when
> > > > > > > > >>>>>> calling
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design
> > > > > change
> > > > > > > > based on
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>> discussion
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the
> > > > > new-partition
> > > > > > > > >>>> reset
> > > > > > > > >>>>>>>>>>>>>> behavior
> > > > > > > > >>>>>>>>>>>>>>>>>> from
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>>>>>>>> now applies to all
> > > > > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration,
> > > none).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the
> > > > > consumer
> > > > > > > > resets
> > > > > > > > >>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>>>>>>>> .partitions
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions,
> > > the
> > > > > base
> > > > > > > > >>>>>>>>>>>>>> auto.offset.reset
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > > > > > > represented
> > > > > > > > >>>>>> by a
> > > > > > > > >>>>>>>>>>>>>>> separate
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>> .partitions,
> > > > > > > > >>>>>>>>>>>>>> currently fixed to
> > > > > > > > >>>>>>>>>>>>>>>>> earliest).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> This
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the
> > > > > behavior to a
> > > > > > > > >>>>>> public
> > > > > > > > >>>>>>>>>>>>>>>>>> user-facing
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > > 於
> > > > > > > > 2026年5月16日
> > > > > > > > >>>>>> 清晨7:46
> > > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The
> > > proposal
> > > > > from me
> > > > > > > > is
> > > > > > > > >>>>>> listed
> > > > > > > > >>>>>>>>>>>>>>> below:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>> .partitions
> > > > > > > > >>>>>>>>>>>>>> with a default value
> > > > > > > > >>>>>>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from
> > > both
> > > > > > > > >>>>>> by_duration and
> > > > > > > > >>>>>>>>>>>>>>>>> latest,
> > > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> it
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > > > > > > >>>>>> auto.offset.reset=earliest.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>> .partitions
> > > > > > > > >>>>>>>>>>>>>> as an internal
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>>>>>>>> .partitions=earliest
> > > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss
> > > > > the use
> > > > > > > > cases
> > > > > > > > >>>>>> of
> > > > > > > > >>>>>>>>>>>>>> other
> > > > > > > > >>>>>>>>>>>>>>>>>> values
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> in a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs,
> > > auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>> .partitions
> > > > > > > > >>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > > > > > > > >>>>>>> ,
> > > > > > > > >>>>>>>>>>>>>> will be applied to all for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via
> > > dev
> > > > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the
> > > KIP
> > > > > says
> > > > > > > > "When
> > > > > > > > >>>>>> a Kafka
> > > > > > > > >>>>>>>>>>>>>>>>> topic
> > > > > > > > >>>>>>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers
> > > using
> > > > > the
> > > > > > > > latest
> > > > > > > > >>>>>> auto
> > > > > > > > >>>>>>>>>>>>>> offset
> > > > > > > > >>>>>>>>>>>>>>>>>> reset
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records
> > > > > produced to
> > > > > > > > those
> > > > > > > > >>>>>>>>>>>>>> partitions
> > > > > > > > >>>>>>>>>>>>>>>>>> before
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a
> > > user
> > > > > sets
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec,
> > > > > the same
> > > > > > > > >>>>>> record loss
> > > > > > > > >>>>>>>>>>>>>>>>> issue
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> could
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking
> > > > > auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>> .partitions
> > > > > > > > >>>>>>>>>>>>>> will
> > > > > > > > >>>>>>>>>>>>>>> take
> > > > > > > > >>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> same
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a
> > > user
> > > > > could
> > > > > > > > set
> > > > > > > > >>>>>> it
> > > > > > > > >>>>>>>>>>>>>>>>>> by_duration if
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM
> > > > > Chia-Ping
> > > > > > > > Tsai <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might
> > > be
> > > > > missing
> > > > > > > > >>>>>> something
> > > > > > > > >>>>>>>>>>>>>>>>>> important
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> from
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with
> > > me as
> > > > > I try
> > > > > > > > to
> > > > > > > > >>>>>> clarify
> > > > > > > > >>>>>>>>>>>>>> with
> > > > > > > > >>>>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>> few
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for
> > > > > extending
> > > > > > > > this
> > > > > > > > >>>>>> logic
> > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>> other
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> reset
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies
> > > like
> > > > > > > > earliest
> > > > > > > > >>>>>> or
> > > > > > > > >>>>>>>>>>>>>>>>> by_duration
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> don't
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent
> > > data
> > > > > loss
> > > > > > > > issue
> > > > > > > > >>>>>> when a
> > > > > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect
> > > users
> > > > > to
> > > > > > > > >>>>>> configure for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>> .partitions?
> > > > > > > > >>>>>>>>>>>>>> If they set it to
> > > > > > > > >>>>>>>>>>>>>>>>> earliest
> > > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same
> > > edge
> > > > > > > > cases. For
> > > > > > > > >>>>>>>>>>>>>> example,
> > > > > > > > >>>>>>>>>>>>>>>>> if a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new
> > > > > partition is
> > > > > > > > >>>>>> created
> > > > > > > > >>>>>>>>>>>>>> during
> > > > > > > > >>>>>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to
> > > skip
> > > > > to
> > > > > > > > latest
> > > > > > > > >>>>>> when
> > > > > > > > >>>>>>>>>>>>>>>>> resuming,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> rather
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just
> > > because the
> > > > > > > > partition
> > > > > > > > >>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>> technically
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > > > > > > > introducing a
> > > > > > > > >>>>>> max.age
> > > > > > > > >>>>>>>>>>>>>>>>>> threshold.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> It
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define
> > > what
> > > > > is
> > > > > > > > >>>>>> genuinely
> > > > > > > > >>>>>>>>>>>>>>> "hot/new"
> > > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't
> > > > > seen yet.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao
> > > via dev
> > > > > > > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find
> > > > > auto.offset.reset.latest.max.age a
> > > > > > > > bit
> > > > > > > > >>>>>> weird. It
> > > > > > > > >>>>>>>>>>>>>>>>> only
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest.
> > > > > However, it
> > > > > > > > seems
> > > > > > > > >>>>>> that the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> motivation
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is
> > > > > set to
> > > > > > > > other
> > > > > > > > >>>>>> values
> > > > > > > > >>>>>>>>>>>>>> like
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to
> > > have a
> > > > > > > > separate
> > > > > > > > >>>>>> way to
> > > > > > > > >>>>>>>>>>>>>>> control
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> newly
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions
> > > > > when the
> > > > > > > > >>>> group
> > > > > > > > >>>>>>>>>>>>>> starts.
> > > > > > > > >>>>>>>>>>>>>>>>>> Have we
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > > > > > > auto.offset.reset.new
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>>>>>>>>> .partitions?
> > > > > > > > >>>>>>>>>>>>>>>>>> If
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset
> > > reset
> > > > > policy
> > > > > > > > >>>>>> defaults to
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user
> > > could
> > > > > set it
> > > > > > > > >>>>>> explicitly
> > > > > > > > >>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> customize
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM
> > > 黃竣陽 <
> > > > > > > > >>>>>> s7133700@gmail.com
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this
> > > > > thread.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於
> > > > > 2026年5月1日
> > > > > > > > 晚上10:37
> > > > > > > > >>>>>> 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from
> > > v13 to
> > > > > v14.
> > > > > > > > The
> > > > > > > > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64,
> > > > > default -1),
> > > > > > > > >>>>>> computed
> > > > > > > > >>>>>>>>>>>>>>>>>> server-side
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> by
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > > > > > > >>>> partition_creation_time.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat
> > > > > flow.
> > > > > > > > when
> > > > > > > > >>>>>>>>>>>>>>>>>> MembershipManager
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly
> > > > > invalidates the
> > > > > > > > >>>>>> metadata for
> > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> affected
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh
> > > MetadataRequest
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset
> > > > > decision,
> > > > > > > > even
> > > > > > > > >>>>>> if the
> > > > > > > > >>>>>>>>>>>>>>>>> topic
> > > > > > > > >>>>>>>>>>>>>>>>>> ID
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the
> > > broker's
> > > > > maximum
> > > > > > > > >>>>>> supported
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at
> > > > > connection
> > > > > > > > time.
> > > > > > > > >>>>>> If the
> > > > > > > > >>>>>>>>>>>>>>>>>> negotiated
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not
> > > support
> > > > > > > > >>>>>> PartitionAgeMs at
> > > > > > > > >>>>>>>>>>>>>> all
> > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> can
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than
> > > silently
> > > > > falling
> > > > > > > > >>>>>> back to
> > > > > > > > >>>>>>>>>>>>>> latest
> > > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any
> > > > > operator-visible
> > > > > > > > signal.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes
> > > in
> > > > > the
> > > > > > > > KIP.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > > > > chia7712@gmail.com> 於
> > > > > > > > >>>>>> 2026年4月29日
> > > > > > > > >>>>>>>>>>>>>>> 下午4:04
> > > > > > > > >>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of
> > > > > moving the
> > > > > > > > >>>>>> 'age'
> > > > > > > > >>>>>>>>>>>>>>>>> resolution
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata
> > > API
> > > > > to keep
> > > > > > > > the
> > > > > > > > >>>>>> control
> > > > > > > > >>>>>>>>>>>>>>>>> plane
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before,
> > > is
> > > > > > > > introducing
> > > > > > > > >>>>>>>>>>>>>>> inter-broker
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> clock
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach
> > > provided
> > > > > a
> > > > > > > > single
> > > > > > > > >>>>>> source of
> > > > > > > > >>>>>>>>>>>>>>>>> truth
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this
> > > time
> > > > > skew
> > > > > > > > >>>>>> should be
> > > > > > > > >>>>>>>>>>>>>>>>>> negligible.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will
> > > likely be
> > > > > > > > >>>>>> configured in
> > > > > > > > >>>>>>>>>>>>>>> minutes
> > > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in
> > > milliseconds)
> > > > > between
> > > > > > > > >>>>>> brokers
> > > > > > > > >>>>>>>>>>>>>> won't
> > > > > > > > >>>>>>>>>>>>>>>>>> impact
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > > > > > > dev@kafka.apache.org>
> > > > > > > > >>>>>> 於
> > > > > > > > >>>>>>>>>>>>>>>>> 2026年4月29日
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really
> > > followed
> > > > > the
> > > > > > > > >>>>>> previous
> > > > > > > > >>>>>>>>>>>>>>>>>> conversation
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but I
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly
> > > understand
> > > > > the
> > > > > > > > flow
> > > > > > > > >>>>>> with the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP.
> > > There
> > > > > is a
> > > > > > > > new
> > > > > > > > >>>>>> boolean;
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>> KIP
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned
> > > only
> > > > > when
> > > > > > > > this
> > > > > > > > >>>>>> boolean
> > > > > > > > >>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>> set.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer
> > > > > receives
> > > > > > > > a new
> > > > > > > > >>>>>>>>>>>>>> partition,
> > > > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> will
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean
> > > set
> > > > > to
> > > > > > > > receive
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>>>>>>>> ages.
> > > > > > > > >>>>>>>>>>>>>>> Is
> > > > > > > > >>>>>>>>>>>>>>>>>> my
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps
> > > > > clarify the
> > > > > > > > flow
> > > > > > > > >>>>>> and
> > > > > > > > >>>>>>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>>>>>>>> explain
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g.
> > > list
> > > > > > > > offsets,
> > > > > > > > >>>>>> fetch
> > > > > > > > >>>>>>>>>>>>>>>>> offsets,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is
> > > > > correct, I
> > > > > > > > >>>>>> wonder if
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat
> > > API
> > > > > is the
> > > > > > > > right
> > > > > > > > >>>>>> place
> > > > > > > > >>>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>>>>>> this
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> given
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > > > > > > Alternatively,
> > > > > > > > >>>>>> it could
> > > > > > > > >>>>>>>>>>>>>>>>> simply
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we
> > > should be
> > > > > > > > rather
> > > > > > > > >>>>>> cautious
> > > > > > > > >>>>>>>>>>>>>>> about
> > > > > > > > >>>>>>>>>>>>>>>>>> not
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat
> > > API
> > > > > with
> > > > > > > > >>>>>> unrelated
> > > > > > > > >>>>>>>>>>>>>>> concepts.
> > > > > > > > >>>>>>>>>>>>>>>>>> The
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> API
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for
> > > assigning
> > > > > or
> > > > > > > > revoking
> > > > > > > > >>>>>>>>>>>>>>> partitions.
> > > > > > > > >>>>>>>>>>>>>>>>>> The
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> fact
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > > > > > > corresponding
> > > > > > > > >>>>>> Streams
> > > > > > > > >>>>>>>>>>>>>> API
> > > > > > > > >>>>>>>>>>>>>>>>>> also
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right.
> > > > > What
> > > > > > > > would we
> > > > > > > > >>>>>> do if
> > > > > > > > >>>>>>>>>>>>>> we
> > > > > > > > >>>>>>>>>>>>>>>>>> want to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at
> > > 12:28 AM
> > > > > > > > >>>>>> Muralidhar Basani
> > > > > > > > >>>>>>>>>>>>>>> via
> > > > > > > > >>>>>>>>>>>>>>>>>> dev
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org>
> > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great
> > > kip.
> > > > > Good to
> > > > > > > > know
> > > > > > > > >>>>>> about
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>> gap.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version
> > > > > bump for
> > > > > > > > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on
> > > > > response,
> > > > > > > > >>>>>> PartitionAges
> > > > > > > > >>>>>>>>>>>>>> on
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version
> > > > > bump?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config,
> > > is
> > > > > there a
> > > > > > > > >>>>>> recommended
> > > > > > > > >>>>>>>>>>>>>>>>> value
> > > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it
> > > should
> > > > > based
> > > > > > > > on the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > > > > > > >>>>>>>>>>>>>> <
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of
> > > > > javadocs I
> > > > > > > > >>>>>> guess.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are
> > > no
> > > > > > > > changes to
> > > > > > > > >>>>>> Kafka
> > > > > > > > >>>>>>>>>>>>>>>>> Streams,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> would
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > > > > > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > > > > > > >>>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users
> > > > > configure
> > > > > > > > it?
> > > > > > > > >>>>>> This is
> > > > > > > > >>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>> most
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users
> > > might
> > > > > easily
> > > > > > > > >>>>>> mistakenly
> > > > > > > > >>>>>>>>>>>>>>>>>> configure
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing
> > > "the
> > > > > > > > consumer
> > > > > > > > >>>>>> falls
> > > > > > > > >>>>>>>>>>>>>> back
> > > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being
> > > changed
> > > > > > > > >>>>>> per-partition
> > > > > > > > >>>>>>>>>>>>>> which
> > > > > > > > >>>>>>>>>>>>>>>>>> isn't
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something
> > > like
> > > > > > > > "consumer
> > > > > > > > >>>>>> resolves
> > > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> initial
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that
> > > > > partition" as if
> > > > > > > > >>>>>> earliest
> > > > > > > > >>>>>>>>>>>>>> was
> > > > > > > > >>>>>>>>>>>>>>>>>> applied
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and
> > > > > auto.offset.reset
> > > > > > > > >>>>>> config is
> > > > > > > > >>>>>>>>>>>>>>>>>> unchanged.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at
> > > > > 2:48 PM 黃竣陽 <
> > > > > > > > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to
> > > > > include
> > > > > > > > this
> > > > > > > > >>>>>> change.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > > > > chia7712@apache.org>
> > > > > > > > 於
> > > > > > > > >>>>>>>>>>>>>> 2026年4月28日
> > > > > > > > >>>>>>>>>>>>>>>>>> 晚上8:03
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose
> > > the
> > > > > > > > partition
> > > > > > > > >>>>>> creation
> > > > > > > > >>>>>>>>>>>>>>> time
> > > > > > > > >>>>>>>>>>>>>>>>>> via
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be
> > > valuable
> > > > > for
> > > > > > > > users
> > > > > > > > >>>> to
> > > > > > > > >>>>>>>>>>>>>> diagnose
> > > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > > > > > > >>>>>> auto.offset.reset.latest.max.age
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽
> > > > > wrote:
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> > > > > discussion
> > > > > > > > on
> > > > > > > > >>>>>> KIP-1327
> > > > > > > > >>>>>>>>>>>>>>>>>> Prevent
> > > > > > > > >>>>>>>>>>>>>>>>>>>>> Hot
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for
> > > Latest
> > > > > Policy
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > >
> > > > >
> > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
> > > > > introduces
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets
> > > the
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy
> > > > > distinguish
> > > > > > > > newly
> > > > > > > > >>>>>> expanded
> > > > > > > > >>>>>>>>>>>>>>>>> (hot)
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones.
> > > > > Partitions
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the
> > > configured
> > > > > > > > threshold
> > > > > > > > >>>>>>>>>>>>>>> automatically
> > > > > > > > >>>>>>>>>>>>>>>>>> fall
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent
> > > > > data loss
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion
> > > > > without
> > > > > > > > forcing
> > > > > > > > >>>>>> a full
> > > > > > > > >>>>>>>>>>>>>>>>>> historical
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Andrew Schofield <sc...@apache.org>.
Hi Jun,
This is handled because share groups do not support regex subscriptions :) I think this KIP is a nice improvement, but I also think that we don't need it for share groups with their current behaviour. I am perfectly amenable to having it extended to cover share groups too, but I don't think it will make a practical difference.

Thanks,
Andrew

On 2026/06/26 18:57:54 Jun Rao via dev wrote:
> Hi, Andrew,
> 
> There is another case where a share consumer subscribes to a regex and a
> new topic is later created that matches that regex. Intuitively, they
> should use the earliest offset for consumption. But how does the share
> coordinator distinguish these from existing topics explicitly added by the
> user without the creation timestamp?
> 
> Thanks,
> 
> Jun
> 
> On Fri, Jun 26, 2026 at 11:00 AM Andrew Schofield <sc...@apache.org>
> wrote:
> 
> > Hi Jun,
> > When a user adds an existing topic to a share group subscription, we do
> > use the auto.offset.reset policy because those partitions already exist and
> > may well have data on them. We only use 0 in the case where a topic already
> > being consumed gained some additional new partitions. These partitions will
> > be empty at the time of creation, so zero seems like a reasonable starting
> > point. Partitions being consumed in a share group always have a known
> > starting position once assigned. We don't have the equivalent of a consumer
> > group not having a committed offset.
> >
> > Thanks,
> > Andrew
> >
> > On 2026/06/26 17:36:31 Jun Rao via dev wrote:
> > > Hi, Andrew,
> > >
> > > Thanks for the reply.
> > >
> > > I am not sure it's always a good idea to start new partitions with
> > offset 0
> > > for the share group. For example, once a share group starts, a user could
> > > add a subscription to an existing topic. That existing topic could have a
> > > large backlog and starting it with offset 0 forces the user to consume
> > that
> > > entire backlog. In KIP-1327, such a partition will be treated as an
> > > existing partition and will follow the auto.offset.reset policy. This
> > seems
> > > more intuitive. It would be useful to consider if the share group should
> > > adopt the same policy.
> > >
> > > Jun
> > >
> > > On Fri, Jun 26, 2026 at 3:42 AM Andrew Schofield <sc...@apache.org>
> > > wrote:
> > >
> > > > Hi Jiunn-Yang et al.,
> > > > Some background on the problem I hinted at for the KIP-932
> > implementation.
> > > > KIP-932 states "If the number of partitions is increased for a topic
> > with a
> > > > subscription in a share group, the SPSO for the newly created
> > > > share-partitions is initialized to 0 (which is of course both the
> > earliest
> > > > and latest offset for an empty topic-partition). This means there is no
> > > > doubt about what happens when the number of partitions is increased."
> > > >
> > > > The problem was that we had not implemented the use of 0, but instead
> > were
> > > > using the share.auto.offset.reset config to choose the initial offset
> > in
> > > > this case. We write a ShareGroupStatePartitionMetadata record when a
> > topic
> > > > is first being initialized for consumption by a share group, and this
> > > > contains a list of the partitions which we are initializing. As a
> > result, I
> > > > believe we can already tell when partitions are added, and we do know
> > when
> > > > to initialize to 0 as it says in the KIP. We will put in a patch that
> > uses
> > > > 0 as intended.
> > > >
> > > > I do not mind whether we want to incorporate share groups into
> > KIP-1327,
> > > > but I do not think that it is necessary to do so.
> > > >
> > > > Thanks,
> > > > Andrew
> > > >
> > > > On 2026/06/26 04:31:28 Chia-Ping Tsai wrote:
> > > > > > JR5. Since this changes the on-disk data format for
> > PartitionRecord, we
> > > > > > need to gate this by a new MV. It also changes the format of
> > > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not
> > sure if
> > > > this
> > > > > > requires a new MV or a new GV.
> > > > >
> > > > > That is a good point. If we want to maintain consistency for the
> > > > NewPartitions field when the group coordinator migrates between new
> > and old
> > > > binaries, a version gate is definitely required. For example:
> > > > >
> > > > > GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())
> > > > >
> > > > > The group coordinator should NOT set the NewPartitions field when
> > the gv
> > > > is smaller than gv_2
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> > > > > > Hi, Jiunn-Yang,
> > > > > >
> > > > > > JR2. We should try to maintain consistency across our APIs. So, it
> > > > would be
> > > > > > useful to understand if the user-facing part of this KIP is what
> > share
> > > > > > consumer wants too. It would be problematic if we proceed with this
> > > > > > approach only for the share consumer to choose a different one
> > later.
> > > > > >
> > > > > > JR3. I was thinking that if we add the NewPartitions field in
> > > > > > GroupHeartbeatResponse, there is no need to expose the
> > group/partition
> > > > > > creation timestamp to other RPCs and the Cli.
> > > > > >
> > > > > > JR5. Since this changes the on-disk data format for
> > PartitionRecord, we
> > > > > > need to gate this by a new MV. It also changes the format of
> > > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not
> > sure if
> > > > this
> > > > > > requires a new MV or a new GV.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > > >
> > > > > > > Hello Jun, chia,
> > > > > > >
> > > > > > > Thanks for your feedback.
> > > > > > >
> > > > > > > JR3 JR4 I have updated the KIP.
> > > > > > >
> > > > > > > JR2 The new-partition detection mechanism is reusable. The
> > > > coordinator
> > > > > > > already has
> > > > > > > access to both timestamps required for classification, so the
> > > > > > > determination can be made
> > > > > > > server-side without introducing any new RPCs. The main
> > difference is
> > > > where
> > > > > > > the policy is
> > > > > > > enforced: modern consumer resolves it on the client, whereas a
> > share
> > > > group
> > > > > > > resolves the
> > > > > > > starting offset on the partition leader. As a result, the
> > > > classification
> > > > > > > decision would need to
> > > > > > > be propagated through the existing share-state initialization
> > path.
> > > > We
> > > > > > > would also need to persist
> > > > > > > the group's creation time as part of the share-group state, since
> > > > that
> > > > > > > information is not stored today.
> > > > > > >
> > > > > > > None of this is a blocker, but it does require changes to the
> > > > share-state
> > > > > > > format and introduces
> > > > > > > a share-specific configuration. Given that additional scope, I
> > would
> > > > > > > prefer to keep share groups out
> > > > > > > of scope for this KIP and address them in a follow-up effort,
> > while
> > > > > > > documenting the relevant reuse
> > > > > > > opportunities here.
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Jiunn-Yang
> > > > > > >
> > > > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > > > > > > >
> > > > > > > >> It inspires me to think about mirror clusters. What if the
> > mirror
> > > > group
> > > > > > > is created 'after' the expanded partitions? In this scenario,
> > the new
> > > > > > > configuration 'max.age.ms' could handle it if users are aware
> > of the
> > > > > > > migration and set a suitable time interval. Of course, this would
> > > > require
> > > > > > > both creation times to be exposed to clients.
> > > > > > > >
> > > > > > > > Thinking more about the mirror cluster scenario we discussed, I
> > > > realize
> > > > > > > it introduces significant complexity. Handling it properly would
> > > > require a
> > > > > > > tolerance time window rather than a strict time interval to
> > account
> > > > for
> > > > > > > replication lag. To prevent scope creep and keep KIP-1327
> > focused on
> > > > its
> > > > > > > primary goal, I think it would be best to defer the
> > > > multi-cluster/mirroring
> > > > > > > scenario and explore those approaches in a separate KIP
> > > > > > > >
> > > > > > > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > > > > > > >> hi all,
> > > > > > > >>
> > > > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > > > client.
> > > > > > > I am
> > > > > > > >>> not sure if there are other use cases for those timestamps.
> > An
> > > > > > > alternative
> > > > > > > >>> is to avoid exposing those timestamps to the client. The
> > group
> > > > > > > coordinator
> > > > > > > >>> will categorize the partitions using those timestamps and
> > > > include the
> > > > > > > >>> categorization in the assigned partitions in
> > > > GroupHeartbeatResponse.
> > > > > > > >>
> > > > > > > >> It inspires me to think about mirror clusters. What if the
> > mirror
> > > > group
> > > > > > > is created 'after' the expanded partitions? In this scenario,
> > the new
> > > > > > > configuration 'max.age.ms' could handle it if users are aware
> > of the
> > > > > > > migration and set a suitable time interval. Of course, this would
> > > > require
> > > > > > > both creation times to be exposed to clients.
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Chia-Ping
> > > > > > > >>
> > > > > > > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > > > > > > >>> Hi, Jiunn-Yang,
> > > > > > > >>>
> > > > > > > >>> Thanks for the updated KIP.
> > > > > > > >>>
> > > > > > > >>> JR2. Share consumer (KIP-932) states: "If the number of
> > > > partitions is
> > > > > > > >>> increased for a topic with a subscription in a share group,
> > the
> > > > SPSO
> > > > > > > for
> > > > > > > >>> the newly created share-partitions is initialized to 0 ".
> > > > However, this
> > > > > > > >>> part has not been implemented yet. Currently, all partitions
> > > > without
> > > > > > > offset
> > > > > > > >>> follows the strategy given by share.auto.offset.reset. So, we
> > > > have an
> > > > > > > >>> opportunity to see if it's better to align this KIP's changes
> > > > with the
> > > > > > > >>> share consumer.
> > > > > > > >>> JR2.1 The first part is whether customizing the offset for
> > new
> > > > > > > partitions
> > > > > > > >>> to something other than 0 is useful in the share consumer. It
> > > > seems
> > > > > > > that
> > > > > > > >>> the motivation in this KIP applies to share consumer too.
> > > > > > > >>> JR2.2 The second part is how new partitions are detected. The
> > > > mechanism
> > > > > > > >>> introduced in this KIP seems quite effective. A group
> > > > coordinator could
> > > > > > > >>> pick up newly created partitions for an existing topic or
> > newly
> > > > created
> > > > > > > >>> topics matching the regex. Those partitions are categorized
> > as
> > > > new
> > > > > > > >>> partitions and will follow the strategy given by
> > > > > > > >>> auto.offset.reset.new.partitions.
> > > > > > > >>> This makes sense since those partitions typically don't have
> > a
> > > > large
> > > > > > > >>> backlog. A consumer could also additionally subscribe to an
> > > > existing
> > > > > > > topic
> > > > > > > >>> in an existing group. Those partitions will be categorized as
> > > > existing
> > > > > > > >>> partitions. This also makes sense since those partitions
> > could
> > > > have a
> > > > > > > large
> > > > > > > >>> backlog and it is better to follow the strategy given by
> > > > > > > auto.offset.reset. It
> > > > > > > >>> will be useful to think through whether the logic for
> > > > categorizing new
> > > > > > > >>> partitions should also be reused in the share consumer.
> > > > > > > >>>
> > > > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > > > client.
> > > > > > > I am
> > > > > > > >>> not sure if there are other use cases for those timestamps.
> > An
> > > > > > > alternative
> > > > > > > >>> is to avoid exposing those timestamps to the client. The
> > group
> > > > > > > coordinator
> > > > > > > >>> will categorize the partitions using those timestamps and
> > > > include the
> > > > > > > >>> categorization in the assigned partitions in
> > > > GroupHeartbeatResponse.
> > > > > > > >>>
> > > > > > > >>> JR4. It would be useful to support this feature on existing
> > > > groups as
> > > > > > > well.
> > > > > > > >>> For example, any partition without a creation timestamp can
> > > > always be
> > > > > > > >>> categorized as an existing partition.
> > > > > > > >>>
> > > > > > > >>> Jun
> > > > > > > >>>
> > > > > > > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > > > > > > >>>
> > > > > > > >>>> Hello Andrew,
> > > > > > > >>>>
> > > > > > > >>>> Thanks for your feedback,
> > > > > > > >>>>>> I've just been re-reading this KIP to ensure that share
> > group
> > > > > > > support is
> > > > > > > >>>>>> not needed. I've discovered that we made a mistake
> > > > implementing
> > > > > > > KIP-932
> > > > > > > >>>> in
> > > > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > > > really tight
> > > > > > > >>>> enough
> > > > > > > >>>>>> not to require KIP-1327.
> > > > > > > >>>>
> > > > > > > >>>> Could you walk us through the specific area where the
> > mistake
> > > > was
> > > > > > > found?
> > > > > > > >>>> That would
> > > > > > > >>>> help us assess whether it naturally fits within the scope of
> > > > KIP-1327
> > > > > > > or
> > > > > > > >>>> should be addressed
> > > > > > > >>>> separately.
> > > > > > > >>>>
> > > > > > > >>>> AS1 I have updated the KIP
> > > > > > > >>>>
> > > > > > > >>>> Best Regards,
> > > > > > > >>>> Jiunn-Yang
> > > > > > > >>>>
> > > > > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49
> > 寫道:
> > > > > > > >>>>>
> > > > > > > >>>>> hi Andrew
> > > > > > > >>>>>
> > > > > > > >>>>>> I've discovered that we made a mistake implementing
> > KIP-932
> > > > in this
> > > > > > > >>>> area,
> > > > > > > >>>>> so we'll rectify that and check whether it's really tight
> > > > enough not
> > > > > > > to
> > > > > > > >>>>> require KIP-1327.
> > > > > > > >>>>>
> > > > > > > >>>>> Would you mind sharing more details about that? If it's not
> > > > anything
> > > > > > > >>>> close
> > > > > > > >>>>> to NP-hard, we could definitely look into covering it
> > within
> > > > KIP-1327
> > > > > > > >>>>>
> > > > > > > >>>>> Best,
> > > > > > > >>>>> Chia-Ping
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三
> > > > 下午7:49寫道:
> > > > > > > >>>>>
> > > > > > > >>>>>> Hi Jiunn-Yang,
> > > > > > > >>>>>> I've just been re-reading this KIP to ensure that share
> > group
> > > > > > > support is
> > > > > > > >>>>>> not needed. I've discovered that we made a mistake
> > > > implementing
> > > > > > > KIP-932
> > > > > > > >>>> in
> > > > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > > > really tight
> > > > > > > >>>> enough
> > > > > > > >>>>>> not to require KIP-1327.
> > > > > > > >>>>>>
> > > > > > > >>>>>> One trivial comments for consistency.
> > > > > > > >>>>>>
> > > > > > > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > > > > > > represented by
> > > > > > > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > > > > > > >>>>>>
> > > > > > > >>>>>> Thanks,
> > > > > > > >>>>>> Andrew
> > > > > > > >>>>>>
> > > > > > > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > > > > > > >>>>>>> Hi all,
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Manually bumping this thread.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Best Regards,
> > > > > > > >>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Hello chia,
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Best Regards,
> > > > > > > >>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日
> > 凌晨12:47
> > > > 寫道:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> hi Jiunn-Yang
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> When the config is set on a cluster that has not yet
> > been
> > > > > > > >>>>>> upgraded... classification cannot occur... the consumer
> > falls
> > > > back
> > > > > > > to
> > > > > > > >>>> the
> > > > > > > >>>>>> base auto.offset.reset for the affected partitions. No
> > > > exception is
> > > > > > > >>>> thrown,
> > > > > > > >>>>>> and no operational disruption results.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Existing group can't take advantage of this excellent
> > new
> > > > > > > >>>>>> configuration. Allowing users to modify the group creation
> > > > time
> > > > > > > might be
> > > > > > > >>>>>> overkill. Instead, we could print a useful warning
> > message to
> > > > guide
> > > > > > > >>>> users.
> > > > > > > >>>>>> For example, we can suggest that they re-create the group
> > > > with their
> > > > > > > >>>>>> existing committed offsets
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> Protocol changes
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Would you mind listing those RPC changes in a table
> > format?
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> The full interaction matrix between the base policy
> > and
> > > > the
> > > > > > > >>>>>> new-partition policy is:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Please add a filed to describe the target scenario when
> > > > using
> > > > > > > these
> > > > > > > >>>>>> policies
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Best,
> > > > > > > >>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > > > > > > >>>>>>>>>> Hello Jun, chia,
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for
> > the
> > > > new
> > > > > > > >>>>>>>>>> approach, PTAL
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日
> > 上午8:23
> > > > 寫道:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> hi Jun
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Yes, your approach is great. I think the combination
> > of
> > > > latest
> > > > > > > (for
> > > > > > > >>>>>> existing partitions) and by_duration (for new partitions)
> > can
> > > > > > > address
> > > > > > > >>>> 99%
> > > > > > > >>>>>> of the complaints I have heard regarding this issue.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Also, leveraging the group creation time here opens
> > the
> > > > door to
> > > > > > > >>>>>> implementing a new policy based on timestamp seek in the
> > > > future,
> > > > > > > should
> > > > > > > >>>> the
> > > > > > > >>>>>> community want to pursue that.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Thanks for your patience and constructive feedback.
> > We
> > > > will
> > > > > > > update
> > > > > > > >>>>>> the KIP accordingly.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> Best, Chia-Ping
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日
> > > > 清晨5:11
> > > > > > > 寫道:
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Thanks for the reply.
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > > > > > > configure a
> > > > > > > >>>>>> different
> > > > > > > >>>>>>>>>>>> offset policy for existing partitions vs new
> > partitions.
> > > > > > > However,
> > > > > > > >>>>>> using
> > > > > > > >>>>>>>>>>>> group creation time to capture that seems more
> > > > intuitive.
> > > > > > > Here is
> > > > > > > >>>>>> another
> > > > > > > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and
> > > > categorize
> > > > > > > new
> > > > > > > >>>>>> partitions
> > > > > > > >>>>>>>>>>>> based on group creation time. Introduce
> > > > > > > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose
> > > > values
> > > > > > > can be
> > > > > > > >>>>>> earliest,
> > > > > > > >>>>>>>>>>>> latest and by_duration, the same as
> > auto.offset.reset.
> > > > Users
> > > > > > > can
> > > > > > > >>>>>> set
> > > > > > > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if
> > > > they
> > > > > > > want to
> > > > > > > >>>>>> guarantee
> > > > > > > >>>>>>>>>>>> no data loss on new partitions. They can also use
> > > > by_duration
> > > > > > > to
> > > > > > > >>>>>> set an
> > > > > > > >>>>>>>>>>>> upper bound on the backlog replayed, which can be
> > > > different
> > > > > > > from
> > > > > > > >>>>>> that of
> > > > > > > >>>>>>>>>>>> the existing partitions. This will address your
> > concern
> > > > about
> > > > > > > too
> > > > > > > >>>>>> much
> > > > > > > >>>>>>>>>>>> backlog being replayed when the offsets are lost.
> > What
> > > > do you
> > > > > > > >>>>>> think?
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > > > > > > >>>>>> chia7712@gmail.com> wrote:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> hi Jun
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> The most important part of this story is how users
> > > > should
> > > > > > > expect
> > > > > > > >>>>>> the data
> > > > > > > >>>>>>>>>>>>> they can see when using the latest or by_duration
> > > > policy with
> > > > > > > >>>>>> expanded
> > > > > > > >>>>>>>>>>>>> partitions.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss,
> > > > but it is
> > > > > > > >>>>>>>>>>>>> non-deterministic, which means users will either
> > read
> > > > too
> > > > > > > many
> > > > > > > >>>>>> historical
> > > > > > > >>>>>>>>>>>>> records from existing partitions or lose some
> > records
> > > > from
> > > > > > > >>>>>> expanded
> > > > > > > >>>>>>>>>>>>> partitions.
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > > > > > > >>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>>>>>>> is a bit hard to understand, and that is why I
> > > > preferred
> > > > > > > having a
> > > > > > > >>>>>> whole new
> > > > > > > >>>>>>>>>>>>> policy based entirely on group creation time
> > (KIP-1282)
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > 2026年6月16日週二
> > > > > > > 上午1:08寫道:
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to
> > understand
> > > > the
> > > > > > > value
> > > > > > > >>>>>> of the new
> > > > > > > >>>>>>>>>>>>>> configs with the KIP.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't
> > want
> > > > to
> > > > > > > miss
> > > > > > > >>>>>> the data if
> > > > > > > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing
> > > > partition
> > > > > > > is
> > > > > > > >>>>>> easy to
> > > > > > > >>>>>>>>>>>>>> understand because it relates to retention time.
> > The
> > > > > > > backlog for
> > > > > > > >>>>>> the new
> > > > > > > >>>>>>>>>>>>>> partition is a bit subtle to understand since it
> > > > depends on
> > > > > > > the
> > > > > > > >>>>>> metadata
> > > > > > > >>>>>>>>>>>>>> refresh delay. To set
> > auto.offset.reset.max.age.ms
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > >>>>>>> ,
> > > > > > > >>>>>>>>>>>>>> the user needs to
> > > > > > > >>>>>>>>>>>>>> understand the metadata refresh delay on the
> > consumer
> > > > side
> > > > > > > and
> > > > > > > >>>>>> use it to
> > > > > > > >>>>>>>>>>>>>> set the config.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the
> > same
> > > > value
> > > > > > > for
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three
> > > > issues with
> > > > > > > >>>> this
> > > > > > > >>>>>>>>>>>>>> approach.
> > > > > > > >>>>>>>>>>>>>> 1. It computes the seek target client-side as
> > now() -
> > > > > > > duration,
> > > > > > > >>>>>> which
> > > > > > > >>>>>>>>>>>>>> introduces clock skew across consumers and forces
> > > > operators
> > > > > > > to
> > > > > > > >>>>>> choose
> > > > > > > >>>>>>>>>>>>>> overly large durations, causing unnecessary
> > > > reprocessing.
> > > > > > > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each
> > retry,
> > > > so
> > > > > > > failed
> > > > > > > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target
> > > > forward and
> > > > > > > >>>>>> potentially
> > > > > > > >>>>>>>>>>>>>> miss records produced between attempts.
> > > > > > > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without
> > > > committed
> > > > > > > >>>>>> offsets, and
> > > > > > > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > > > > > > long-existing
> > > > > > > >>>>>> partitions
> > > > > > > >>>>>>>>>>>>>> newly assigned to the group, leading to
> > unnecessary
> > > > replay.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated
> > by
> > > > adding
> > > > > > > a bit
> > > > > > > >>>>>> buffer to
> > > > > > > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > > > > > > improving the
> > > > > > > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh
> > > > delay is
> > > > > > > >>>>>> typically low
> > > > > > > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer
> > > > and tens
> > > > > > > of
> > > > > > > >>>>>> seconds
> > > > > > > >>>>>>>>>>>>>> with
> > > > > > > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading
> > that
> > > > much
> > > > > > > >>>>>> backlog for new
> > > > > > > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the
> > same
> > > > for
> > > > > > > existing
> > > > > > > >>>>>>>>>>>>>> partitions.
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> So, instead of introducing a new config, could we
> > just
> > > > > > > reuse the
> > > > > > > >>>>>> existing
> > > > > > > >>>>>>>>>>>>>> config with better documentation and/or
> > > > implementation?
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <
> > > > s7133700@gmail.com>
> > > > > > > >>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Hello Jun,
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> You're right that group creation time is the more
> > > > intuitive
> > > > > > > >>>>>> answer at
> > > > > > > >>>>>>>>>>>>>>> first glance,
> > > > > > > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions
> > that
> > > > > > > "predate
> > > > > > > >>>>>> the group"
> > > > > > > >>>>>>>>>>>>>>> vs partitions
> > > > > > > >>>>>>>>>>>>>>> "created during group runtime," which directly
> > > > points to a
> > > > > > > >>>>>>>>>>>>>> group-lifecycle
> > > > > > > >>>>>>>>>>>>>>> classifier.
> > > > > > > >>>>>>>>>>>>>>> I'd like to walk through why we landed on
> > partition
> > > > age,
> > > > > > > and
> > > > > > > >>>> the
> > > > > > > >>>>>>>>>>>>>>> trade-offs we considered.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> This covers the metadata blindness window, but
> > has
> > > > issues
> > > > > > > the
> > > > > > > >>>>>> KIP
> > > > > > > >>>>>>>>>>>>>>> currently documents
> > > > > > > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock
> > > > skew
> > > > > > > across
> > > > > > > >>>>>> consumers.
> > > > > > > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > > > > > potentially
> > > > > > > >>>>>> missing
> > > > > > > >>>>>>>>>>>>>>> records produced between
> > > > > > > >>>>>>>>>>>>>>> attempts.
> > > > > > > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without
> > > > committed
> > > > > > > >>>>>> offsets,
> > > > > > > >>>>>>>>>>>>>>> including pre-existing partitions
> > > > > > > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary
> > > > replay.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively
> > > > running.
> > > > > > > Our
> > > > > > > >>>>>> concern
> > > > > > > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> T=0:         Group created.
> > > > > > > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected,
> > > > etc.).
> > > > > > > >>>>>>>>>>>>>>> T=50:        Partition added during the idle
> > window.
> > > > > > > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Under group creation time, the new partition is
> > > > classified
> > > > > > > as
> > > > > > > >>>>>> new
> > > > > > > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying
> > > > everything
> > > > > > > from
> > > > > > > >>>>>> T=50.
> > > > > > > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also
> > > > accumulated
> > > > > > > >>>>>> data that
> > > > > > > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely
> > the
> > > > > > > contract
> > > > > > > >>>> of
> > > > > > > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no
> > principled
> > > > reason
> > > > > > > to
> > > > > > > >>>>>> treat
> > > > > > > >>>>>>>>>>>>>>> the new partition differently; both contain
> > backlog
> > > > > > > accumulated
> > > > > > > >>>>>> during
> > > > > > > >>>>>>>>>>>>>>> the same idle window.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog”
> > principle
> > > > you
> > > > > > > raised
> > > > > > > >>>>>> in
> > > > > > > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has
> > tolerated
> > > > some
> > > > > > > backlog
> > > > > > > >>>>>> on
> > > > > > > >>>>>>>>>>>>>>> every other partition during the same idle
> > period;
> > > > forcing
> > > > > > > >>>>>> 0-backlog
> > > > > > > >>>>>>>>>>>>>>> tolerance only on new partitions would be
> > > > inconsistent with
> > > > > > > >>>> that
> > > > > > > >>>>>>>>>>>>>>> tolerance.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent
> > data
> > > > loss
> > > > > > > >>>> window,
> > > > > > > >>>>>>>>>>>>>>> the gap between partition creation and the
> > consumer’s
> > > > > > > metadata
> > > > > > > >>>>>>>>>>>>>>> refresh. Within this window, data loss is
> > genuinely
> > > > > > > silent: the
> > > > > > > >>>>>>>>>>>>>>> consumer had no opportunity to know about the
> > > > partition.
> > > > > > > >>>>>> Outside this
> > > > > > > >>>>>>>>>>>>>>> window, missing data reflects either:
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with
> > idle
> > > > > > > consumers,
> > > > > > > >>>>>> or
> > > > > > > >>>>>>>>>>>>>>> - (b) an operational issue to surface via
> > > > monitoring, not
> > > > > > > via
> > > > > > > >>>>>> reset
> > > > > > > >>>>>>>>>>>>>> policy.
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> We did not choose partition age because it is
> > more
> > > > elegant
> > > > > > > than
> > > > > > > >>>>>> group
> > > > > > > >>>>>>>>>>>>>>> creation time — we chose it because its failure
> > mode
> > > > > > > (requires
> > > > > > > >>>> a
> > > > > > > >>>>>>>>>>>>>>> threshold) is
> > > > > > > >>>>>>>>>>>>>>> less invasive than the failure mode of group
> > > > creation time
> > > > > > > >>>>>> (overrides
> > > > > > > >>>>>>>>>>>>>>> user-stated
> > > > > > > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > 2026年6月13日
> > > > 上午11:52
> > > > > > > 寫道:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Hi Jun,
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Relying on both creation times will create an
> > > > inconsistent
> > > > > > > >>>>>> scenario. A
> > > > > > > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long
> > sleep
> > > > will
> > > > > > > seek
> > > > > > > >>>>>> to the
> > > > > > > >>>>>>>>>>>>>>>> beginning for the partitions created later than
> > the
> > > > group.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to
> > fix
> > > > the
> > > > > > > >>>>>> inconsistency
> > > > > > > >>>>>>>>>>>>>>> using a
> > > > > > > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach
> > a
> > > > > > > consensus,
> > > > > > > >>>>>> KIP-1327
> > > > > > > >>>>>>>>>>>>>>> goes
> > > > > > > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent
> > > > users
> > > > > > > from
> > > > > > > >>>>>> falling
> > > > > > > >>>>>>>>>>>>>> into
> > > > > > > >>>>>>>>>>>>>>>> that pitfall.
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > > 2026年6月13日週六
> > > > > > > >>>>>> 上午6:49寫道:
> > > > > > > >>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late
> > reply.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> JR1. The design of
> > auto.offset.reset.max.age.ms
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>>>>>>>> still feels weird to
> > > > > > > >>>>>>>>>>>>>>> me.
> > > > > > > >>>>>>>>>>>>>>>>> It
> > > > > > > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing
> > based on
> > > > the
> > > > > > > >>>>>> partition
> > > > > > > >>>>>>>>>>>>>>> creation
> > > > > > > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be
> > > > based on
> > > > > > > the
> > > > > > > >>>>>> group
> > > > > > > >>>>>>>>>>>>>>> creation
> > > > > > > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is
> > > > created
> > > > > > > are
> > > > > > > >>>>>> existing
> > > > > > > >>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>> all partitions created after the group creation
> > > > are new
> > > > > > > >>>>>> partitions.
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <
> > > > s7133700@gmail.com>
> > > > > > > >>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no
> > > > further
> > > > > > > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16
> > 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common
> > > > operational
> > > > > > > task,
> > > > > > > >>>>>> not an
> > > > > > > >>>>>>>>>>>>>> edge
> > > > > > > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section
> > > > accordingly.
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日
> > > > 下午5:49 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be
> > useful
> > > > to
> > > > > > > clarify
> > > > > > > >>>>>> that
> > > > > > > >>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>> is a
> > > > > > > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case,
> > which
> > > > > > > further
> > > > > > > >>>>>>>>>>>>>> demonstrates
> > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > > > > > > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common
> > operational
> > > > task in
> > > > > > > >>>>>> Kafka: To
> > > > > > > >>>>>>>>>>>>>>>>> balance
> > > > > > > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are
> > > > typically
> > > > > > > >>>>>> created with a
> > > > > > > >>>>>>>>>>>>>>>>>> moderate
> > > > > > > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic
> > > > grows
> > > > > > > over
> > > > > > > >>>>>> time, it
> > > > > > > >>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>> often
> > > > > > > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of
> > partitions
> > > > to
> > > > > > > >>>>>> accommodate the
> > > > > > > >>>>>>>>>>>>>>>>> higher
> > > > > > > >>>>>>>>>>>>>>>>>>>> workload.
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> Regards
> > > > > > > >>>>>>>>>>>>>>>>>>>> Jian
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六
> > 22:31写道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the
> > > > KIP!
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > > 2026年5月30日
> > > > > > > 晚上8:29
> > > > > > > >>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot"
> > and
> > > > "cold"
> > > > > > > when
> > > > > > > >>>>>>>>>>>>>> describing
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you
> > are
> > > > using
> > > > > > > them
> > > > > > > >>>> to
> > > > > > > >>>>>>>>>>>>>> describe
> > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the
> > > > records, but
> > > > > > > >>>>>> applying
> > > > > > > >>>>>>>>>>>>>> these
> > > > > > > >>>>>>>>>>>>>>>>>> terms
> > > > > > > >>>>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit
> > unnatural.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't
> > > > really care
> > > > > > > >>>>>> whether a
> > > > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only
> > expectation
> > > > is
> > > > > > > that
> > > > > > > >>>>>> they won't
> > > > > > > >>>>>>>>>>>>>>>>>> silently
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the
> > topic
> > > > during
> > > > > > > their
> > > > > > > >>>>>> active
> > > > > > > >>>>>>>>>>>>>>>>>>>>> consumption.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六
> > > > 下午12:30寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated
> > the
> > > > KIP
> > > > > > > >>>>>> motivation
> > > > > > > >>>>>>>>>>>>>>> section.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org>
> > 於
> > > > > > > 2026年5月30日
> > > > > > > >>>>>> 凌晨1:12
> > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a
> > > > stronger
> > > > > > > >>>>>> motivation
> > > > > > > >>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that
> > not
> > > > all
> > > > > > > >>>>>> partitions
> > > > > > > >>>>>>>>>>>>>> without
> > > > > > > >>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly
> > > > expanded
> > > > > > > >>>>>> partition
> > > > > > > >>>>>>>>>>>>>> (hot)
> > > > > > > >>>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a
> > partition the
> > > > > > > consumer
> > > > > > > >>>>>> has
> > > > > > > >>>>>>>>>>>>>> never
> > > > > > > >>>>>>>>>>>>>>>>> seen
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)."
> > Why
> > > > is the
> > > > > > > hot
> > > > > > > >>>>>>>>>>>>>> partition
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration
> > > > policy is
> > > > > > > also
> > > > > > > >>>>>>>>>>>>>>> insufficient
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> because:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() -
> > > > duration)
> > > > > > > varies
> > > > > > > >>>>>> across
> > > > > > > >>>>>>>>>>>>>> nodes
> > > > > > > >>>>>>>>>>>>>>>>>> due
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set
> > an
> > > > overly
> > > > > > > large
> > > > > > > >>>>>>>>>>>>>> duration,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> causing
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client
> > > > recalculates the
> > > > > > > seek
> > > > > > > >>>>>> time on
> > > > > > > >>>>>>>>>>>>>>>>> retry,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward
> > and
> > > > risking
> > > > > > > data
> > > > > > > >>>>>> loss."
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are
> > rare.
> > > > If
> > > > > > > these
> > > > > > > >>>>>> issues
> > > > > > > >>>>>>>>>>>>>>>>> persist,
> > > > > > > >>>>>>>>>>>>>>>>>>>>> more
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere.
> > Rare
> > > > > > > >>>>>> situations don't
> > > > > > > >>>>>>>>>>>>>>>>> need a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> common
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > > > > > situations,
> > > > > > > >>>>>> they can
> > > > > > > >>>>>>>>>>>>>>>>>> implement
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > ConsumerRebalanceListener.onPartitionsAssigned().
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > > > > > >>>>>> s7133700@gmail.com>
> > > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the
> > returned
> > > > value
> > > > > > > >>>>>> should
> > > > > > > >>>>>>>>>>>>>> always
> > > > > > > >>>>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the
> > > > KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > > > positive
> > > > > > > >>>>>> any
> > > > > > > >>>>>>>>>>>>>>>>>> field
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can
> > detect
> > > > the
> > > > > > > version
> > > > > > > >>>>>>>>>>>>>> mismatch
> > > > > > > >>>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>>>> during
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment,
> > > > which
> > > > > > > occurs
> > > > > > > >>>>>> inside
> > > > > > > >>>>>>>>>>>>>>>>> poll().
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > > > > > UnsupportedVersionException
> > > > > > > >>>>>> from
> > > > > > > >>>>>>>>>>>>>> poll().
> > > > > > > >>>>>>>>>>>>>>>>>> I’ll
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > 於
> > > > > > > 2026年5月17日
> > > > > > > >>>>>> 下午4:50
> > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1):
> > The
> > > > age of
> > > > > > > this
> > > > > > > >>>>>>>>>>>>>> partition
> > > > > > > >>>>>>>>>>>>>>>>> in
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by
> > the
> > > > broker
> > > > > > > as
> > > > > > > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> -
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if
> > the
> > > > broker
> > > > > > > >>>>>> does not
> > > > > > > >>>>>>>>>>>>>>>>> support
> > > > > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time
> > is
> > > > > > > unknown.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the
> > returned
> > > > value
> > > > > > > >>>>>> should
> > > > > > > >>>>>>>>>>>>>> always
> > > > > > > >>>>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > > > positive
> > > > > > > >>>>>> any
> > > > > > > >>>>>>>>>>>>>>>>>> field
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter
> > > > UnsupportedVersionException
> > > > > > > when
> > > > > > > >>>>>> calling
> > > > > > > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design
> > > > change
> > > > > > > based on
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>> discussion
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the
> > > > new-partition
> > > > > > > >>>> reset
> > > > > > > >>>>>>>>>>>>>> behavior
> > > > > > > >>>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>>>>>>>> now applies to all
> > > > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration,
> > none).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the
> > > > consumer
> > > > > > > resets
> > > > > > > >>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>>>>>>>> .partitions
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions,
> > the
> > > > base
> > > > > > > >>>>>>>>>>>>>> auto.offset.reset
> > > > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > > > > > represented
> > > > > > > >>>>>> by a
> > > > > > > >>>>>>>>>>>>>>> separate
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>> .partitions,
> > > > > > > >>>>>>>>>>>>>> currently fixed to
> > > > > > > >>>>>>>>>>>>>>>>> earliest).
> > > > > > > >>>>>>>>>>>>>>>>>>>>> This
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the
> > > > behavior to a
> > > > > > > >>>>>> public
> > > > > > > >>>>>>>>>>>>>>>>>> user-facing
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > 於
> > > > > > > 2026年5月16日
> > > > > > > >>>>>> 清晨7:46
> > > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The
> > proposal
> > > > from me
> > > > > > > is
> > > > > > > >>>>>> listed
> > > > > > > >>>>>>>>>>>>>>> below:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>> .partitions
> > > > > > > >>>>>>>>>>>>>> with a default value
> > > > > > > >>>>>>>>>>>>>>>>> of
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from
> > both
> > > > > > > >>>>>> by_duration and
> > > > > > > >>>>>>>>>>>>>>>>> latest,
> > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > > > > > >>>>>> auto.offset.reset=earliest.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>> .partitions
> > > > > > > >>>>>>>>>>>>>> as an internal
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>>>>>>>> .partitions=earliest
> > > > > > > >>>>>>>>>>>>>>> already
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss
> > > > the use
> > > > > > > cases
> > > > > > > >>>>>> of
> > > > > > > >>>>>>>>>>>>>> other
> > > > > > > >>>>>>>>>>>>>>>>>> values
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> in a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs,
> > auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>> .partitions
> > > > > > > >>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > > > > > > >>>>>>> ,
> > > > > > > >>>>>>>>>>>>>> will be applied to all for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via
> > dev
> > > > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the
> > KIP
> > > > says
> > > > > > > "When
> > > > > > > >>>>>> a Kafka
> > > > > > > >>>>>>>>>>>>>>>>> topic
> > > > > > > >>>>>>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers
> > using
> > > > the
> > > > > > > latest
> > > > > > > >>>>>> auto
> > > > > > > >>>>>>>>>>>>>> offset
> > > > > > > >>>>>>>>>>>>>>>>>> reset
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records
> > > > produced to
> > > > > > > those
> > > > > > > >>>>>>>>>>>>>> partitions
> > > > > > > >>>>>>>>>>>>>>>>>> before
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a
> > user
> > > > sets
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec,
> > > > the same
> > > > > > > >>>>>> record loss
> > > > > > > >>>>>>>>>>>>>>>>> issue
> > > > > > > >>>>>>>>>>>>>>>>>>>>> could
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking
> > > > auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>> .partitions
> > > > > > > >>>>>>>>>>>>>> will
> > > > > > > >>>>>>>>>>>>>>> take
> > > > > > > >>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> same
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a
> > user
> > > > could
> > > > > > > set
> > > > > > > >>>>>> it
> > > > > > > >>>>>>>>>>>>>>>>>> by_duration if
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM
> > > > Chia-Ping
> > > > > > > Tsai <
> > > > > > > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might
> > be
> > > > missing
> > > > > > > >>>>>> something
> > > > > > > >>>>>>>>>>>>>>>>>> important
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with
> > me as
> > > > I try
> > > > > > > to
> > > > > > > >>>>>> clarify
> > > > > > > >>>>>>>>>>>>>> with
> > > > > > > >>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>> few
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for
> > > > extending
> > > > > > > this
> > > > > > > >>>>>> logic
> > > > > > > >>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>> other
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> reset
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies
> > like
> > > > > > > earliest
> > > > > > > >>>>>> or
> > > > > > > >>>>>>>>>>>>>>>>> by_duration
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> don't
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent
> > data
> > > > loss
> > > > > > > issue
> > > > > > > >>>>>> when a
> > > > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect
> > users
> > > > to
> > > > > > > >>>>>> configure for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>> .partitions?
> > > > > > > >>>>>>>>>>>>>> If they set it to
> > > > > > > >>>>>>>>>>>>>>>>> earliest
> > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same
> > edge
> > > > > > > cases. For
> > > > > > > >>>>>>>>>>>>>> example,
> > > > > > > >>>>>>>>>>>>>>>>> if a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new
> > > > partition is
> > > > > > > >>>>>> created
> > > > > > > >>>>>>>>>>>>>> during
> > > > > > > >>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to
> > skip
> > > > to
> > > > > > > latest
> > > > > > > >>>>>> when
> > > > > > > >>>>>>>>>>>>>>>>> resuming,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> rather
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just
> > because the
> > > > > > > partition
> > > > > > > >>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>> technically
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > > > > > > introducing a
> > > > > > > >>>>>> max.age
> > > > > > > >>>>>>>>>>>>>>>>>> threshold.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> It
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define
> > what
> > > > is
> > > > > > > >>>>>> genuinely
> > > > > > > >>>>>>>>>>>>>>> "hot/new"
> > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't
> > > > seen yet.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao
> > via dev
> > > > > > > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find
> > > > auto.offset.reset.latest.max.age a
> > > > > > > bit
> > > > > > > >>>>>> weird. It
> > > > > > > >>>>>>>>>>>>>>>>> only
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest.
> > > > However, it
> > > > > > > seems
> > > > > > > >>>>>> that the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> motivation
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is
> > > > set to
> > > > > > > other
> > > > > > > >>>>>> values
> > > > > > > >>>>>>>>>>>>>> like
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to
> > have a
> > > > > > > separate
> > > > > > > >>>>>> way to
> > > > > > > >>>>>>>>>>>>>>> control
> > > > > > > >>>>>>>>>>>>>>>>>>>>> newly
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions
> > > > when the
> > > > > > > >>>> group
> > > > > > > >>>>>>>>>>>>>> starts.
> > > > > > > >>>>>>>>>>>>>>>>>> Have we
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > > > > > auto.offset.reset.new
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>>>>>>>>> .partitions?
> > > > > > > >>>>>>>>>>>>>>>>>> If
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset
> > reset
> > > > policy
> > > > > > > >>>>>> defaults to
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user
> > could
> > > > set it
> > > > > > > >>>>>> explicitly
> > > > > > > >>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>> customize
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM
> > 黃竣陽 <
> > > > > > > >>>>>> s7133700@gmail.com
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this
> > > > thread.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於
> > > > 2026年5月1日
> > > > > > > 晚上10:37
> > > > > > > >>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from
> > v13 to
> > > > v14.
> > > > > > > The
> > > > > > > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64,
> > > > default -1),
> > > > > > > >>>>>> computed
> > > > > > > >>>>>>>>>>>>>>>>>> server-side
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> by
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > > > > > >>>> partition_creation_time.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat
> > > > flow.
> > > > > > > when
> > > > > > > >>>>>>>>>>>>>>>>>> MembershipManager
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly
> > > > invalidates the
> > > > > > > >>>>>> metadata for
> > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> affected
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh
> > MetadataRequest
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset
> > > > decision,
> > > > > > > even
> > > > > > > >>>>>> if the
> > > > > > > >>>>>>>>>>>>>>>>> topic
> > > > > > > >>>>>>>>>>>>>>>>>> ID
> > > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the
> > broker's
> > > > maximum
> > > > > > > >>>>>> supported
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at
> > > > connection
> > > > > > > time.
> > > > > > > >>>>>> If the
> > > > > > > >>>>>>>>>>>>>>>>>> negotiated
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not
> > support
> > > > > > > >>>>>> PartitionAgeMs at
> > > > > > > >>>>>>>>>>>>>> all
> > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>>>>>> can
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than
> > silently
> > > > falling
> > > > > > > >>>>>> back to
> > > > > > > >>>>>>>>>>>>>> latest
> > > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any
> > > > operator-visible
> > > > > > > signal.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes
> > in
> > > > the
> > > > > > > KIP.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > > > chia7712@gmail.com> 於
> > > > > > > >>>>>> 2026年4月29日
> > > > > > > >>>>>>>>>>>>>>> 下午4:04
> > > > > > > >>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of
> > > > moving the
> > > > > > > >>>>>> 'age'
> > > > > > > >>>>>>>>>>>>>>>>> resolution
> > > > > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata
> > API
> > > > to keep
> > > > > > > the
> > > > > > > >>>>>> control
> > > > > > > >>>>>>>>>>>>>>>>> plane
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before,
> > is
> > > > > > > introducing
> > > > > > > >>>>>>>>>>>>>>> inter-broker
> > > > > > > >>>>>>>>>>>>>>>>>>>>> clock
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach
> > provided
> > > > a
> > > > > > > single
> > > > > > > >>>>>> source of
> > > > > > > >>>>>>>>>>>>>>>>> truth
> > > > > > > >>>>>>>>>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this
> > time
> > > > skew
> > > > > > > >>>>>> should be
> > > > > > > >>>>>>>>>>>>>>>>>> negligible.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will
> > likely be
> > > > > > > >>>>>> configured in
> > > > > > > >>>>>>>>>>>>>>> minutes
> > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in
> > milliseconds)
> > > > between
> > > > > > > >>>>>> brokers
> > > > > > > >>>>>>>>>>>>>> won't
> > > > > > > >>>>>>>>>>>>>>>>>> impact
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > > > > > dev@kafka.apache.org>
> > > > > > > >>>>>> 於
> > > > > > > >>>>>>>>>>>>>>>>> 2026年4月29日
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really
> > followed
> > > > the
> > > > > > > >>>>>> previous
> > > > > > > >>>>>>>>>>>>>>>>>> conversation
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> but I
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly
> > understand
> > > > the
> > > > > > > flow
> > > > > > > >>>>>> with the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP.
> > There
> > > > is a
> > > > > > > new
> > > > > > > >>>>>> boolean;
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>> KIP
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned
> > only
> > > > when
> > > > > > > this
> > > > > > > >>>>>> boolean
> > > > > > > >>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>> set.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer
> > > > receives
> > > > > > > a new
> > > > > > > >>>>>>>>>>>>>> partition,
> > > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>>>>>>>>> will
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean
> > set
> > > > to
> > > > > > > receive
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>>>>>>>> ages.
> > > > > > > >>>>>>>>>>>>>>> Is
> > > > > > > >>>>>>>>>>>>>>>>>> my
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps
> > > > clarify the
> > > > > > > flow
> > > > > > > >>>>>> and
> > > > > > > >>>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>>>>> explain
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g.
> > list
> > > > > > > offsets,
> > > > > > > >>>>>> fetch
> > > > > > > >>>>>>>>>>>>>>>>> offsets,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is
> > > > correct, I
> > > > > > > >>>>>> wonder if
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat
> > API
> > > > is the
> > > > > > > right
> > > > > > > >>>>>> place
> > > > > > > >>>>>>>>>>>>>> for
> > > > > > > >>>>>>>>>>>>>>>>>> this
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> given
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > > > > > Alternatively,
> > > > > > > >>>>>> it could
> > > > > > > >>>>>>>>>>>>>>>>> simply
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we
> > should be
> > > > > > > rather
> > > > > > > >>>>>> cautious
> > > > > > > >>>>>>>>>>>>>>> about
> > > > > > > >>>>>>>>>>>>>>>>>> not
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat
> > API
> > > > with
> > > > > > > >>>>>> unrelated
> > > > > > > >>>>>>>>>>>>>>> concepts.
> > > > > > > >>>>>>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> API
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for
> > assigning
> > > > or
> > > > > > > revoking
> > > > > > > >>>>>>>>>>>>>>> partitions.
> > > > > > > >>>>>>>>>>>>>>>>>> The
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> fact
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > > > > > corresponding
> > > > > > > >>>>>> Streams
> > > > > > > >>>>>>>>>>>>>> API
> > > > > > > >>>>>>>>>>>>>>>>>> also
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right.
> > > > What
> > > > > > > would we
> > > > > > > >>>>>> do if
> > > > > > > >>>>>>>>>>>>>> we
> > > > > > > >>>>>>>>>>>>>>>>>> want to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at
> > 12:28 AM
> > > > > > > >>>>>> Muralidhar Basani
> > > > > > > >>>>>>>>>>>>>>> via
> > > > > > > >>>>>>>>>>>>>>>>>> dev
> > > > > > > >>>>>>>>>>>>>>>>>>>>> <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org>
> > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great
> > kip.
> > > > Good to
> > > > > > > know
> > > > > > > >>>>>> about
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>> gap.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version
> > > > bump for
> > > > > > > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on
> > > > response,
> > > > > > > >>>>>> PartitionAges
> > > > > > > >>>>>>>>>>>>>> on
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version
> > > > bump?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config,
> > is
> > > > there a
> > > > > > > >>>>>> recommended
> > > > > > > >>>>>>>>>>>>>>>>> value
> > > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it
> > should
> > > > based
> > > > > > > on the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > > > > > >>>>>>>>>>>>>> <
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of
> > > > javadocs I
> > > > > > > >>>>>> guess.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are
> > no
> > > > > > > changes to
> > > > > > > >>>>>> Kafka
> > > > > > > >>>>>>>>>>>>>>>>> Streams,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> would
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > > > > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > > > > > >>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users
> > > > configure
> > > > > > > it?
> > > > > > > >>>>>> This is
> > > > > > > >>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>> most
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users
> > might
> > > > easily
> > > > > > > >>>>>> mistakenly
> > > > > > > >>>>>>>>>>>>>>>>>> configure
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing
> > "the
> > > > > > > consumer
> > > > > > > >>>>>> falls
> > > > > > > >>>>>>>>>>>>>> back
> > > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being
> > changed
> > > > > > > >>>>>> per-partition
> > > > > > > >>>>>>>>>>>>>> which
> > > > > > > >>>>>>>>>>>>>>>>>> isn't
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something
> > like
> > > > > > > "consumer
> > > > > > > >>>>>> resolves
> > > > > > > >>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> initial
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that
> > > > partition" as if
> > > > > > > >>>>>> earliest
> > > > > > > >>>>>>>>>>>>>> was
> > > > > > > >>>>>>>>>>>>>>>>>> applied
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and
> > > > auto.offset.reset
> > > > > > > >>>>>> config is
> > > > > > > >>>>>>>>>>>>>>>>>> unchanged.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at
> > > > 2:48 PM 黃竣陽 <
> > > > > > > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to
> > > > include
> > > > > > > this
> > > > > > > >>>>>> change.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > > > chia7712@apache.org>
> > > > > > > 於
> > > > > > > >>>>>>>>>>>>>> 2026年4月28日
> > > > > > > >>>>>>>>>>>>>>>>>> 晚上8:03
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose
> > the
> > > > > > > partition
> > > > > > > >>>>>> creation
> > > > > > > >>>>>>>>>>>>>>> time
> > > > > > > >>>>>>>>>>>>>>>>>> via
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be
> > valuable
> > > > for
> > > > > > > users
> > > > > > > >>>> to
> > > > > > > >>>>>>>>>>>>>> diagnose
> > > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > > > > > >>>>>> auto.offset.reset.latest.max.age
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽
> > > > wrote:
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> > > > discussion
> > > > > > > on
> > > > > > > >>>>>> KIP-1327
> > > > > > > >>>>>>>>>>>>>>>>>> Prevent
> > > > > > > >>>>>>>>>>>>>>>>>>>>> Hot
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for
> > Latest
> > > > Policy
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > >
> > > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
> > > > introduces
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets
> > the
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy
> > > > distinguish
> > > > > > > newly
> > > > > > > >>>>>> expanded
> > > > > > > >>>>>>>>>>>>>>>>> (hot)
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones.
> > > > Partitions
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the
> > configured
> > > > > > > threshold
> > > > > > > >>>>>>>>>>>>>>> automatically
> > > > > > > >>>>>>>>>>>>>>>>>> fall
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent
> > > > data loss
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion
> > > > without
> > > > > > > forcing
> > > > > > > >>>>>> a full
> > > > > > > >>>>>>>>>>>>>>>>>> historical
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Andrew,

There is another case where a share consumer subscribes to a regex and a
new topic is later created that matches that regex. Intuitively, they
should use the earliest offset for consumption. But how does the share
coordinator distinguish these from existing topics explicitly added by the
user without the creation timestamp?

Thanks,

Jun

On Fri, Jun 26, 2026 at 11:00 AM Andrew Schofield <sc...@apache.org>
wrote:

> Hi Jun,
> When a user adds an existing topic to a share group subscription, we do
> use the auto.offset.reset policy because those partitions already exist and
> may well have data on them. We only use 0 in the case where a topic already
> being consumed gained some additional new partitions. These partitions will
> be empty at the time of creation, so zero seems like a reasonable starting
> point. Partitions being consumed in a share group always have a known
> starting position once assigned. We don't have the equivalent of a consumer
> group not having a committed offset.
>
> Thanks,
> Andrew
>
> On 2026/06/26 17:36:31 Jun Rao via dev wrote:
> > Hi, Andrew,
> >
> > Thanks for the reply.
> >
> > I am not sure it's always a good idea to start new partitions with
> offset 0
> > for the share group. For example, once a share group starts, a user could
> > add a subscription to an existing topic. That existing topic could have a
> > large backlog and starting it with offset 0 forces the user to consume
> that
> > entire backlog. In KIP-1327, such a partition will be treated as an
> > existing partition and will follow the auto.offset.reset policy. This
> seems
> > more intuitive. It would be useful to consider if the share group should
> > adopt the same policy.
> >
> > Jun
> >
> > On Fri, Jun 26, 2026 at 3:42 AM Andrew Schofield <sc...@apache.org>
> > wrote:
> >
> > > Hi Jiunn-Yang et al.,
> > > Some background on the problem I hinted at for the KIP-932
> implementation.
> > > KIP-932 states "If the number of partitions is increased for a topic
> with a
> > > subscription in a share group, the SPSO for the newly created
> > > share-partitions is initialized to 0 (which is of course both the
> earliest
> > > and latest offset for an empty topic-partition). This means there is no
> > > doubt about what happens when the number of partitions is increased."
> > >
> > > The problem was that we had not implemented the use of 0, but instead
> were
> > > using the share.auto.offset.reset config to choose the initial offset
> in
> > > this case. We write a ShareGroupStatePartitionMetadata record when a
> topic
> > > is first being initialized for consumption by a share group, and this
> > > contains a list of the partitions which we are initializing. As a
> result, I
> > > believe we can already tell when partitions are added, and we do know
> when
> > > to initialize to 0 as it says in the KIP. We will put in a patch that
> uses
> > > 0 as intended.
> > >
> > > I do not mind whether we want to incorporate share groups into
> KIP-1327,
> > > but I do not think that it is necessary to do so.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > On 2026/06/26 04:31:28 Chia-Ping Tsai wrote:
> > > > > JR5. Since this changes the on-disk data format for
> PartitionRecord, we
> > > > > need to gate this by a new MV. It also changes the format of
> > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not
> sure if
> > > this
> > > > > requires a new MV or a new GV.
> > > >
> > > > That is a good point. If we want to maintain consistency for the
> > > NewPartitions field when the group coordinator migrates between new
> and old
> > > binaries, a version gate is definitely required. For example:
> > > >
> > > > GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())
> > > >
> > > > The group coordinator should NOT set the NewPartitions field when
> the gv
> > > is smaller than gv_2
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > > On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> > > > > Hi, Jiunn-Yang,
> > > > >
> > > > > JR2. We should try to maintain consistency across our APIs. So, it
> > > would be
> > > > > useful to understand if the user-facing part of this KIP is what
> share
> > > > > consumer wants too. It would be problematic if we proceed with this
> > > > > approach only for the share consumer to choose a different one
> later.
> > > > >
> > > > > JR3. I was thinking that if we add the NewPartitions field in
> > > > > GroupHeartbeatResponse, there is no need to expose the
> group/partition
> > > > > creation timestamp to other RPCs and the Cli.
> > > > >
> > > > > JR5. Since this changes the on-disk data format for
> PartitionRecord, we
> > > > > need to gate this by a new MV. It also changes the format of
> > > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not
> sure if
> > > this
> > > > > requires a new MV or a new GV.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >
> > > > > > Hello Jun, chia,
> > > > > >
> > > > > > Thanks for your feedback.
> > > > > >
> > > > > > JR3 JR4 I have updated the KIP.
> > > > > >
> > > > > > JR2 The new-partition detection mechanism is reusable. The
> > > coordinator
> > > > > > already has
> > > > > > access to both timestamps required for classification, so the
> > > > > > determination can be made
> > > > > > server-side without introducing any new RPCs. The main
> difference is
> > > where
> > > > > > the policy is
> > > > > > enforced: modern consumer resolves it on the client, whereas a
> share
> > > group
> > > > > > resolves the
> > > > > > starting offset on the partition leader. As a result, the
> > > classification
> > > > > > decision would need to
> > > > > > be propagated through the existing share-state initialization
> path.
> > > We
> > > > > > would also need to persist
> > > > > > the group's creation time as part of the share-group state, since
> > > that
> > > > > > information is not stored today.
> > > > > >
> > > > > > None of this is a blocker, but it does require changes to the
> > > share-state
> > > > > > format and introduces
> > > > > > a share-specific configuration. Given that additional scope, I
> would
> > > > > > prefer to keep share groups out
> > > > > > of scope for this KIP and address them in a follow-up effort,
> while
> > > > > > documenting the relevant reuse
> > > > > > opportunities here.
> > > > > >
> > > > > > Best Regards,
> > > > > > Jiunn-Yang
> > > > > >
> > > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > > > > > >
> > > > > > >> It inspires me to think about mirror clusters. What if the
> mirror
> > > group
> > > > > > is created 'after' the expanded partitions? In this scenario,
> the new
> > > > > > configuration 'max.age.ms' could handle it if users are aware
> of the
> > > > > > migration and set a suitable time interval. Of course, this would
> > > require
> > > > > > both creation times to be exposed to clients.
> > > > > > >
> > > > > > > Thinking more about the mirror cluster scenario we discussed, I
> > > realize
> > > > > > it introduces significant complexity. Handling it properly would
> > > require a
> > > > > > tolerance time window rather than a strict time interval to
> account
> > > for
> > > > > > replication lag. To prevent scope creep and keep KIP-1327
> focused on
> > > its
> > > > > > primary goal, I think it would be best to defer the
> > > multi-cluster/mirroring
> > > > > > scenario and explore those approaches in a separate KIP
> > > > > > >
> > > > > > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > > > > > >> hi all,
> > > > > > >>
> > > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > > client.
> > > > > > I am
> > > > > > >>> not sure if there are other use cases for those timestamps.
> An
> > > > > > alternative
> > > > > > >>> is to avoid exposing those timestamps to the client. The
> group
> > > > > > coordinator
> > > > > > >>> will categorize the partitions using those timestamps and
> > > include the
> > > > > > >>> categorization in the assigned partitions in
> > > GroupHeartbeatResponse.
> > > > > > >>
> > > > > > >> It inspires me to think about mirror clusters. What if the
> mirror
> > > group
> > > > > > is created 'after' the expanded partitions? In this scenario,
> the new
> > > > > > configuration 'max.age.ms' could handle it if users are aware
> of the
> > > > > > migration and set a suitable time interval. Of course, this would
> > > require
> > > > > > both creation times to be exposed to clients.
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Chia-Ping
> > > > > > >>
> > > > > > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > > > > > >>> Hi, Jiunn-Yang,
> > > > > > >>>
> > > > > > >>> Thanks for the updated KIP.
> > > > > > >>>
> > > > > > >>> JR2. Share consumer (KIP-932) states: "If the number of
> > > partitions is
> > > > > > >>> increased for a topic with a subscription in a share group,
> the
> > > SPSO
> > > > > > for
> > > > > > >>> the newly created share-partitions is initialized to 0 ".
> > > However, this
> > > > > > >>> part has not been implemented yet. Currently, all partitions
> > > without
> > > > > > offset
> > > > > > >>> follows the strategy given by share.auto.offset.reset. So, we
> > > have an
> > > > > > >>> opportunity to see if it's better to align this KIP's changes
> > > with the
> > > > > > >>> share consumer.
> > > > > > >>> JR2.1 The first part is whether customizing the offset for
> new
> > > > > > partitions
> > > > > > >>> to something other than 0 is useful in the share consumer. It
> > > seems
> > > > > > that
> > > > > > >>> the motivation in this KIP applies to share consumer too.
> > > > > > >>> JR2.2 The second part is how new partitions are detected. The
> > > mechanism
> > > > > > >>> introduced in this KIP seems quite effective. A group
> > > coordinator could
> > > > > > >>> pick up newly created partitions for an existing topic or
> newly
> > > created
> > > > > > >>> topics matching the regex. Those partitions are categorized
> as
> > > new
> > > > > > >>> partitions and will follow the strategy given by
> > > > > > >>> auto.offset.reset.new.partitions.
> > > > > > >>> This makes sense since those partitions typically don't have
> a
> > > large
> > > > > > >>> backlog. A consumer could also additionally subscribe to an
> > > existing
> > > > > > topic
> > > > > > >>> in an existing group. Those partitions will be categorized as
> > > existing
> > > > > > >>> partitions. This also makes sense since those partitions
> could
> > > have a
> > > > > > large
> > > > > > >>> backlog and it is better to follow the strategy given by
> > > > > > auto.offset.reset. It
> > > > > > >>> will be useful to think through whether the logic for
> > > categorizing new
> > > > > > >>> partitions should also be reused in the share consumer.
> > > > > > >>>
> > > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > > client.
> > > > > > I am
> > > > > > >>> not sure if there are other use cases for those timestamps.
> An
> > > > > > alternative
> > > > > > >>> is to avoid exposing those timestamps to the client. The
> group
> > > > > > coordinator
> > > > > > >>> will categorize the partitions using those timestamps and
> > > include the
> > > > > > >>> categorization in the assigned partitions in
> > > GroupHeartbeatResponse.
> > > > > > >>>
> > > > > > >>> JR4. It would be useful to support this feature on existing
> > > groups as
> > > > > > well.
> > > > > > >>> For example, any partition without a creation timestamp can
> > > always be
> > > > > > >>> categorized as an existing partition.
> > > > > > >>>
> > > > > > >>> Jun
> > > > > > >>>
> > > > > > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > > > > > >>>
> > > > > > >>>> Hello Andrew,
> > > > > > >>>>
> > > > > > >>>> Thanks for your feedback,
> > > > > > >>>>>> I've just been re-reading this KIP to ensure that share
> group
> > > > > > support is
> > > > > > >>>>>> not needed. I've discovered that we made a mistake
> > > implementing
> > > > > > KIP-932
> > > > > > >>>> in
> > > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > > really tight
> > > > > > >>>> enough
> > > > > > >>>>>> not to require KIP-1327.
> > > > > > >>>>
> > > > > > >>>> Could you walk us through the specific area where the
> mistake
> > > was
> > > > > > found?
> > > > > > >>>> That would
> > > > > > >>>> help us assess whether it naturally fits within the scope of
> > > KIP-1327
> > > > > > or
> > > > > > >>>> should be addressed
> > > > > > >>>> separately.
> > > > > > >>>>
> > > > > > >>>> AS1 I have updated the KIP
> > > > > > >>>>
> > > > > > >>>> Best Regards,
> > > > > > >>>> Jiunn-Yang
> > > > > > >>>>
> > > > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49
> 寫道:
> > > > > > >>>>>
> > > > > > >>>>> hi Andrew
> > > > > > >>>>>
> > > > > > >>>>>> I've discovered that we made a mistake implementing
> KIP-932
> > > in this
> > > > > > >>>> area,
> > > > > > >>>>> so we'll rectify that and check whether it's really tight
> > > enough not
> > > > > > to
> > > > > > >>>>> require KIP-1327.
> > > > > > >>>>>
> > > > > > >>>>> Would you mind sharing more details about that? If it's not
> > > anything
> > > > > > >>>> close
> > > > > > >>>>> to NP-hard, we could definitely look into covering it
> within
> > > KIP-1327
> > > > > > >>>>>
> > > > > > >>>>> Best,
> > > > > > >>>>> Chia-Ping
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三
> > > 下午7:49寫道:
> > > > > > >>>>>
> > > > > > >>>>>> Hi Jiunn-Yang,
> > > > > > >>>>>> I've just been re-reading this KIP to ensure that share
> group
> > > > > > support is
> > > > > > >>>>>> not needed. I've discovered that we made a mistake
> > > implementing
> > > > > > KIP-932
> > > > > > >>>> in
> > > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > > really tight
> > > > > > >>>> enough
> > > > > > >>>>>> not to require KIP-1327.
> > > > > > >>>>>>
> > > > > > >>>>>> One trivial comments for consistency.
> > > > > > >>>>>>
> > > > > > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > > > > > represented by
> > > > > > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks,
> > > > > > >>>>>> Andrew
> > > > > > >>>>>>
> > > > > > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > > > > > >>>>>>> Hi all,
> > > > > > >>>>>>>
> > > > > > >>>>>>> Manually bumping this thread.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Best Regards,
> > > > > > >>>>>>> Jiunn-Yang
> > > > > > >>>>>>>
> > > > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Hello chia,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Best Regards,
> > > > > > >>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日
> 凌晨12:47
> > > 寫道:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> hi Jiunn-Yang
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> When the config is set on a cluster that has not yet
> been
> > > > > > >>>>>> upgraded... classification cannot occur... the consumer
> falls
> > > back
> > > > > > to
> > > > > > >>>> the
> > > > > > >>>>>> base auto.offset.reset for the affected partitions. No
> > > exception is
> > > > > > >>>> thrown,
> > > > > > >>>>>> and no operational disruption results.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Existing group can't take advantage of this excellent
> new
> > > > > > >>>>>> configuration. Allowing users to modify the group creation
> > > time
> > > > > > might be
> > > > > > >>>>>> overkill. Instead, we could print a useful warning
> message to
> > > guide
> > > > > > >>>> users.
> > > > > > >>>>>> For example, we can suggest that they re-create the group
> > > with their
> > > > > > >>>>>> existing committed offsets
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Protocol changes
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Would you mind listing those RPC changes in a table
> format?
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> The full interaction matrix between the base policy
> and
> > > the
> > > > > > >>>>>> new-partition policy is:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Please add a filed to describe the target scenario when
> > > using
> > > > > > these
> > > > > > >>>>>> policies
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Best,
> > > > > > >>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > > > > > >>>>>>>>>> Hello Jun, chia,
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for
> the
> > > new
> > > > > > >>>>>>>>>> approach, PTAL
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日
> 上午8:23
> > > 寫道:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> hi Jun
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Yes, your approach is great. I think the combination
> of
> > > latest
> > > > > > (for
> > > > > > >>>>>> existing partitions) and by_duration (for new partitions)
> can
> > > > > > address
> > > > > > >>>> 99%
> > > > > > >>>>>> of the complaints I have heard regarding this issue.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Also, leveraging the group creation time here opens
> the
> > > door to
> > > > > > >>>>>> implementing a new policy based on timestamp seek in the
> > > future,
> > > > > > should
> > > > > > >>>> the
> > > > > > >>>>>> community want to pursue that.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Thanks for your patience and constructive feedback.
> We
> > > will
> > > > > > update
> > > > > > >>>>>> the KIP accordingly.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> Best, Chia-Ping
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日
> > > 清晨5:11
> > > > > > 寫道:
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > > > > > configure a
> > > > > > >>>>>> different
> > > > > > >>>>>>>>>>>> offset policy for existing partitions vs new
> partitions.
> > > > > > However,
> > > > > > >>>>>> using
> > > > > > >>>>>>>>>>>> group creation time to capture that seems more
> > > intuitive.
> > > > > > Here is
> > > > > > >>>>>> another
> > > > > > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and
> > > categorize
> > > > > > new
> > > > > > >>>>>> partitions
> > > > > > >>>>>>>>>>>> based on group creation time. Introduce
> > > > > > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose
> > > values
> > > > > > can be
> > > > > > >>>>>> earliest,
> > > > > > >>>>>>>>>>>> latest and by_duration, the same as
> auto.offset.reset.
> > > Users
> > > > > > can
> > > > > > >>>>>> set
> > > > > > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if
> > > they
> > > > > > want to
> > > > > > >>>>>> guarantee
> > > > > > >>>>>>>>>>>> no data loss on new partitions. They can also use
> > > by_duration
> > > > > > to
> > > > > > >>>>>> set an
> > > > > > >>>>>>>>>>>> upper bound on the backlog replayed, which can be
> > > different
> > > > > > from
> > > > > > >>>>>> that of
> > > > > > >>>>>>>>>>>> the existing partitions. This will address your
> concern
> > > about
> > > > > > too
> > > > > > >>>>>> much
> > > > > > >>>>>>>>>>>> backlog being replayed when the offsets are lost.
> What
> > > do you
> > > > > > >>>>>> think?
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > > > > > >>>>>> chia7712@gmail.com> wrote:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> hi Jun
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> The most important part of this story is how users
> > > should
> > > > > > expect
> > > > > > >>>>>> the data
> > > > > > >>>>>>>>>>>>> they can see when using the latest or by_duration
> > > policy with
> > > > > > >>>>>> expanded
> > > > > > >>>>>>>>>>>>> partitions.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss,
> > > but it is
> > > > > > >>>>>>>>>>>>> non-deterministic, which means users will either
> read
> > > too
> > > > > > many
> > > > > > >>>>>> historical
> > > > > > >>>>>>>>>>>>> records from existing partitions or lose some
> records
> > > from
> > > > > > >>>>>> expanded
> > > > > > >>>>>>>>>>>>> partitions.
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > > > > > >>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>> is a bit hard to understand, and that is why I
> > > preferred
> > > > > > having a
> > > > > > >>>>>> whole new
> > > > > > >>>>>>>>>>>>> policy based entirely on group creation time
> (KIP-1282)
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> 2026年6月16日週二
> > > > > > 上午1:08寫道:
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to
> understand
> > > the
> > > > > > value
> > > > > > >>>>>> of the new
> > > > > > >>>>>>>>>>>>>> configs with the KIP.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't
> want
> > > to
> > > > > > miss
> > > > > > >>>>>> the data if
> > > > > > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing
> > > partition
> > > > > > is
> > > > > > >>>>>> easy to
> > > > > > >>>>>>>>>>>>>> understand because it relates to retention time.
> The
> > > > > > backlog for
> > > > > > >>>>>> the new
> > > > > > >>>>>>>>>>>>>> partition is a bit subtle to understand since it
> > > depends on
> > > > > > the
> > > > > > >>>>>> metadata
> > > > > > >>>>>>>>>>>>>> refresh delay. To set
> auto.offset.reset.max.age.ms
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > >>>>>>> ,
> > > > > > >>>>>>>>>>>>>> the user needs to
> > > > > > >>>>>>>>>>>>>> understand the metadata refresh delay on the
> consumer
> > > side
> > > > > > and
> > > > > > >>>>>> use it to
> > > > > > >>>>>>>>>>>>>> set the config.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the
> same
> > > value
> > > > > > for
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three
> > > issues with
> > > > > > >>>> this
> > > > > > >>>>>>>>>>>>>> approach.
> > > > > > >>>>>>>>>>>>>> 1. It computes the seek target client-side as
> now() -
> > > > > > duration,
> > > > > > >>>>>> which
> > > > > > >>>>>>>>>>>>>> introduces clock skew across consumers and forces
> > > operators
> > > > > > to
> > > > > > >>>>>> choose
> > > > > > >>>>>>>>>>>>>> overly large durations, causing unnecessary
> > > reprocessing.
> > > > > > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each
> retry,
> > > so
> > > > > > failed
> > > > > > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target
> > > forward and
> > > > > > >>>>>> potentially
> > > > > > >>>>>>>>>>>>>> miss records produced between attempts.
> > > > > > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without
> > > committed
> > > > > > >>>>>> offsets, and
> > > > > > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > > > > > long-existing
> > > > > > >>>>>> partitions
> > > > > > >>>>>>>>>>>>>> newly assigned to the group, leading to
> unnecessary
> > > replay.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated
> by
> > > adding
> > > > > > a bit
> > > > > > >>>>>> buffer to
> > > > > > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > > > > > improving the
> > > > > > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh
> > > delay is
> > > > > > >>>>>> typically low
> > > > > > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer
> > > and tens
> > > > > > of
> > > > > > >>>>>> seconds
> > > > > > >>>>>>>>>>>>>> with
> > > > > > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading
> that
> > > much
> > > > > > >>>>>> backlog for new
> > > > > > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the
> same
> > > for
> > > > > > existing
> > > > > > >>>>>>>>>>>>>> partitions.
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> So, instead of introducing a new config, could we
> just
> > > > > > reuse the
> > > > > > >>>>>> existing
> > > > > > >>>>>>>>>>>>>> config with better documentation and/or
> > > implementation?
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <
> > > s7133700@gmail.com>
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Hello Jun,
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> You're right that group creation time is the more
> > > intuitive
> > > > > > >>>>>> answer at
> > > > > > >>>>>>>>>>>>>>> first glance,
> > > > > > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions
> that
> > > > > > "predate
> > > > > > >>>>>> the group"
> > > > > > >>>>>>>>>>>>>>> vs partitions
> > > > > > >>>>>>>>>>>>>>> "created during group runtime," which directly
> > > points to a
> > > > > > >>>>>>>>>>>>>> group-lifecycle
> > > > > > >>>>>>>>>>>>>>> classifier.
> > > > > > >>>>>>>>>>>>>>> I'd like to walk through why we landed on
> partition
> > > age,
> > > > > > and
> > > > > > >>>> the
> > > > > > >>>>>>>>>>>>>>> trade-offs we considered.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> This covers the metadata blindness window, but
> has
> > > issues
> > > > > > the
> > > > > > >>>>>> KIP
> > > > > > >>>>>>>>>>>>>>> currently documents
> > > > > > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock
> > > skew
> > > > > > across
> > > > > > >>>>>> consumers.
> > > > > > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > > > > potentially
> > > > > > >>>>>> missing
> > > > > > >>>>>>>>>>>>>>> records produced between
> > > > > > >>>>>>>>>>>>>>> attempts.
> > > > > > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without
> > > committed
> > > > > > >>>>>> offsets,
> > > > > > >>>>>>>>>>>>>>> including pre-existing partitions
> > > > > > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary
> > > replay.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively
> > > running.
> > > > > > Our
> > > > > > >>>>>> concern
> > > > > > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> T=0:         Group created.
> > > > > > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected,
> > > etc.).
> > > > > > >>>>>>>>>>>>>>> T=50:        Partition added during the idle
> window.
> > > > > > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Under group creation time, the new partition is
> > > classified
> > > > > > as
> > > > > > >>>>>> new
> > > > > > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying
> > > everything
> > > > > > from
> > > > > > >>>>>> T=50.
> > > > > > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also
> > > accumulated
> > > > > > >>>>>> data that
> > > > > > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely
> the
> > > > > > contract
> > > > > > >>>> of
> > > > > > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no
> principled
> > > reason
> > > > > > to
> > > > > > >>>>>> treat
> > > > > > >>>>>>>>>>>>>>> the new partition differently; both contain
> backlog
> > > > > > accumulated
> > > > > > >>>>>> during
> > > > > > >>>>>>>>>>>>>>> the same idle window.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog”
> principle
> > > you
> > > > > > raised
> > > > > > >>>>>> in
> > > > > > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has
> tolerated
> > > some
> > > > > > backlog
> > > > > > >>>>>> on
> > > > > > >>>>>>>>>>>>>>> every other partition during the same idle
> period;
> > > forcing
> > > > > > >>>>>> 0-backlog
> > > > > > >>>>>>>>>>>>>>> tolerance only on new partitions would be
> > > inconsistent with
> > > > > > >>>> that
> > > > > > >>>>>>>>>>>>>>> tolerance.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent
> data
> > > loss
> > > > > > >>>> window,
> > > > > > >>>>>>>>>>>>>>> the gap between partition creation and the
> consumer’s
> > > > > > metadata
> > > > > > >>>>>>>>>>>>>>> refresh. Within this window, data loss is
> genuinely
> > > > > > silent: the
> > > > > > >>>>>>>>>>>>>>> consumer had no opportunity to know about the
> > > partition.
> > > > > > >>>>>> Outside this
> > > > > > >>>>>>>>>>>>>>> window, missing data reflects either:
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with
> idle
> > > > > > consumers,
> > > > > > >>>>>> or
> > > > > > >>>>>>>>>>>>>>> - (b) an operational issue to surface via
> > > monitoring, not
> > > > > > via
> > > > > > >>>>>> reset
> > > > > > >>>>>>>>>>>>>> policy.
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> We did not choose partition age because it is
> more
> > > elegant
> > > > > > than
> > > > > > >>>>>> group
> > > > > > >>>>>>>>>>>>>>> creation time — we chose it because its failure
> mode
> > > > > > (requires
> > > > > > >>>> a
> > > > > > >>>>>>>>>>>>>>> threshold) is
> > > > > > >>>>>>>>>>>>>>> less invasive than the failure mode of group
> > > creation time
> > > > > > >>>>>> (overrides
> > > > > > >>>>>>>>>>>>>>> user-stated
> > > > > > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> 2026年6月13日
> > > 上午11:52
> > > > > > 寫道:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Hi Jun,
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Relying on both creation times will create an
> > > inconsistent
> > > > > > >>>>>> scenario. A
> > > > > > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long
> sleep
> > > will
> > > > > > seek
> > > > > > >>>>>> to the
> > > > > > >>>>>>>>>>>>>>>> beginning for the partitions created later than
> the
> > > group.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to
> fix
> > > the
> > > > > > >>>>>> inconsistency
> > > > > > >>>>>>>>>>>>>>> using a
> > > > > > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach
> a
> > > > > > consensus,
> > > > > > >>>>>> KIP-1327
> > > > > > >>>>>>>>>>>>>>> goes
> > > > > > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent
> > > users
> > > > > > from
> > > > > > >>>>>> falling
> > > > > > >>>>>>>>>>>>>> into
> > > > > > >>>>>>>>>>>>>>>> that pitfall.
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > 2026年6月13日週六
> > > > > > >>>>>> 上午6:49寫道:
> > > > > > >>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late
> reply.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> JR1. The design of
> auto.offset.reset.max.age.ms
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>>> still feels weird to
> > > > > > >>>>>>>>>>>>>>> me.
> > > > > > >>>>>>>>>>>>>>>>> It
> > > > > > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing
> based on
> > > the
> > > > > > >>>>>> partition
> > > > > > >>>>>>>>>>>>>>> creation
> > > > > > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be
> > > based on
> > > > > > the
> > > > > > >>>>>> group
> > > > > > >>>>>>>>>>>>>>> creation
> > > > > > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is
> > > created
> > > > > > are
> > > > > > >>>>>> existing
> > > > > > >>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>> all partitions created after the group creation
> > > are new
> > > > > > >>>>>> partitions.
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <
> > > s7133700@gmail.com>
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no
> > > further
> > > > > > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16
> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common
> > > operational
> > > > > > task,
> > > > > > >>>>>> not an
> > > > > > >>>>>>>>>>>>>> edge
> > > > > > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section
> > > accordingly.
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日
> > > 下午5:49 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be
> useful
> > > to
> > > > > > clarify
> > > > > > >>>>>> that
> > > > > > >>>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>>> is a
> > > > > > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case,
> which
> > > > > > further
> > > > > > >>>>>>>>>>>>>> demonstrates
> > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > > > > > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common
> operational
> > > task in
> > > > > > >>>>>> Kafka: To
> > > > > > >>>>>>>>>>>>>>>>> balance
> > > > > > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are
> > > typically
> > > > > > >>>>>> created with a
> > > > > > >>>>>>>>>>>>>>>>>> moderate
> > > > > > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic
> > > grows
> > > > > > over
> > > > > > >>>>>> time, it
> > > > > > >>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>> often
> > > > > > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of
> partitions
> > > to
> > > > > > >>>>>> accommodate the
> > > > > > >>>>>>>>>>>>>>>>> higher
> > > > > > >>>>>>>>>>>>>>>>>>>> workload.
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> Regards
> > > > > > >>>>>>>>>>>>>>>>>>>> Jian
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六
> 22:31写道:
> > > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the
> > > KIP!
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > 2026年5月30日
> > > > > > 晚上8:29
> > > > > > >>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot"
> and
> > > "cold"
> > > > > > when
> > > > > > >>>>>>>>>>>>>> describing
> > > > > > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you
> are
> > > using
> > > > > > them
> > > > > > >>>> to
> > > > > > >>>>>>>>>>>>>> describe
> > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the
> > > records, but
> > > > > > >>>>>> applying
> > > > > > >>>>>>>>>>>>>> these
> > > > > > >>>>>>>>>>>>>>>>>> terms
> > > > > > >>>>>>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit
> unnatural.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't
> > > really care
> > > > > > >>>>>> whether a
> > > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only
> expectation
> > > is
> > > > > > that
> > > > > > >>>>>> they won't
> > > > > > >>>>>>>>>>>>>>>>>> silently
> > > > > > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the
> topic
> > > during
> > > > > > their
> > > > > > >>>>>> active
> > > > > > >>>>>>>>>>>>>>>>>>>>> consumption.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六
> > > 下午12:30寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated
> the
> > > KIP
> > > > > > >>>>>> motivation
> > > > > > >>>>>>>>>>>>>>> section.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org>
> 於
> > > > > > 2026年5月30日
> > > > > > >>>>>> 凌晨1:12
> > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a
> > > stronger
> > > > > > >>>>>> motivation
> > > > > > >>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>> KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that
> not
> > > all
> > > > > > >>>>>> partitions
> > > > > > >>>>>>>>>>>>>> without
> > > > > > >>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly
> > > expanded
> > > > > > >>>>>> partition
> > > > > > >>>>>>>>>>>>>> (hot)
> > > > > > >>>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a
> partition the
> > > > > > consumer
> > > > > > >>>>>> has
> > > > > > >>>>>>>>>>>>>> never
> > > > > > >>>>>>>>>>>>>>>>> seen
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)."
> Why
> > > is the
> > > > > > hot
> > > > > > >>>>>>>>>>>>>> partition
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration
> > > policy is
> > > > > > also
> > > > > > >>>>>>>>>>>>>>> insufficient
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> because:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() -
> > > duration)
> > > > > > varies
> > > > > > >>>>>> across
> > > > > > >>>>>>>>>>>>>> nodes
> > > > > > >>>>>>>>>>>>>>>>>> due
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set
> an
> > > overly
> > > > > > large
> > > > > > >>>>>>>>>>>>>> duration,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> causing
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client
> > > recalculates the
> > > > > > seek
> > > > > > >>>>>> time on
> > > > > > >>>>>>>>>>>>>>>>> retry,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward
> and
> > > risking
> > > > > > data
> > > > > > >>>>>> loss."
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are
> rare.
> > > If
> > > > > > these
> > > > > > >>>>>> issues
> > > > > > >>>>>>>>>>>>>>>>> persist,
> > > > > > >>>>>>>>>>>>>>>>>>>>> more
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere.
> Rare
> > > > > > >>>>>> situations don't
> > > > > > >>>>>>>>>>>>>>>>> need a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> common
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > > > > situations,
> > > > > > >>>>>> they can
> > > > > > >>>>>>>>>>>>>>>>>> implement
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > ConsumerRebalanceListener.onPartitionsAssigned().
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > > > > >>>>>> s7133700@gmail.com>
> > > > > > >>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the
> returned
> > > value
> > > > > > >>>>>> should
> > > > > > >>>>>>>>>>>>>> always
> > > > > > >>>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the
> > > KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > > positive
> > > > > > >>>>>> any
> > > > > > >>>>>>>>>>>>>>>>>> field
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can
> detect
> > > the
> > > > > > version
> > > > > > >>>>>>>>>>>>>> mismatch
> > > > > > >>>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>>>> during
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment,
> > > which
> > > > > > occurs
> > > > > > >>>>>> inside
> > > > > > >>>>>>>>>>>>>>>>> poll().
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > > > > UnsupportedVersionException
> > > > > > >>>>>> from
> > > > > > >>>>>>>>>>>>>> poll().
> > > > > > >>>>>>>>>>>>>>>>>> I’ll
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> 於
> > > > > > 2026年5月17日
> > > > > > >>>>>> 下午4:50
> > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1):
> The
> > > age of
> > > > > > this
> > > > > > >>>>>>>>>>>>>> partition
> > > > > > >>>>>>>>>>>>>>>>> in
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by
> the
> > > broker
> > > > > > as
> > > > > > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> -
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if
> the
> > > broker
> > > > > > >>>>>> does not
> > > > > > >>>>>>>>>>>>>>>>> support
> > > > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time
> is
> > > > > > unknown.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the
> returned
> > > value
> > > > > > >>>>>> should
> > > > > > >>>>>>>>>>>>>> always
> > > > > > >>>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > > positive
> > > > > > >>>>>> any
> > > > > > >>>>>>>>>>>>>>>>>> field
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter
> > > UnsupportedVersionException
> > > > > > when
> > > > > > >>>>>> calling
> > > > > > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design
> > > change
> > > > > > based on
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> discussion
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the
> > > new-partition
> > > > > > >>>> reset
> > > > > > >>>>>>>>>>>>>> behavior
> > > > > > >>>>>>>>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>>> now applies to all
> > > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration,
> none).
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the
> > > consumer
> > > > > > resets
> > > > > > >>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>>> .partitions
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions,
> the
> > > base
> > > > > > >>>>>>>>>>>>>> auto.offset.reset
> > > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > > > > represented
> > > > > > >>>>>> by a
> > > > > > >>>>>>>>>>>>>>> separate
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>> .partitions,
> > > > > > >>>>>>>>>>>>>> currently fixed to
> > > > > > >>>>>>>>>>>>>>>>> earliest).
> > > > > > >>>>>>>>>>>>>>>>>>>>> This
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the
> > > behavior to a
> > > > > > >>>>>> public
> > > > > > >>>>>>>>>>>>>>>>>> user-facing
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> 於
> > > > > > 2026年5月16日
> > > > > > >>>>>> 清晨7:46
> > > > > > >>>>>>>>>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The
> proposal
> > > from me
> > > > > > is
> > > > > > >>>>>> listed
> > > > > > >>>>>>>>>>>>>>> below:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>> .partitions
> > > > > > >>>>>>>>>>>>>> with a default value
> > > > > > >>>>>>>>>>>>>>>>> of
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from
> both
> > > > > > >>>>>> by_duration and
> > > > > > >>>>>>>>>>>>>>>>> latest,
> > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > > > > >>>>>> auto.offset.reset=earliest.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>> .partitions
> > > > > > >>>>>>>>>>>>>> as an internal
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>>> .partitions=earliest
> > > > > > >>>>>>>>>>>>>>> already
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss
> > > the use
> > > > > > cases
> > > > > > >>>>>> of
> > > > > > >>>>>>>>>>>>>> other
> > > > > > >>>>>>>>>>>>>>>>>> values
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> in a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs,
> auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>> .partitions
> > > > > > >>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > > > > > >>>>>>> ,
> > > > > > >>>>>>>>>>>>>> will be applied to all for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via
> dev
> > > wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the
> KIP
> > > says
> > > > > > "When
> > > > > > >>>>>> a Kafka
> > > > > > >>>>>>>>>>>>>>>>> topic
> > > > > > >>>>>>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers
> using
> > > the
> > > > > > latest
> > > > > > >>>>>> auto
> > > > > > >>>>>>>>>>>>>> offset
> > > > > > >>>>>>>>>>>>>>>>>> reset
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records
> > > produced to
> > > > > > those
> > > > > > >>>>>>>>>>>>>> partitions
> > > > > > >>>>>>>>>>>>>>>>>> before
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a
> user
> > > sets
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec,
> > > the same
> > > > > > >>>>>> record loss
> > > > > > >>>>>>>>>>>>>>>>> issue
> > > > > > >>>>>>>>>>>>>>>>>>>>> could
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking
> > > auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>> .partitions
> > > > > > >>>>>>>>>>>>>> will
> > > > > > >>>>>>>>>>>>>>> take
> > > > > > >>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> same
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a
> user
> > > could
> > > > > > set
> > > > > > >>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>> by_duration if
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM
> > > Chia-Ping
> > > > > > Tsai <
> > > > > > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might
> be
> > > missing
> > > > > > >>>>>> something
> > > > > > >>>>>>>>>>>>>>>>>> important
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with
> me as
> > > I try
> > > > > > to
> > > > > > >>>>>> clarify
> > > > > > >>>>>>>>>>>>>> with
> > > > > > >>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>> few
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for
> > > extending
> > > > > > this
> > > > > > >>>>>> logic
> > > > > > >>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>> other
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> reset
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies
> like
> > > > > > earliest
> > > > > > >>>>>> or
> > > > > > >>>>>>>>>>>>>>>>> by_duration
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> don't
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent
> data
> > > loss
> > > > > > issue
> > > > > > >>>>>> when a
> > > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect
> users
> > > to
> > > > > > >>>>>> configure for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>> .partitions?
> > > > > > >>>>>>>>>>>>>> If they set it to
> > > > > > >>>>>>>>>>>>>>>>> earliest
> > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same
> edge
> > > > > > cases. For
> > > > > > >>>>>>>>>>>>>> example,
> > > > > > >>>>>>>>>>>>>>>>> if a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new
> > > partition is
> > > > > > >>>>>> created
> > > > > > >>>>>>>>>>>>>> during
> > > > > > >>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to
> skip
> > > to
> > > > > > latest
> > > > > > >>>>>> when
> > > > > > >>>>>>>>>>>>>>>>> resuming,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> rather
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just
> because the
> > > > > > partition
> > > > > > >>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>> technically
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > > > > > introducing a
> > > > > > >>>>>> max.age
> > > > > > >>>>>>>>>>>>>>>>>> threshold.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> It
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define
> what
> > > is
> > > > > > >>>>>> genuinely
> > > > > > >>>>>>>>>>>>>>> "hot/new"
> > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't
> > > seen yet.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao
> via dev
> > > > > > wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find
> > > auto.offset.reset.latest.max.age a
> > > > > > bit
> > > > > > >>>>>> weird. It
> > > > > > >>>>>>>>>>>>>>>>> only
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest.
> > > However, it
> > > > > > seems
> > > > > > >>>>>> that the
> > > > > > >>>>>>>>>>>>>>>>>>>>> motivation
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is
> > > set to
> > > > > > other
> > > > > > >>>>>> values
> > > > > > >>>>>>>>>>>>>> like
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to
> have a
> > > > > > separate
> > > > > > >>>>>> way to
> > > > > > >>>>>>>>>>>>>>> control
> > > > > > >>>>>>>>>>>>>>>>>>>>> newly
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions
> > > when the
> > > > > > >>>> group
> > > > > > >>>>>>>>>>>>>> starts.
> > > > > > >>>>>>>>>>>>>>>>>> Have we
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > > > > auto.offset.reset.new
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>>>> .partitions?
> > > > > > >>>>>>>>>>>>>>>>>> If
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset
> reset
> > > policy
> > > > > > >>>>>> defaults to
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user
> could
> > > set it
> > > > > > >>>>>> explicitly
> > > > > > >>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>> customize
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM
> 黃竣陽 <
> > > > > > >>>>>> s7133700@gmail.com
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this
> > > thread.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於
> > > 2026年5月1日
> > > > > > 晚上10:37
> > > > > > >>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from
> v13 to
> > > v14.
> > > > > > The
> > > > > > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64,
> > > default -1),
> > > > > > >>>>>> computed
> > > > > > >>>>>>>>>>>>>>>>>> server-side
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> by
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > > > > >>>> partition_creation_time.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat
> > > flow.
> > > > > > when
> > > > > > >>>>>>>>>>>>>>>>>> MembershipManager
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly
> > > invalidates the
> > > > > > >>>>>> metadata for
> > > > > > >>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> affected
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh
> MetadataRequest
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset
> > > decision,
> > > > > > even
> > > > > > >>>>>> if the
> > > > > > >>>>>>>>>>>>>>>>> topic
> > > > > > >>>>>>>>>>>>>>>>>> ID
> > > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the
> broker's
> > > maximum
> > > > > > >>>>>> supported
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at
> > > connection
> > > > > > time.
> > > > > > >>>>>> If the
> > > > > > >>>>>>>>>>>>>>>>>> negotiated
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not
> support
> > > > > > >>>>>> PartitionAgeMs at
> > > > > > >>>>>>>>>>>>>> all
> > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>>>>>> can
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than
> silently
> > > falling
> > > > > > >>>>>> back to
> > > > > > >>>>>>>>>>>>>> latest
> > > > > > >>>>>>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any
> > > operator-visible
> > > > > > signal.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes
> in
> > > the
> > > > > > KIP.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > > chia7712@gmail.com> 於
> > > > > > >>>>>> 2026年4月29日
> > > > > > >>>>>>>>>>>>>>> 下午4:04
> > > > > > >>>>>>>>>>>>>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of
> > > moving the
> > > > > > >>>>>> 'age'
> > > > > > >>>>>>>>>>>>>>>>> resolution
> > > > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata
> API
> > > to keep
> > > > > > the
> > > > > > >>>>>> control
> > > > > > >>>>>>>>>>>>>>>>> plane
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before,
> is
> > > > > > introducing
> > > > > > >>>>>>>>>>>>>>> inter-broker
> > > > > > >>>>>>>>>>>>>>>>>>>>> clock
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach
> provided
> > > a
> > > > > > single
> > > > > > >>>>>> source of
> > > > > > >>>>>>>>>>>>>>>>> truth
> > > > > > >>>>>>>>>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this
> time
> > > skew
> > > > > > >>>>>> should be
> > > > > > >>>>>>>>>>>>>>>>>> negligible.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will
> likely be
> > > > > > >>>>>> configured in
> > > > > > >>>>>>>>>>>>>>> minutes
> > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in
> milliseconds)
> > > between
> > > > > > >>>>>> brokers
> > > > > > >>>>>>>>>>>>>> won't
> > > > > > >>>>>>>>>>>>>>>>>> impact
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > > > > dev@kafka.apache.org>
> > > > > > >>>>>> 於
> > > > > > >>>>>>>>>>>>>>>>> 2026年4月29日
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really
> followed
> > > the
> > > > > > >>>>>> previous
> > > > > > >>>>>>>>>>>>>>>>>> conversation
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> but I
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly
> understand
> > > the
> > > > > > flow
> > > > > > >>>>>> with the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP.
> There
> > > is a
> > > > > > new
> > > > > > >>>>>> boolean;
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> KIP
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned
> only
> > > when
> > > > > > this
> > > > > > >>>>>> boolean
> > > > > > >>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>> set.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer
> > > receives
> > > > > > a new
> > > > > > >>>>>>>>>>>>>> partition,
> > > > > > >>>>>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>>>>> will
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean
> set
> > > to
> > > > > > receive
> > > > > > >>>>>> the
> > > > > > >>>>>>>>>>>>>> ages.
> > > > > > >>>>>>>>>>>>>>> Is
> > > > > > >>>>>>>>>>>>>>>>>> my
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps
> > > clarify the
> > > > > > flow
> > > > > > >>>>>> and
> > > > > > >>>>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>>>>> explain
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g.
> list
> > > > > > offsets,
> > > > > > >>>>>> fetch
> > > > > > >>>>>>>>>>>>>>>>> offsets,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is
> > > correct, I
> > > > > > >>>>>> wonder if
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat
> API
> > > is the
> > > > > > right
> > > > > > >>>>>> place
> > > > > > >>>>>>>>>>>>>> for
> > > > > > >>>>>>>>>>>>>>>>>> this
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> given
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > > > > Alternatively,
> > > > > > >>>>>> it could
> > > > > > >>>>>>>>>>>>>>>>> simply
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we
> should be
> > > > > > rather
> > > > > > >>>>>> cautious
> > > > > > >>>>>>>>>>>>>>> about
> > > > > > >>>>>>>>>>>>>>>>>> not
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat
> API
> > > with
> > > > > > >>>>>> unrelated
> > > > > > >>>>>>>>>>>>>>> concepts.
> > > > > > >>>>>>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> API
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for
> assigning
> > > or
> > > > > > revoking
> > > > > > >>>>>>>>>>>>>>> partitions.
> > > > > > >>>>>>>>>>>>>>>>>> The
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> fact
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > > > > corresponding
> > > > > > >>>>>> Streams
> > > > > > >>>>>>>>>>>>>> API
> > > > > > >>>>>>>>>>>>>>>>>> also
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right.
> > > What
> > > > > > would we
> > > > > > >>>>>> do if
> > > > > > >>>>>>>>>>>>>> we
> > > > > > >>>>>>>>>>>>>>>>>> want to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at
> 12:28 AM
> > > > > > >>>>>> Muralidhar Basani
> > > > > > >>>>>>>>>>>>>>> via
> > > > > > >>>>>>>>>>>>>>>>>> dev
> > > > > > >>>>>>>>>>>>>>>>>>>>> <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org>
> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great
> kip.
> > > Good to
> > > > > > know
> > > > > > >>>>>> about
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> gap.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version
> > > bump for
> > > > > > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on
> > > response,
> > > > > > >>>>>> PartitionAges
> > > > > > >>>>>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version
> > > bump?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config,
> is
> > > there a
> > > > > > >>>>>> recommended
> > > > > > >>>>>>>>>>>>>>>>> value
> > > > > > >>>>>>>>>>>>>>>>>> or
> > > > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it
> should
> > > based
> > > > > > on the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > > > > >>>>>>>>>>>>>> <
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > > > > > >>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of
> > > javadocs I
> > > > > > >>>>>> guess.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are
> no
> > > > > > changes to
> > > > > > >>>>>> Kafka
> > > > > > >>>>>>>>>>>>>>>>> Streams,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> would
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > > > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > > > > >>>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users
> > > configure
> > > > > > it?
> > > > > > >>>>>> This is
> > > > > > >>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>> most
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users
> might
> > > easily
> > > > > > >>>>>> mistakenly
> > > > > > >>>>>>>>>>>>>>>>>> configure
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing
> "the
> > > > > > consumer
> > > > > > >>>>>> falls
> > > > > > >>>>>>>>>>>>>> back
> > > > > > >>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being
> changed
> > > > > > >>>>>> per-partition
> > > > > > >>>>>>>>>>>>>> which
> > > > > > >>>>>>>>>>>>>>>>>> isn't
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something
> like
> > > > > > "consumer
> > > > > > >>>>>> resolves
> > > > > > >>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> initial
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that
> > > partition" as if
> > > > > > >>>>>> earliest
> > > > > > >>>>>>>>>>>>>> was
> > > > > > >>>>>>>>>>>>>>>>>> applied
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and
> > > auto.offset.reset
> > > > > > >>>>>> config is
> > > > > > >>>>>>>>>>>>>>>>>> unchanged.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at
> > > 2:48 PM 黃竣陽 <
> > > > > > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to
> > > include
> > > > > > this
> > > > > > >>>>>> change.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > > chia7712@apache.org>
> > > > > > 於
> > > > > > >>>>>>>>>>>>>> 2026年4月28日
> > > > > > >>>>>>>>>>>>>>>>>> 晚上8:03
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose
> the
> > > > > > partition
> > > > > > >>>>>> creation
> > > > > > >>>>>>>>>>>>>>> time
> > > > > > >>>>>>>>>>>>>>>>>> via
> > > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be
> valuable
> > > for
> > > > > > users
> > > > > > >>>> to
> > > > > > >>>>>>>>>>>>>> diagnose
> > > > > > >>>>>>>>>>>>>>>>> and
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > > > > >>>>>> auto.offset.reset.latest.max.age
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽
> > > wrote:
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> > > discussion
> > > > > > on
> > > > > > >>>>>> KIP-1327
> > > > > > >>>>>>>>>>>>>>>>>> Prevent
> > > > > > >>>>>>>>>>>>>>>>>>>>> Hot
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for
> Latest
> > > Policy
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > >
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
> > > introduces
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets
> the
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy
> > > distinguish
> > > > > > newly
> > > > > > >>>>>> expanded
> > > > > > >>>>>>>>>>>>>>>>> (hot)
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones.
> > > Partitions
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the
> configured
> > > > > > threshold
> > > > > > >>>>>>>>>>>>>>> automatically
> > > > > > >>>>>>>>>>>>>>>>>> fall
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent
> > > data loss
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion
> > > without
> > > > > > forcing
> > > > > > >>>>>> a full
> > > > > > >>>>>>>>>>>>>>>>>> historical
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>>
> > > > > > >>>>>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Andrew Schofield <sc...@apache.org>.
Hi Jun,
When a user adds an existing topic to a share group subscription, we do use the auto.offset.reset policy because those partitions already exist and may well have data on them. We only use 0 in the case where a topic already being consumed gained some additional new partitions. These partitions will be empty at the time of creation, so zero seems like a reasonable starting point. Partitions being consumed in a share group always have a known starting position once assigned. We don't have the equivalent of a consumer group not having a committed offset.

Thanks,
Andrew

On 2026/06/26 17:36:31 Jun Rao via dev wrote:
> Hi, Andrew,
> 
> Thanks for the reply.
> 
> I am not sure it's always a good idea to start new partitions with offset 0
> for the share group. For example, once a share group starts, a user could
> add a subscription to an existing topic. That existing topic could have a
> large backlog and starting it with offset 0 forces the user to consume that
> entire backlog. In KIP-1327, such a partition will be treated as an
> existing partition and will follow the auto.offset.reset policy. This seems
> more intuitive. It would be useful to consider if the share group should
> adopt the same policy.
> 
> Jun
> 
> On Fri, Jun 26, 2026 at 3:42 AM Andrew Schofield <sc...@apache.org>
> wrote:
> 
> > Hi Jiunn-Yang et al.,
> > Some background on the problem I hinted at for the KIP-932 implementation.
> > KIP-932 states "If the number of partitions is increased for a topic with a
> > subscription in a share group, the SPSO for the newly created
> > share-partitions is initialized to 0 (which is of course both the earliest
> > and latest offset for an empty topic-partition). This means there is no
> > doubt about what happens when the number of partitions is increased."
> >
> > The problem was that we had not implemented the use of 0, but instead were
> > using the share.auto.offset.reset config to choose the initial offset in
> > this case. We write a ShareGroupStatePartitionMetadata record when a topic
> > is first being initialized for consumption by a share group, and this
> > contains a list of the partitions which we are initializing. As a result, I
> > believe we can already tell when partitions are added, and we do know when
> > to initialize to 0 as it says in the KIP. We will put in a patch that uses
> > 0 as intended.
> >
> > I do not mind whether we want to incorporate share groups into KIP-1327,
> > but I do not think that it is necessary to do so.
> >
> > Thanks,
> > Andrew
> >
> > On 2026/06/26 04:31:28 Chia-Ping Tsai wrote:
> > > > JR5. Since this changes the on-disk data format for PartitionRecord, we
> > > > need to gate this by a new MV. It also changes the format of
> > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if
> > this
> > > > requires a new MV or a new GV.
> > >
> > > That is a good point. If we want to maintain consistency for the
> > NewPartitions field when the group coordinator migrates between new and old
> > binaries, a version gate is definitely required. For example:
> > >
> > > GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())
> > >
> > > The group coordinator should NOT set the NewPartitions field when the gv
> > is smaller than gv_2
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> > > > Hi, Jiunn-Yang,
> > > >
> > > > JR2. We should try to maintain consistency across our APIs. So, it
> > would be
> > > > useful to understand if the user-facing part of this KIP is what share
> > > > consumer wants too. It would be problematic if we proceed with this
> > > > approach only for the share consumer to choose a different one later.
> > > >
> > > > JR3. I was thinking that if we add the NewPartitions field in
> > > > GroupHeartbeatResponse, there is no need to expose the group/partition
> > > > creation timestamp to other RPCs and the Cli.
> > > >
> > > > JR5. Since this changes the on-disk data format for PartitionRecord, we
> > > > need to gate this by a new MV. It also changes the format of
> > > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if
> > this
> > > > requires a new MV or a new GV.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >
> > > > > Hello Jun, chia,
> > > > >
> > > > > Thanks for your feedback.
> > > > >
> > > > > JR3 JR4 I have updated the KIP.
> > > > >
> > > > > JR2 The new-partition detection mechanism is reusable. The
> > coordinator
> > > > > already has
> > > > > access to both timestamps required for classification, so the
> > > > > determination can be made
> > > > > server-side without introducing any new RPCs. The main difference is
> > where
> > > > > the policy is
> > > > > enforced: modern consumer resolves it on the client, whereas a share
> > group
> > > > > resolves the
> > > > > starting offset on the partition leader. As a result, the
> > classification
> > > > > decision would need to
> > > > > be propagated through the existing share-state initialization path.
> > We
> > > > > would also need to persist
> > > > > the group's creation time as part of the share-group state, since
> > that
> > > > > information is not stored today.
> > > > >
> > > > > None of this is a blocker, but it does require changes to the
> > share-state
> > > > > format and introduces
> > > > > a share-specific configuration. Given that additional scope, I would
> > > > > prefer to keep share groups out
> > > > > of scope for this KIP and address them in a follow-up effort, while
> > > > > documenting the relevant reuse
> > > > > opportunities here.
> > > > >
> > > > > Best Regards,
> > > > > Jiunn-Yang
> > > > >
> > > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > > > > >
> > > > > >> It inspires me to think about mirror clusters. What if the mirror
> > group
> > > > > is created 'after' the expanded partitions? In this scenario, the new
> > > > > configuration 'max.age.ms' could handle it if users are aware of the
> > > > > migration and set a suitable time interval. Of course, this would
> > require
> > > > > both creation times to be exposed to clients.
> > > > > >
> > > > > > Thinking more about the mirror cluster scenario we discussed, I
> > realize
> > > > > it introduces significant complexity. Handling it properly would
> > require a
> > > > > tolerance time window rather than a strict time interval to account
> > for
> > > > > replication lag. To prevent scope creep and keep KIP-1327 focused on
> > its
> > > > > primary goal, I think it would be best to defer the
> > multi-cluster/mirroring
> > > > > scenario and explore those approaches in a separate KIP
> > > > > >
> > > > > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > > > > >> hi all,
> > > > > >>
> > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > client.
> > > > > I am
> > > > > >>> not sure if there are other use cases for those timestamps. An
> > > > > alternative
> > > > > >>> is to avoid exposing those timestamps to the client. The group
> > > > > coordinator
> > > > > >>> will categorize the partitions using those timestamps and
> > include the
> > > > > >>> categorization in the assigned partitions in
> > GroupHeartbeatResponse.
> > > > > >>
> > > > > >> It inspires me to think about mirror clusters. What if the mirror
> > group
> > > > > is created 'after' the expanded partitions? In this scenario, the new
> > > > > configuration 'max.age.ms' could handle it if users are aware of the
> > > > > migration and set a suitable time interval. Of course, this would
> > require
> > > > > both creation times to be exposed to clients.
> > > > > >>
> > > > > >> Best,
> > > > > >> Chia-Ping
> > > > > >>
> > > > > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > > > > >>> Hi, Jiunn-Yang,
> > > > > >>>
> > > > > >>> Thanks for the updated KIP.
> > > > > >>>
> > > > > >>> JR2. Share consumer (KIP-932) states: "If the number of
> > partitions is
> > > > > >>> increased for a topic with a subscription in a share group, the
> > SPSO
> > > > > for
> > > > > >>> the newly created share-partitions is initialized to 0 ".
> > However, this
> > > > > >>> part has not been implemented yet. Currently, all partitions
> > without
> > > > > offset
> > > > > >>> follows the strategy given by share.auto.offset.reset. So, we
> > have an
> > > > > >>> opportunity to see if it's better to align this KIP's changes
> > with the
> > > > > >>> share consumer.
> > > > > >>> JR2.1 The first part is whether customizing the offset for new
> > > > > partitions
> > > > > >>> to something other than 0 is useful in the share consumer. It
> > seems
> > > > > that
> > > > > >>> the motivation in this KIP applies to share consumer too.
> > > > > >>> JR2.2 The second part is how new partitions are detected. The
> > mechanism
> > > > > >>> introduced in this KIP seems quite effective. A group
> > coordinator could
> > > > > >>> pick up newly created partitions for an existing topic or newly
> > created
> > > > > >>> topics matching the regex. Those partitions are categorized as
> > new
> > > > > >>> partitions and will follow the strategy given by
> > > > > >>> auto.offset.reset.new.partitions.
> > > > > >>> This makes sense since those partitions typically don't have a
> > large
> > > > > >>> backlog. A consumer could also additionally subscribe to an
> > existing
> > > > > topic
> > > > > >>> in an existing group. Those partitions will be categorized as
> > existing
> > > > > >>> partitions. This also makes sense since those partitions could
> > have a
> > > > > large
> > > > > >>> backlog and it is better to follow the strategy given by
> > > > > auto.offset.reset. It
> > > > > >>> will be useful to think through whether the logic for
> > categorizing new
> > > > > >>> partitions should also be reused in the share consumer.
> > > > > >>>
> > > > > >>> JR3. The KIP exposes the group/partition creation time to the
> > client.
> > > > > I am
> > > > > >>> not sure if there are other use cases for those timestamps. An
> > > > > alternative
> > > > > >>> is to avoid exposing those timestamps to the client. The group
> > > > > coordinator
> > > > > >>> will categorize the partitions using those timestamps and
> > include the
> > > > > >>> categorization in the assigned partitions in
> > GroupHeartbeatResponse.
> > > > > >>>
> > > > > >>> JR4. It would be useful to support this feature on existing
> > groups as
> > > > > well.
> > > > > >>> For example, any partition without a creation timestamp can
> > always be
> > > > > >>> categorized as an existing partition.
> > > > > >>>
> > > > > >>> Jun
> > > > > >>>
> > > > > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > > >>>
> > > > > >>>> Hello Andrew,
> > > > > >>>>
> > > > > >>>> Thanks for your feedback,
> > > > > >>>>>> I've just been re-reading this KIP to ensure that share group
> > > > > support is
> > > > > >>>>>> not needed. I've discovered that we made a mistake
> > implementing
> > > > > KIP-932
> > > > > >>>> in
> > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > really tight
> > > > > >>>> enough
> > > > > >>>>>> not to require KIP-1327.
> > > > > >>>>
> > > > > >>>> Could you walk us through the specific area where the mistake
> > was
> > > > > found?
> > > > > >>>> That would
> > > > > >>>> help us assess whether it naturally fits within the scope of
> > KIP-1327
> > > > > or
> > > > > >>>> should be addressed
> > > > > >>>> separately.
> > > > > >>>>
> > > > > >>>> AS1 I have updated the KIP
> > > > > >>>>
> > > > > >>>> Best Regards,
> > > > > >>>> Jiunn-Yang
> > > > > >>>>
> > > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> > > > > >>>>>
> > > > > >>>>> hi Andrew
> > > > > >>>>>
> > > > > >>>>>> I've discovered that we made a mistake implementing KIP-932
> > in this
> > > > > >>>> area,
> > > > > >>>>> so we'll rectify that and check whether it's really tight
> > enough not
> > > > > to
> > > > > >>>>> require KIP-1327.
> > > > > >>>>>
> > > > > >>>>> Would you mind sharing more details about that? If it's not
> > anything
> > > > > >>>> close
> > > > > >>>>> to NP-hard, we could definitely look into covering it within
> > KIP-1327
> > > > > >>>>>
> > > > > >>>>> Best,
> > > > > >>>>> Chia-Ping
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三
> > 下午7:49寫道:
> > > > > >>>>>
> > > > > >>>>>> Hi Jiunn-Yang,
> > > > > >>>>>> I've just been re-reading this KIP to ensure that share group
> > > > > support is
> > > > > >>>>>> not needed. I've discovered that we made a mistake
> > implementing
> > > > > KIP-932
> > > > > >>>> in
> > > > > >>>>>> this area, so we'll rectify that and check whether it's
> > really tight
> > > > > >>>> enough
> > > > > >>>>>> not to require KIP-1327.
> > > > > >>>>>>
> > > > > >>>>>> One trivial comments for consistency.
> > > > > >>>>>>
> > > > > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > > > > represented by
> > > > > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>> Andrew
> > > > > >>>>>>
> > > > > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > > > > >>>>>>> Hi all,
> > > > > >>>>>>>
> > > > > >>>>>>> Manually bumping this thread.
> > > > > >>>>>>>
> > > > > >>>>>>> Best Regards,
> > > > > >>>>>>> Jiunn-Yang
> > > > > >>>>>>>
> > > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Hello chia,
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Best Regards,
> > > > > >>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47
> > 寫道:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> hi Jiunn-Yang
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> When the config is set on a cluster that has not yet been
> > > > > >>>>>> upgraded... classification cannot occur... the consumer falls
> > back
> > > > > to
> > > > > >>>> the
> > > > > >>>>>> base auto.offset.reset for the affected partitions. No
> > exception is
> > > > > >>>> thrown,
> > > > > >>>>>> and no operational disruption results.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Existing group can't take advantage of this excellent new
> > > > > >>>>>> configuration. Allowing users to modify the group creation
> > time
> > > > > might be
> > > > > >>>>>> overkill. Instead, we could print a useful warning message to
> > guide
> > > > > >>>> users.
> > > > > >>>>>> For example, we can suggest that they re-create the group
> > with their
> > > > > >>>>>> existing committed offsets
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Protocol changes
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Would you mind listing those RPC changes in a table format?
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> The full interaction matrix between the base policy and
> > the
> > > > > >>>>>> new-partition policy is:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Please add a filed to describe the target scenario when
> > using
> > > > > these
> > > > > >>>>>> policies
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Best,
> > > > > >>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > > > > >>>>>>>>>> Hello Jun, chia,
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for the
> > new
> > > > > >>>>>>>>>> approach, PTAL
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23
> > 寫道:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> hi Jun
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Yes, your approach is great. I think the combination of
> > latest
> > > > > (for
> > > > > >>>>>> existing partitions) and by_duration (for new partitions) can
> > > > > address
> > > > > >>>> 99%
> > > > > >>>>>> of the complaints I have heard regarding this issue.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Also, leveraging the group creation time here opens the
> > door to
> > > > > >>>>>> implementing a new policy based on timestamp seek in the
> > future,
> > > > > should
> > > > > >>>> the
> > > > > >>>>>> community want to pursue that.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks for your patience and constructive feedback. We
> > will
> > > > > update
> > > > > >>>>>> the KIP accordingly.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> Best, Chia-Ping
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日
> > 清晨5:11
> > > > > 寫道:
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Hi, Chia-Ping,
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Thanks for the reply.
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > > > > configure a
> > > > > >>>>>> different
> > > > > >>>>>>>>>>>> offset policy for existing partitions vs new partitions.
> > > > > However,
> > > > > >>>>>> using
> > > > > >>>>>>>>>>>> group creation time to capture that seems more
> > intuitive.
> > > > > Here is
> > > > > >>>>>> another
> > > > > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and
> > categorize
> > > > > new
> > > > > >>>>>> partitions
> > > > > >>>>>>>>>>>> based on group creation time. Introduce
> > > > > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose
> > values
> > > > > can be
> > > > > >>>>>> earliest,
> > > > > >>>>>>>>>>>> latest and by_duration, the same as auto.offset.reset.
> > Users
> > > > > can
> > > > > >>>>>> set
> > > > > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if
> > they
> > > > > want to
> > > > > >>>>>> guarantee
> > > > > >>>>>>>>>>>> no data loss on new partitions. They can also use
> > by_duration
> > > > > to
> > > > > >>>>>> set an
> > > > > >>>>>>>>>>>> upper bound on the backlog replayed, which can be
> > different
> > > > > from
> > > > > >>>>>> that of
> > > > > >>>>>>>>>>>> the existing partitions. This will address your concern
> > about
> > > > > too
> > > > > >>>>>> much
> > > > > >>>>>>>>>>>> backlog being replayed when the offsets are lost. What
> > do you
> > > > > >>>>>> think?
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > > > > >>>>>> chia7712@gmail.com> wrote:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> hi Jun
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> The most important part of this story is how users
> > should
> > > > > expect
> > > > > >>>>>> the data
> > > > > >>>>>>>>>>>>> they can see when using the latest or by_duration
> > policy with
> > > > > >>>>>> expanded
> > > > > >>>>>>>>>>>>> partitions.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss,
> > but it is
> > > > > >>>>>>>>>>>>> non-deterministic, which means users will either read
> > too
> > > > > many
> > > > > >>>>>> historical
> > > > > >>>>>>>>>>>>> records from existing partitions or lose some records
> > from
> > > > > >>>>>> expanded
> > > > > >>>>>>>>>>>>> partitions.
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > > > > >>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>>> is a bit hard to understand, and that is why I
> > preferred
> > > > > having a
> > > > > >>>>>> whole new
> > > > > >>>>>>>>>>>>> policy based entirely on group creation time (KIP-1282)
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > > > > 上午1:08寫道:
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand
> > the
> > > > > value
> > > > > >>>>>> of the new
> > > > > >>>>>>>>>>>>>> configs with the KIP.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want
> > to
> > > > > miss
> > > > > >>>>>> the data if
> > > > > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing
> > partition
> > > > > is
> > > > > >>>>>> easy to
> > > > > >>>>>>>>>>>>>> understand because it relates to retention time. The
> > > > > backlog for
> > > > > >>>>>> the new
> > > > > >>>>>>>>>>>>>> partition is a bit subtle to understand since it
> > depends on
> > > > > the
> > > > > >>>>>> metadata
> > > > > >>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > >>>>>>> ,
> > > > > >>>>>>>>>>>>>> the user needs to
> > > > > >>>>>>>>>>>>>> understand the metadata refresh delay on the consumer
> > side
> > > > > and
> > > > > >>>>>> use it to
> > > > > >>>>>>>>>>>>>> set the config.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the same
> > value
> > > > > for
> > > > > >>>>>> the
> > > > > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three
> > issues with
> > > > > >>>> this
> > > > > >>>>>>>>>>>>>> approach.
> > > > > >>>>>>>>>>>>>> 1. It computes the seek target client-side as now() -
> > > > > duration,
> > > > > >>>>>> which
> > > > > >>>>>>>>>>>>>> introduces clock skew across consumers and forces
> > operators
> > > > > to
> > > > > >>>>>> choose
> > > > > >>>>>>>>>>>>>> overly large durations, causing unnecessary
> > reprocessing.
> > > > > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry,
> > so
> > > > > failed
> > > > > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target
> > forward and
> > > > > >>>>>> potentially
> > > > > >>>>>>>>>>>>>> miss records produced between attempts.
> > > > > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without
> > committed
> > > > > >>>>>> offsets, and
> > > > > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > > > > long-existing
> > > > > >>>>>> partitions
> > > > > >>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary
> > replay.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by
> > adding
> > > > > a bit
> > > > > >>>>>> buffer to
> > > > > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > > > > improving the
> > > > > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh
> > delay is
> > > > > >>>>>> typically low
> > > > > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer
> > and tens
> > > > > of
> > > > > >>>>>> seconds
> > > > > >>>>>>>>>>>>>> with
> > > > > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading that
> > much
> > > > > >>>>>> backlog for new
> > > > > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the same
> > for
> > > > > existing
> > > > > >>>>>>>>>>>>>> partitions.
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> So, instead of introducing a new config, could we just
> > > > > reuse the
> > > > > >>>>>> existing
> > > > > >>>>>>>>>>>>>> config with better documentation and/or
> > implementation?
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <
> > s7133700@gmail.com>
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Hello Jun,
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> You're right that group creation time is the more
> > intuitive
> > > > > >>>>>> answer at
> > > > > >>>>>>>>>>>>>>> first glance,
> > > > > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that
> > > > > "predate
> > > > > >>>>>> the group"
> > > > > >>>>>>>>>>>>>>> vs partitions
> > > > > >>>>>>>>>>>>>>> "created during group runtime," which directly
> > points to a
> > > > > >>>>>>>>>>>>>> group-lifecycle
> > > > > >>>>>>>>>>>>>>> classifier.
> > > > > >>>>>>>>>>>>>>> I'd like to walk through why we landed on partition
> > age,
> > > > > and
> > > > > >>>> the
> > > > > >>>>>>>>>>>>>>> trade-offs we considered.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> This covers the metadata blindness window, but has
> > issues
> > > > > the
> > > > > >>>>>> KIP
> > > > > >>>>>>>>>>>>>>> currently documents
> > > > > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock
> > skew
> > > > > across
> > > > > >>>>>> consumers.
> > > > > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > > > potentially
> > > > > >>>>>> missing
> > > > > >>>>>>>>>>>>>>> records produced between
> > > > > >>>>>>>>>>>>>>> attempts.
> > > > > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without
> > committed
> > > > > >>>>>> offsets,
> > > > > >>>>>>>>>>>>>>> including pre-existing partitions
> > > > > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary
> > replay.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively
> > running.
> > > > > Our
> > > > > >>>>>> concern
> > > > > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> T=0:         Group created.
> > > > > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected,
> > etc.).
> > > > > >>>>>>>>>>>>>>> T=50:        Partition added during the idle window.
> > > > > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Under group creation time, the new partition is
> > classified
> > > > > as
> > > > > >>>>>> new
> > > > > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying
> > everything
> > > > > from
> > > > > >>>>>> T=50.
> > > > > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also
> > accumulated
> > > > > >>>>>> data that
> > > > > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the
> > > > > contract
> > > > > >>>> of
> > > > > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled
> > reason
> > > > > to
> > > > > >>>>>> treat
> > > > > >>>>>>>>>>>>>>> the new partition differently; both contain backlog
> > > > > accumulated
> > > > > >>>>>> during
> > > > > >>>>>>>>>>>>>>> the same idle window.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle
> > you
> > > > > raised
> > > > > >>>>>> in
> > > > > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated
> > some
> > > > > backlog
> > > > > >>>>>> on
> > > > > >>>>>>>>>>>>>>> every other partition during the same idle period;
> > forcing
> > > > > >>>>>> 0-backlog
> > > > > >>>>>>>>>>>>>>> tolerance only on new partitions would be
> > inconsistent with
> > > > > >>>> that
> > > > > >>>>>>>>>>>>>>> tolerance.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent data
> > loss
> > > > > >>>> window,
> > > > > >>>>>>>>>>>>>>> the gap between partition creation and the consumer’s
> > > > > metadata
> > > > > >>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely
> > > > > silent: the
> > > > > >>>>>>>>>>>>>>> consumer had no opportunity to know about the
> > partition.
> > > > > >>>>>> Outside this
> > > > > >>>>>>>>>>>>>>> window, missing data reflects either:
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> > > > > consumers,
> > > > > >>>>>> or
> > > > > >>>>>>>>>>>>>>> - (b) an operational issue to surface via
> > monitoring, not
> > > > > via
> > > > > >>>>>> reset
> > > > > >>>>>>>>>>>>>> policy.
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> We did not choose partition age because it is more
> > elegant
> > > > > than
> > > > > >>>>>> group
> > > > > >>>>>>>>>>>>>>> creation time — we chose it because its failure mode
> > > > > (requires
> > > > > >>>> a
> > > > > >>>>>>>>>>>>>>> threshold) is
> > > > > >>>>>>>>>>>>>>> less invasive than the failure mode of group
> > creation time
> > > > > >>>>>> (overrides
> > > > > >>>>>>>>>>>>>>> user-stated
> > > > > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日
> > 上午11:52
> > > > > 寫道:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Hi Jun,
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Relying on both creation times will create an
> > inconsistent
> > > > > >>>>>> scenario. A
> > > > > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep
> > will
> > > > > seek
> > > > > >>>>>> to the
> > > > > >>>>>>>>>>>>>>>> beginning for the partitions created later than the
> > group.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix
> > the
> > > > > >>>>>> inconsistency
> > > > > >>>>>>>>>>>>>>> using a
> > > > > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> > > > > consensus,
> > > > > >>>>>> KIP-1327
> > > > > >>>>>>>>>>>>>>> goes
> > > > > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent
> > users
> > > > > from
> > > > > >>>>>> falling
> > > > > >>>>>>>>>>>>>> into
> > > > > >>>>>>>>>>>>>>>> that pitfall.
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > 2026年6月13日週六
> > > > > >>>>>> 上午6:49寫道:
> > > > > >>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>>>> still feels weird to
> > > > > >>>>>>>>>>>>>>> me.
> > > > > >>>>>>>>>>>>>>>>> It
> > > > > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on
> > the
> > > > > >>>>>> partition
> > > > > >>>>>>>>>>>>>>> creation
> > > > > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be
> > based on
> > > > > the
> > > > > >>>>>> group
> > > > > >>>>>>>>>>>>>>> creation
> > > > > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is
> > created
> > > > > are
> > > > > >>>>>> existing
> > > > > >>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>> all partitions created after the group creation
> > are new
> > > > > >>>>>> partitions.
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <
> > s7133700@gmail.com>
> > > > > >>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Hi all,
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no
> > further
> > > > > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common
> > operational
> > > > > task,
> > > > > >>>>>> not an
> > > > > >>>>>>>>>>>>>> edge
> > > > > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section
> > accordingly.
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日
> > 下午5:49 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful
> > to
> > > > > clarify
> > > > > >>>>>> that
> > > > > >>>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>>> is a
> > > > > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which
> > > > > further
> > > > > >>>>>>>>>>>>>> demonstrates
> > > > > >>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > > > > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational
> > task in
> > > > > >>>>>> Kafka: To
> > > > > >>>>>>>>>>>>>>>>> balance
> > > > > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are
> > typically
> > > > > >>>>>> created with a
> > > > > >>>>>>>>>>>>>>>>>> moderate
> > > > > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic
> > grows
> > > > > over
> > > > > >>>>>> time, it
> > > > > >>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>> often
> > > > > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions
> > to
> > > > > >>>>>> accommodate the
> > > > > >>>>>>>>>>>>>>>>> higher
> > > > > >>>>>>>>>>>>>>>>>>>> workload.
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> Regards
> > > > > >>>>>>>>>>>>>>>>>>>> Jian
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > > > >>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the
> > KIP!
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > 2026年5月30日
> > > > > 晚上8:29
> > > > > >>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and
> > "cold"
> > > > > when
> > > > > >>>>>>>>>>>>>> describing
> > > > > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are
> > using
> > > > > them
> > > > > >>>> to
> > > > > >>>>>>>>>>>>>> describe
> > > > > >>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the
> > records, but
> > > > > >>>>>> applying
> > > > > >>>>>>>>>>>>>> these
> > > > > >>>>>>>>>>>>>>>>>> terms
> > > > > >>>>>>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't
> > really care
> > > > > >>>>>> whether a
> > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation
> > is
> > > > > that
> > > > > >>>>>> they won't
> > > > > >>>>>>>>>>>>>>>>>> silently
> > > > > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic
> > during
> > > > > their
> > > > > >>>>>> active
> > > > > >>>>>>>>>>>>>>>>>>>>> consumption.
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六
> > 下午12:30寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the
> > KIP
> > > > > >>>>>> motivation
> > > > > >>>>>>>>>>>>>>> section.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > > > 2026年5月30日
> > > > > >>>>>> 凌晨1:12
> > > > > >>>>>>>>>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a
> > stronger
> > > > > >>>>>> motivation
> > > > > >>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>> KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not
> > all
> > > > > >>>>>> partitions
> > > > > >>>>>>>>>>>>>> without
> > > > > >>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly
> > expanded
> > > > > >>>>>> partition
> > > > > >>>>>>>>>>>>>> (hot)
> > > > > >>>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> > > > > consumer
> > > > > >>>>>> has
> > > > > >>>>>>>>>>>>>> never
> > > > > >>>>>>>>>>>>>>>>> seen
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why
> > is the
> > > > > hot
> > > > > >>>>>>>>>>>>>> partition
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration
> > policy is
> > > > > also
> > > > > >>>>>>>>>>>>>>> insufficient
> > > > > >>>>>>>>>>>>>>>>>>>>>>> because:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() -
> > duration)
> > > > > varies
> > > > > >>>>>> across
> > > > > >>>>>>>>>>>>>> nodes
> > > > > >>>>>>>>>>>>>>>>>> due
> > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an
> > overly
> > > > > large
> > > > > >>>>>>>>>>>>>> duration,
> > > > > >>>>>>>>>>>>>>>>>>>>>>> causing
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client
> > recalculates the
> > > > > seek
> > > > > >>>>>> time on
> > > > > >>>>>>>>>>>>>>>>> retry,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and
> > risking
> > > > > data
> > > > > >>>>>> loss."
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare.
> > If
> > > > > these
> > > > > >>>>>> issues
> > > > > >>>>>>>>>>>>>>>>> persist,
> > > > > >>>>>>>>>>>>>>>>>>>>> more
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > > > > >>>>>> situations don't
> > > > > >>>>>>>>>>>>>>>>> need a
> > > > > >>>>>>>>>>>>>>>>>>>>>>> common
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > > > situations,
> > > > > >>>>>> they can
> > > > > >>>>>>>>>>>>>>>>>> implement
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > ConsumerRebalanceListener.onPartitionsAssigned().
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > > > >>>>>> s7133700@gmail.com>
> > > > > >>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned
> > value
> > > > > >>>>>> should
> > > > > >>>>>>>>>>>>>> always
> > > > > >>>>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the
> > KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > positive
> > > > > >>>>>> any
> > > > > >>>>>>>>>>>>>>>>>> field
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect
> > the
> > > > > version
> > > > > >>>>>>>>>>>>>> mismatch
> > > > > >>>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>>>> during
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment,
> > which
> > > > > occurs
> > > > > >>>>>> inside
> > > > > >>>>>>>>>>>>>>>>> poll().
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > > > UnsupportedVersionException
> > > > > >>>>>> from
> > > > > >>>>>>>>>>>>>> poll().
> > > > > >>>>>>>>>>>>>>>>>> I’ll
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > > 2026年5月17日
> > > > > >>>>>> 下午4:50
> > > > > >>>>>>>>>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The
> > age of
> > > > > this
> > > > > >>>>>>>>>>>>>> partition
> > > > > >>>>>>>>>>>>>>>>> in
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the
> > broker
> > > > > as
> > > > > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > > > > >>>>>>>>>>>>>>>>>>>>>>> -
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the
> > broker
> > > > > >>>>>> does not
> > > > > >>>>>>>>>>>>>>>>> support
> > > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is
> > > > > unknown.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned
> > value
> > > > > >>>>>> should
> > > > > >>>>>>>>>>>>>> always
> > > > > >>>>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> > positive
> > > > > >>>>>> any
> > > > > >>>>>>>>>>>>>>>>>> field
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter
> > UnsupportedVersionException
> > > > > when
> > > > > >>>>>> calling
> > > > > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design
> > change
> > > > > based on
> > > > > >>>>>> the
> > > > > >>>>>>>>>>>>>>>>>> discussion
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the
> > new-partition
> > > > > >>>> reset
> > > > > >>>>>>>>>>>>>> behavior
> > > > > >>>>>>>>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>>>> now applies to all
> > > > > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the
> > consumer
> > > > > resets
> > > > > >>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>>>> .partitions
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the
> > base
> > > > > >>>>>>>>>>>>>> auto.offset.reset
> > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > > > represented
> > > > > >>>>>> by a
> > > > > >>>>>>>>>>>>>>> separate
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>> .partitions,
> > > > > >>>>>>>>>>>>>> currently fixed to
> > > > > >>>>>>>>>>>>>>>>> earliest).
> > > > > >>>>>>>>>>>>>>>>>>>>> This
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the
> > behavior to a
> > > > > >>>>>> public
> > > > > >>>>>>>>>>>>>>>>>> user-facing
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > > 2026年5月16日
> > > > > >>>>>> 清晨7:46
> > > > > >>>>>>>>>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal
> > from me
> > > > > is
> > > > > >>>>>> listed
> > > > > >>>>>>>>>>>>>>> below:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>> .partitions
> > > > > >>>>>>>>>>>>>> with a default value
> > > > > >>>>>>>>>>>>>>>>> of
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > > > > >>>>>> by_duration and
> > > > > >>>>>>>>>>>>>>>>> latest,
> > > > > >>>>>>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > > > >>>>>> auto.offset.reset=earliest.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>> .partitions
> > > > > >>>>>>>>>>>>>> as an internal
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>>>> .partitions=earliest
> > > > > >>>>>>>>>>>>>>> already
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss
> > the use
> > > > > cases
> > > > > >>>>>> of
> > > > > >>>>>>>>>>>>>> other
> > > > > >>>>>>>>>>>>>>>>>> values
> > > > > >>>>>>>>>>>>>>>>>>>>>>> in a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>> .partitions
> > > > > >>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > > > > >>>>>>> ,
> > > > > >>>>>>>>>>>>>> will be applied to all for
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev
> > wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP
> > says
> > > > > "When
> > > > > >>>>>> a Kafka
> > > > > >>>>>>>>>>>>>>>>> topic
> > > > > >>>>>>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using
> > the
> > > > > latest
> > > > > >>>>>> auto
> > > > > >>>>>>>>>>>>>> offset
> > > > > >>>>>>>>>>>>>>>>>> reset
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records
> > produced to
> > > > > those
> > > > > >>>>>>>>>>>>>> partitions
> > > > > >>>>>>>>>>>>>>>>>> before
> > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user
> > sets
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec,
> > the same
> > > > > >>>>>> record loss
> > > > > >>>>>>>>>>>>>>>>> issue
> > > > > >>>>>>>>>>>>>>>>>>>>> could
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking
> > auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>> .partitions
> > > > > >>>>>>>>>>>>>> will
> > > > > >>>>>>>>>>>>>>> take
> > > > > >>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>> same
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user
> > could
> > > > > set
> > > > > >>>>>> it
> > > > > >>>>>>>>>>>>>>>>>> by_duration if
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM
> > Chia-Ping
> > > > > Tsai <
> > > > > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be
> > missing
> > > > > >>>>>> something
> > > > > >>>>>>>>>>>>>>>>>> important
> > > > > >>>>>>>>>>>>>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as
> > I try
> > > > > to
> > > > > >>>>>> clarify
> > > > > >>>>>>>>>>>>>> with
> > > > > >>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>> few
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for
> > extending
> > > > > this
> > > > > >>>>>> logic
> > > > > >>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>> other
> > > > > >>>>>>>>>>>>>>>>>>>>>>> reset
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > > > > earliest
> > > > > >>>>>> or
> > > > > >>>>>>>>>>>>>>>>> by_duration
> > > > > >>>>>>>>>>>>>>>>>>>>>>> don't
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data
> > loss
> > > > > issue
> > > > > >>>>>> when a
> > > > > >>>>>>>>>>>>>>>>>> partition
> > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users
> > to
> > > > > >>>>>> configure for
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>> .partitions?
> > > > > >>>>>>>>>>>>>> If they set it to
> > > > > >>>>>>>>>>>>>>>>> earliest
> > > > > >>>>>>>>>>>>>>>>>> or
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge
> > > > > cases. For
> > > > > >>>>>>>>>>>>>> example,
> > > > > >>>>>>>>>>>>>>>>> if a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new
> > partition is
> > > > > >>>>>> created
> > > > > >>>>>>>>>>>>>> during
> > > > > >>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip
> > to
> > > > > latest
> > > > > >>>>>> when
> > > > > >>>>>>>>>>>>>>>>> resuming,
> > > > > >>>>>>>>>>>>>>>>>>>>>>> rather
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > > > > partition
> > > > > >>>>>> is
> > > > > >>>>>>>>>>>>>>>>>> technically
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > > > > introducing a
> > > > > >>>>>> max.age
> > > > > >>>>>>>>>>>>>>>>>> threshold.
> > > > > >>>>>>>>>>>>>>>>>>>>>>> It
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what
> > is
> > > > > >>>>>> genuinely
> > > > > >>>>>>>>>>>>>>> "hot/new"
> > > > > >>>>>>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't
> > seen yet.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev
> > > > > wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find
> > auto.offset.reset.latest.max.age a
> > > > > bit
> > > > > >>>>>> weird. It
> > > > > >>>>>>>>>>>>>>>>> only
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest.
> > However, it
> > > > > seems
> > > > > >>>>>> that the
> > > > > >>>>>>>>>>>>>>>>>>>>> motivation
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is
> > set to
> > > > > other
> > > > > >>>>>> values
> > > > > >>>>>>>>>>>>>> like
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a
> > > > > separate
> > > > > >>>>>> way to
> > > > > >>>>>>>>>>>>>>> control
> > > > > >>>>>>>>>>>>>>>>>>>>> newly
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions
> > when the
> > > > > >>>> group
> > > > > >>>>>>>>>>>>>> starts.
> > > > > >>>>>>>>>>>>>>>>>> Have we
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > > > auto.offset.reset.new
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>>>>> .partitions?
> > > > > >>>>>>>>>>>>>>>>>> If
> > > > > >>>>>>>>>>>>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset
> > policy
> > > > > >>>>>> defaults to
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could
> > set it
> > > > > >>>>>> explicitly
> > > > > >>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>> customize
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > > > >>>>>> s7133700@gmail.com
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this
> > thread.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於
> > 2026年5月1日
> > > > > 晚上10:37
> > > > > >>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to
> > v14.
> > > > > The
> > > > > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64,
> > default -1),
> > > > > >>>>>> computed
> > > > > >>>>>>>>>>>>>>>>>> server-side
> > > > > >>>>>>>>>>>>>>>>>>>>>>> by
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > > > >>>> partition_creation_time.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat
> > flow.
> > > > > when
> > > > > >>>>>>>>>>>>>>>>>> MembershipManager
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly
> > invalidates the
> > > > > >>>>>> metadata for
> > > > > >>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>> affected
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset
> > decision,
> > > > > even
> > > > > >>>>>> if the
> > > > > >>>>>>>>>>>>>>>>> topic
> > > > > >>>>>>>>>>>>>>>>>> ID
> > > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's
> > maximum
> > > > > >>>>>> supported
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at
> > connection
> > > > > time.
> > > > > >>>>>> If the
> > > > > >>>>>>>>>>>>>>>>>> negotiated
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > > > > >>>>>> PartitionAgeMs at
> > > > > >>>>>>>>>>>>>> all
> > > > > >>>>>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>>>>>> can
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently
> > falling
> > > > > >>>>>> back to
> > > > > >>>>>>>>>>>>>> latest
> > > > > >>>>>>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any
> > operator-visible
> > > > > signal.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in
> > the
> > > > > KIP.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > chia7712@gmail.com> 於
> > > > > >>>>>> 2026年4月29日
> > > > > >>>>>>>>>>>>>>> 下午4:04
> > > > > >>>>>>>>>>>>>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of
> > moving the
> > > > > >>>>>> 'age'
> > > > > >>>>>>>>>>>>>>>>> resolution
> > > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API
> > to keep
> > > > > the
> > > > > >>>>>> control
> > > > > >>>>>>>>>>>>>>>>> plane
> > > > > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > > > > introducing
> > > > > >>>>>>>>>>>>>>> inter-broker
> > > > > >>>>>>>>>>>>>>>>>>>>> clock
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided
> > a
> > > > > single
> > > > > >>>>>> source of
> > > > > >>>>>>>>>>>>>>>>> truth
> > > > > >>>>>>>>>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time
> > skew
> > > > > >>>>>> should be
> > > > > >>>>>>>>>>>>>>>>>> negligible.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > > > > >>>>>> configured in
> > > > > >>>>>>>>>>>>>>> minutes
> > > > > >>>>>>>>>>>>>>>>>> or
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds)
> > between
> > > > > >>>>>> brokers
> > > > > >>>>>>>>>>>>>> won't
> > > > > >>>>>>>>>>>>>>>>>> impact
> > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > > > dev@kafka.apache.org>
> > > > > >>>>>> 於
> > > > > >>>>>>>>>>>>>>>>> 2026年4月29日
> > > > > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed
> > the
> > > > > >>>>>> previous
> > > > > >>>>>>>>>>>>>>>>>> conversation
> > > > > >>>>>>>>>>>>>>>>>>>>>>> but I
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand
> > the
> > > > > flow
> > > > > >>>>>> with the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There
> > is a
> > > > > new
> > > > > >>>>>> boolean;
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>> KIP
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only
> > when
> > > > > this
> > > > > >>>>>> boolean
> > > > > >>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>> set.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer
> > receives
> > > > > a new
> > > > > >>>>>>>>>>>>>> partition,
> > > > > >>>>>>>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>>>>>>>>> will
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set
> > to
> > > > > receive
> > > > > >>>>>> the
> > > > > >>>>>>>>>>>>>> ages.
> > > > > >>>>>>>>>>>>>>> Is
> > > > > >>>>>>>>>>>>>>>>>> my
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps
> > clarify the
> > > > > flow
> > > > > >>>>>> and
> > > > > >>>>>>>>>>>>>> also
> > > > > >>>>>>>>>>>>>>>>>> explain
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > > > > offsets,
> > > > > >>>>>> fetch
> > > > > >>>>>>>>>>>>>>>>> offsets,
> > > > > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is
> > correct, I
> > > > > >>>>>> wonder if
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API
> > is the
> > > > > right
> > > > > >>>>>> place
> > > > > >>>>>>>>>>>>>> for
> > > > > >>>>>>>>>>>>>>>>>> this
> > > > > >>>>>>>>>>>>>>>>>>>>>>> given
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > > > Alternatively,
> > > > > >>>>>> it could
> > > > > >>>>>>>>>>>>>>>>> simply
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be
> > > > > rather
> > > > > >>>>>> cautious
> > > > > >>>>>>>>>>>>>>> about
> > > > > >>>>>>>>>>>>>>>>>> not
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API
> > with
> > > > > >>>>>> unrelated
> > > > > >>>>>>>>>>>>>>> concepts.
> > > > > >>>>>>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>>>>>>>> API
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning
> > or
> > > > > revoking
> > > > > >>>>>>>>>>>>>>> partitions.
> > > > > >>>>>>>>>>>>>>>>>> The
> > > > > >>>>>>>>>>>>>>>>>>>>>>> fact
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > > > corresponding
> > > > > >>>>>> Streams
> > > > > >>>>>>>>>>>>>> API
> > > > > >>>>>>>>>>>>>>>>>> also
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right.
> > What
> > > > > would we
> > > > > >>>>>> do if
> > > > > >>>>>>>>>>>>>> we
> > > > > >>>>>>>>>>>>>>>>>> want to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > > > > >>>>>> Muralidhar Basani
> > > > > >>>>>>>>>>>>>>> via
> > > > > >>>>>>>>>>>>>>>>>> dev
> > > > > >>>>>>>>>>>>>>>>>>>>> <
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip.
> > Good to
> > > > > know
> > > > > >>>>>> about
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>> gap.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version
> > bump for
> > > > > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on
> > response,
> > > > > >>>>>> PartitionAges
> > > > > >>>>>>>>>>>>>> on
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version
> > bump?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is
> > there a
> > > > > >>>>>> recommended
> > > > > >>>>>>>>>>>>>>>>> value
> > > > > >>>>>>>>>>>>>>>>>> or
> > > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should
> > based
> > > > > on the
> > > > > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > > > >>>>>>>>>>>>>> <
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > > > > >>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of
> > javadocs I
> > > > > >>>>>> guess.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no
> > > > > changes to
> > > > > >>>>>> Kafka
> > > > > >>>>>>>>>>>>>>>>> Streams,
> > > > > >>>>>>>>>>>>>>>>>>>>>>> would
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > > > >>>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users
> > configure
> > > > > it?
> > > > > >>>>>> This is
> > > > > >>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>> most
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might
> > easily
> > > > > >>>>>> mistakenly
> > > > > >>>>>>>>>>>>>>>>>> configure
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > > > > consumer
> > > > > >>>>>> falls
> > > > > >>>>>>>>>>>>>> back
> > > > > >>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > > > > >>>>>> per-partition
> > > > > >>>>>>>>>>>>>> which
> > > > > >>>>>>>>>>>>>>>>>> isn't
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > > > > "consumer
> > > > > >>>>>> resolves
> > > > > >>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>> initial
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that
> > partition" as if
> > > > > >>>>>> earliest
> > > > > >>>>>>>>>>>>>> was
> > > > > >>>>>>>>>>>>>>>>>> applied
> > > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and
> > auto.offset.reset
> > > > > >>>>>> config is
> > > > > >>>>>>>>>>>>>>>>>> unchanged.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at
> > 2:48 PM 黃竣陽 <
> > > > > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to
> > include
> > > > > this
> > > > > >>>>>> change.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> > chia7712@apache.org>
> > > > > 於
> > > > > >>>>>>>>>>>>>> 2026年4月28日
> > > > > >>>>>>>>>>>>>>>>>> 晚上8:03
> > > > > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > > > > partition
> > > > > >>>>>> creation
> > > > > >>>>>>>>>>>>>>> time
> > > > > >>>>>>>>>>>>>>>>>> via
> > > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable
> > for
> > > > > users
> > > > > >>>> to
> > > > > >>>>>>>>>>>>>> diagnose
> > > > > >>>>>>>>>>>>>>>>> and
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > > > >>>>>> auto.offset.reset.latest.max.age
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽
> > wrote:
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> > discussion
> > > > > on
> > > > > >>>>>> KIP-1327
> > > > > >>>>>>>>>>>>>>>>>> Prevent
> > > > > >>>>>>>>>>>>>>>>>>>>> Hot
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest
> > Policy
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
> > introduces
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy
> > distinguish
> > > > > newly
> > > > > >>>>>> expanded
> > > > > >>>>>>>>>>>>>>>>> (hot)
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones.
> > Partitions
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > > > > threshold
> > > > > >>>>>>>>>>>>>>> automatically
> > > > > >>>>>>>>>>>>>>>>>> fall
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent
> > data loss
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion
> > without
> > > > > forcing
> > > > > >>>>>> a full
> > > > > >>>>>>>>>>>>>>>>>> historical
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>>
> > > > > >>>>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Andrew,

Thanks for the reply.

I am not sure it's always a good idea to start new partitions with offset 0
for the share group. For example, once a share group starts, a user could
add a subscription to an existing topic. That existing topic could have a
large backlog and starting it with offset 0 forces the user to consume that
entire backlog. In KIP-1327, such a partition will be treated as an
existing partition and will follow the auto.offset.reset policy. This seems
more intuitive. It would be useful to consider if the share group should
adopt the same policy.

Jun

On Fri, Jun 26, 2026 at 3:42 AM Andrew Schofield <sc...@apache.org>
wrote:

> Hi Jiunn-Yang et al.,
> Some background on the problem I hinted at for the KIP-932 implementation.
> KIP-932 states "If the number of partitions is increased for a topic with a
> subscription in a share group, the SPSO for the newly created
> share-partitions is initialized to 0 (which is of course both the earliest
> and latest offset for an empty topic-partition). This means there is no
> doubt about what happens when the number of partitions is increased."
>
> The problem was that we had not implemented the use of 0, but instead were
> using the share.auto.offset.reset config to choose the initial offset in
> this case. We write a ShareGroupStatePartitionMetadata record when a topic
> is first being initialized for consumption by a share group, and this
> contains a list of the partitions which we are initializing. As a result, I
> believe we can already tell when partitions are added, and we do know when
> to initialize to 0 as it says in the KIP. We will put in a patch that uses
> 0 as intended.
>
> I do not mind whether we want to incorporate share groups into KIP-1327,
> but I do not think that it is necessary to do so.
>
> Thanks,
> Andrew
>
> On 2026/06/26 04:31:28 Chia-Ping Tsai wrote:
> > > JR5. Since this changes the on-disk data format for PartitionRecord, we
> > > need to gate this by a new MV. It also changes the format of
> > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if
> this
> > > requires a new MV or a new GV.
> >
> > That is a good point. If we want to maintain consistency for the
> NewPartitions field when the group coordinator migrates between new and old
> binaries, a version gate is definitely required. For example:
> >
> > GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())
> >
> > The group coordinator should NOT set the NewPartitions field when the gv
> is smaller than gv_2
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> > > Hi, Jiunn-Yang,
> > >
> > > JR2. We should try to maintain consistency across our APIs. So, it
> would be
> > > useful to understand if the user-facing part of this KIP is what share
> > > consumer wants too. It would be problematic if we proceed with this
> > > approach only for the share consumer to choose a different one later.
> > >
> > > JR3. I was thinking that if we add the NewPartitions field in
> > > GroupHeartbeatResponse, there is no need to expose the group/partition
> > > creation timestamp to other RPCs and the Cli.
> > >
> > > JR5. Since this changes the on-disk data format for PartitionRecord, we
> > > need to gate this by a new MV. It also changes the format of
> > > ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if
> this
> > > requires a new MV or a new GV.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >
> > > > Hello Jun, chia,
> > > >
> > > > Thanks for your feedback.
> > > >
> > > > JR3 JR4 I have updated the KIP.
> > > >
> > > > JR2 The new-partition detection mechanism is reusable. The
> coordinator
> > > > already has
> > > > access to both timestamps required for classification, so the
> > > > determination can be made
> > > > server-side without introducing any new RPCs. The main difference is
> where
> > > > the policy is
> > > > enforced: modern consumer resolves it on the client, whereas a share
> group
> > > > resolves the
> > > > starting offset on the partition leader. As a result, the
> classification
> > > > decision would need to
> > > > be propagated through the existing share-state initialization path.
> We
> > > > would also need to persist
> > > > the group's creation time as part of the share-group state, since
> that
> > > > information is not stored today.
> > > >
> > > > None of this is a blocker, but it does require changes to the
> share-state
> > > > format and introduces
> > > > a share-specific configuration. Given that additional scope, I would
> > > > prefer to keep share groups out
> > > > of scope for this KIP and address them in a follow-up effort, while
> > > > documenting the relevant reuse
> > > > opportunities here.
> > > >
> > > > Best Regards,
> > > > Jiunn-Yang
> > > >
> > > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > > > >
> > > > >> It inspires me to think about mirror clusters. What if the mirror
> group
> > > > is created 'after' the expanded partitions? In this scenario, the new
> > > > configuration 'max.age.ms' could handle it if users are aware of the
> > > > migration and set a suitable time interval. Of course, this would
> require
> > > > both creation times to be exposed to clients.
> > > > >
> > > > > Thinking more about the mirror cluster scenario we discussed, I
> realize
> > > > it introduces significant complexity. Handling it properly would
> require a
> > > > tolerance time window rather than a strict time interval to account
> for
> > > > replication lag. To prevent scope creep and keep KIP-1327 focused on
> its
> > > > primary goal, I think it would be best to defer the
> multi-cluster/mirroring
> > > > scenario and explore those approaches in a separate KIP
> > > > >
> > > > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > > > >> hi all,
> > > > >>
> > > > >>> JR3. The KIP exposes the group/partition creation time to the
> client.
> > > > I am
> > > > >>> not sure if there are other use cases for those timestamps. An
> > > > alternative
> > > > >>> is to avoid exposing those timestamps to the client. The group
> > > > coordinator
> > > > >>> will categorize the partitions using those timestamps and
> include the
> > > > >>> categorization in the assigned partitions in
> GroupHeartbeatResponse.
> > > > >>
> > > > >> It inspires me to think about mirror clusters. What if the mirror
> group
> > > > is created 'after' the expanded partitions? In this scenario, the new
> > > > configuration 'max.age.ms' could handle it if users are aware of the
> > > > migration and set a suitable time interval. Of course, this would
> require
> > > > both creation times to be exposed to clients.
> > > > >>
> > > > >> Best,
> > > > >> Chia-Ping
> > > > >>
> > > > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > > > >>> Hi, Jiunn-Yang,
> > > > >>>
> > > > >>> Thanks for the updated KIP.
> > > > >>>
> > > > >>> JR2. Share consumer (KIP-932) states: "If the number of
> partitions is
> > > > >>> increased for a topic with a subscription in a share group, the
> SPSO
> > > > for
> > > > >>> the newly created share-partitions is initialized to 0 ".
> However, this
> > > > >>> part has not been implemented yet. Currently, all partitions
> without
> > > > offset
> > > > >>> follows the strategy given by share.auto.offset.reset. So, we
> have an
> > > > >>> opportunity to see if it's better to align this KIP's changes
> with the
> > > > >>> share consumer.
> > > > >>> JR2.1 The first part is whether customizing the offset for new
> > > > partitions
> > > > >>> to something other than 0 is useful in the share consumer. It
> seems
> > > > that
> > > > >>> the motivation in this KIP applies to share consumer too.
> > > > >>> JR2.2 The second part is how new partitions are detected. The
> mechanism
> > > > >>> introduced in this KIP seems quite effective. A group
> coordinator could
> > > > >>> pick up newly created partitions for an existing topic or newly
> created
> > > > >>> topics matching the regex. Those partitions are categorized as
> new
> > > > >>> partitions and will follow the strategy given by
> > > > >>> auto.offset.reset.new.partitions.
> > > > >>> This makes sense since those partitions typically don't have a
> large
> > > > >>> backlog. A consumer could also additionally subscribe to an
> existing
> > > > topic
> > > > >>> in an existing group. Those partitions will be categorized as
> existing
> > > > >>> partitions. This also makes sense since those partitions could
> have a
> > > > large
> > > > >>> backlog and it is better to follow the strategy given by
> > > > auto.offset.reset. It
> > > > >>> will be useful to think through whether the logic for
> categorizing new
> > > > >>> partitions should also be reused in the share consumer.
> > > > >>>
> > > > >>> JR3. The KIP exposes the group/partition creation time to the
> client.
> > > > I am
> > > > >>> not sure if there are other use cases for those timestamps. An
> > > > alternative
> > > > >>> is to avoid exposing those timestamps to the client. The group
> > > > coordinator
> > > > >>> will categorize the partitions using those timestamps and
> include the
> > > > >>> categorization in the assigned partitions in
> GroupHeartbeatResponse.
> > > > >>>
> > > > >>> JR4. It would be useful to support this feature on existing
> groups as
> > > > well.
> > > > >>> For example, any partition without a creation timestamp can
> always be
> > > > >>> categorized as an existing partition.
> > > > >>>
> > > > >>> Jun
> > > > >>>
> > > > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >>>
> > > > >>>> Hello Andrew,
> > > > >>>>
> > > > >>>> Thanks for your feedback,
> > > > >>>>>> I've just been re-reading this KIP to ensure that share group
> > > > support is
> > > > >>>>>> not needed. I've discovered that we made a mistake
> implementing
> > > > KIP-932
> > > > >>>> in
> > > > >>>>>> this area, so we'll rectify that and check whether it's
> really tight
> > > > >>>> enough
> > > > >>>>>> not to require KIP-1327.
> > > > >>>>
> > > > >>>> Could you walk us through the specific area where the mistake
> was
> > > > found?
> > > > >>>> That would
> > > > >>>> help us assess whether it naturally fits within the scope of
> KIP-1327
> > > > or
> > > > >>>> should be addressed
> > > > >>>> separately.
> > > > >>>>
> > > > >>>> AS1 I have updated the KIP
> > > > >>>>
> > > > >>>> Best Regards,
> > > > >>>> Jiunn-Yang
> > > > >>>>
> > > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> > > > >>>>>
> > > > >>>>> hi Andrew
> > > > >>>>>
> > > > >>>>>> I've discovered that we made a mistake implementing KIP-932
> in this
> > > > >>>> area,
> > > > >>>>> so we'll rectify that and check whether it's really tight
> enough not
> > > > to
> > > > >>>>> require KIP-1327.
> > > > >>>>>
> > > > >>>>> Would you mind sharing more details about that? If it's not
> anything
> > > > >>>> close
> > > > >>>>> to NP-hard, we could definitely look into covering it within
> KIP-1327
> > > > >>>>>
> > > > >>>>> Best,
> > > > >>>>> Chia-Ping
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三
> 下午7:49寫道:
> > > > >>>>>
> > > > >>>>>> Hi Jiunn-Yang,
> > > > >>>>>> I've just been re-reading this KIP to ensure that share group
> > > > support is
> > > > >>>>>> not needed. I've discovered that we made a mistake
> implementing
> > > > KIP-932
> > > > >>>> in
> > > > >>>>>> this area, so we'll rectify that and check whether it's
> really tight
> > > > >>>> enough
> > > > >>>>>> not to require KIP-1327.
> > > > >>>>>>
> > > > >>>>>> One trivial comments for consistency.
> > > > >>>>>>
> > > > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > > > represented by
> > > > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > > > >>>>>>
> > > > >>>>>> Thanks,
> > > > >>>>>> Andrew
> > > > >>>>>>
> > > > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > > > >>>>>>> Hi all,
> > > > >>>>>>>
> > > > >>>>>>> Manually bumping this thread.
> > > > >>>>>>>
> > > > >>>>>>> Best Regards,
> > > > >>>>>>> Jiunn-Yang
> > > > >>>>>>>
> > > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > > > >>>>>>>>
> > > > >>>>>>>> Hello chia,
> > > > >>>>>>>>
> > > > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > > > >>>>>>>>
> > > > >>>>>>>> Best Regards,
> > > > >>>>>>>> Jiunn-Yang
> > > > >>>>>>>>
> > > > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47
> 寫道:
> > > > >>>>>>>>>
> > > > >>>>>>>>> hi Jiunn-Yang
> > > > >>>>>>>>>
> > > > >>>>>>>>>> When the config is set on a cluster that has not yet been
> > > > >>>>>> upgraded... classification cannot occur... the consumer falls
> back
> > > > to
> > > > >>>> the
> > > > >>>>>> base auto.offset.reset for the affected partitions. No
> exception is
> > > > >>>> thrown,
> > > > >>>>>> and no operational disruption results.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Existing group can't take advantage of this excellent new
> > > > >>>>>> configuration. Allowing users to modify the group creation
> time
> > > > might be
> > > > >>>>>> overkill. Instead, we could print a useful warning message to
> guide
> > > > >>>> users.
> > > > >>>>>> For example, we can suggest that they re-create the group
> with their
> > > > >>>>>> existing committed offsets
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Protocol changes
> > > > >>>>>>>>>
> > > > >>>>>>>>> Would you mind listing those RPC changes in a table format?
> > > > >>>>>>>>>
> > > > >>>>>>>>>> The full interaction matrix between the base policy and
> the
> > > > >>>>>> new-partition policy is:
> > > > >>>>>>>>>
> > > > >>>>>>>>> Please add a filed to describe the target scenario when
> using
> > > > these
> > > > >>>>>> policies
> > > > >>>>>>>>>
> > > > >>>>>>>>> Best,
> > > > >>>>>>>>> Chia-Ping
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > > > >>>>>>>>>> Hello Jun, chia,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for the
> new
> > > > >>>>>>>>>> approach, PTAL
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Best Regards,
> > > > >>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23
> 寫道:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> hi Jun
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Yes, your approach is great. I think the combination of
> latest
> > > > (for
> > > > >>>>>> existing partitions) and by_duration (for new partitions) can
> > > > address
> > > > >>>> 99%
> > > > >>>>>> of the complaints I have heard regarding this issue.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Also, leveraging the group creation time here opens the
> door to
> > > > >>>>>> implementing a new policy based on timestamp seek in the
> future,
> > > > should
> > > > >>>> the
> > > > >>>>>> community want to pursue that.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks for your patience and constructive feedback. We
> will
> > > > update
> > > > >>>>>> the KIP accordingly.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Best, Chia-Ping
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日
> 清晨5:11
> > > > 寫道:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi, Chia-Ping,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks for the reply.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > > > configure a
> > > > >>>>>> different
> > > > >>>>>>>>>>>> offset policy for existing partitions vs new partitions.
> > > > However,
> > > > >>>>>> using
> > > > >>>>>>>>>>>> group creation time to capture that seems more
> intuitive.
> > > > Here is
> > > > >>>>>> another
> > > > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and
> categorize
> > > > new
> > > > >>>>>> partitions
> > > > >>>>>>>>>>>> based on group creation time. Introduce
> > > > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose
> values
> > > > can be
> > > > >>>>>> earliest,
> > > > >>>>>>>>>>>> latest and by_duration, the same as auto.offset.reset.
> Users
> > > > can
> > > > >>>>>> set
> > > > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if
> they
> > > > want to
> > > > >>>>>> guarantee
> > > > >>>>>>>>>>>> no data loss on new partitions. They can also use
> by_duration
> > > > to
> > > > >>>>>> set an
> > > > >>>>>>>>>>>> upper bound on the backlog replayed, which can be
> different
> > > > from
> > > > >>>>>> that of
> > > > >>>>>>>>>>>> the existing partitions. This will address your concern
> about
> > > > too
> > > > >>>>>> much
> > > > >>>>>>>>>>>> backlog being replayed when the offsets are lost. What
> do you
> > > > >>>>>> think?
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > > > >>>>>> chia7712@gmail.com> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> hi Jun
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> The most important part of this story is how users
> should
> > > > expect
> > > > >>>>>> the data
> > > > >>>>>>>>>>>>> they can see when using the latest or by_duration
> policy with
> > > > >>>>>> expanded
> > > > >>>>>>>>>>>>> partitions.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss,
> but it is
> > > > >>>>>>>>>>>>> non-deterministic, which means users will either read
> too
> > > > many
> > > > >>>>>> historical
> > > > >>>>>>>>>>>>> records from existing partitions or lose some records
> from
> > > > >>>>>> expanded
> > > > >>>>>>>>>>>>> partitions.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > > > >>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > >>>>>>>
> > > > >>>>>>>>>>>>> is a bit hard to understand, and that is why I
> preferred
> > > > having a
> > > > >>>>>> whole new
> > > > >>>>>>>>>>>>> policy based entirely on group creation time (KIP-1282)
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > > > 上午1:08寫道:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand
> the
> > > > value
> > > > >>>>>> of the new
> > > > >>>>>>>>>>>>>> configs with the KIP.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want
> to
> > > > miss
> > > > >>>>>> the data if
> > > > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing
> partition
> > > > is
> > > > >>>>>> easy to
> > > > >>>>>>>>>>>>>> understand because it relates to retention time. The
> > > > backlog for
> > > > >>>>>> the new
> > > > >>>>>>>>>>>>>> partition is a bit subtle to understand since it
> depends on
> > > > the
> > > > >>>>>> metadata
> > > > >>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > >>>>>>> ,
> > > > >>>>>>>>>>>>>> the user needs to
> > > > >>>>>>>>>>>>>> understand the metadata refresh delay on the consumer
> side
> > > > and
> > > > >>>>>> use it to
> > > > >>>>>>>>>>>>>> set the config.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the same
> value
> > > > for
> > > > >>>>>> the
> > > > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three
> issues with
> > > > >>>> this
> > > > >>>>>>>>>>>>>> approach.
> > > > >>>>>>>>>>>>>> 1. It computes the seek target client-side as now() -
> > > > duration,
> > > > >>>>>> which
> > > > >>>>>>>>>>>>>> introduces clock skew across consumers and forces
> operators
> > > > to
> > > > >>>>>> choose
> > > > >>>>>>>>>>>>>> overly large durations, causing unnecessary
> reprocessing.
> > > > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry,
> so
> > > > failed
> > > > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target
> forward and
> > > > >>>>>> potentially
> > > > >>>>>>>>>>>>>> miss records produced between attempts.
> > > > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without
> committed
> > > > >>>>>> offsets, and
> > > > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > > > long-existing
> > > > >>>>>> partitions
> > > > >>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary
> replay.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by
> adding
> > > > a bit
> > > > >>>>>> buffer to
> > > > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > > > improving the
> > > > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh
> delay is
> > > > >>>>>> typically low
> > > > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer
> and tens
> > > > of
> > > > >>>>>> seconds
> > > > >>>>>>>>>>>>>> with
> > > > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading that
> much
> > > > >>>>>> backlog for new
> > > > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the same
> for
> > > > existing
> > > > >>>>>>>>>>>>>> partitions.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> So, instead of introducing a new config, could we just
> > > > reuse the
> > > > >>>>>> existing
> > > > >>>>>>>>>>>>>> config with better documentation and/or
> implementation?
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <
> s7133700@gmail.com>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Hello Jun,
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> You're right that group creation time is the more
> intuitive
> > > > >>>>>> answer at
> > > > >>>>>>>>>>>>>>> first glance,
> > > > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that
> > > > "predate
> > > > >>>>>> the group"
> > > > >>>>>>>>>>>>>>> vs partitions
> > > > >>>>>>>>>>>>>>> "created during group runtime," which directly
> points to a
> > > > >>>>>>>>>>>>>> group-lifecycle
> > > > >>>>>>>>>>>>>>> classifier.
> > > > >>>>>>>>>>>>>>> I'd like to walk through why we landed on partition
> age,
> > > > and
> > > > >>>> the
> > > > >>>>>>>>>>>>>>> trade-offs we considered.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> This covers the metadata blindness window, but has
> issues
> > > > the
> > > > >>>>>> KIP
> > > > >>>>>>>>>>>>>>> currently documents
> > > > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock
> skew
> > > > across
> > > > >>>>>> consumers.
> > > > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > > potentially
> > > > >>>>>> missing
> > > > >>>>>>>>>>>>>>> records produced between
> > > > >>>>>>>>>>>>>>> attempts.
> > > > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without
> committed
> > > > >>>>>> offsets,
> > > > >>>>>>>>>>>>>>> including pre-existing partitions
> > > > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary
> replay.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively
> running.
> > > > Our
> > > > >>>>>> concern
> > > > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> T=0:         Group created.
> > > > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected,
> etc.).
> > > > >>>>>>>>>>>>>>> T=50:        Partition added during the idle window.
> > > > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Under group creation time, the new partition is
> classified
> > > > as
> > > > >>>>>> new
> > > > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying
> everything
> > > > from
> > > > >>>>>> T=50.
> > > > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also
> accumulated
> > > > >>>>>> data that
> > > > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the
> > > > contract
> > > > >>>> of
> > > > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled
> reason
> > > > to
> > > > >>>>>> treat
> > > > >>>>>>>>>>>>>>> the new partition differently; both contain backlog
> > > > accumulated
> > > > >>>>>> during
> > > > >>>>>>>>>>>>>>> the same idle window.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle
> you
> > > > raised
> > > > >>>>>> in
> > > > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated
> some
> > > > backlog
> > > > >>>>>> on
> > > > >>>>>>>>>>>>>>> every other partition during the same idle period;
> forcing
> > > > >>>>>> 0-backlog
> > > > >>>>>>>>>>>>>>> tolerance only on new partitions would be
> inconsistent with
> > > > >>>> that
> > > > >>>>>>>>>>>>>>> tolerance.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent data
> loss
> > > > >>>> window,
> > > > >>>>>>>>>>>>>>> the gap between partition creation and the consumer’s
> > > > metadata
> > > > >>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely
> > > > silent: the
> > > > >>>>>>>>>>>>>>> consumer had no opportunity to know about the
> partition.
> > > > >>>>>> Outside this
> > > > >>>>>>>>>>>>>>> window, missing data reflects either:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> > > > consumers,
> > > > >>>>>> or
> > > > >>>>>>>>>>>>>>> - (b) an operational issue to surface via
> monitoring, not
> > > > via
> > > > >>>>>> reset
> > > > >>>>>>>>>>>>>> policy.
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> We did not choose partition age because it is more
> elegant
> > > > than
> > > > >>>>>> group
> > > > >>>>>>>>>>>>>>> creation time — we chose it because its failure mode
> > > > (requires
> > > > >>>> a
> > > > >>>>>>>>>>>>>>> threshold) is
> > > > >>>>>>>>>>>>>>> less invasive than the failure mode of group
> creation time
> > > > >>>>>> (overrides
> > > > >>>>>>>>>>>>>>> user-stated
> > > > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日
> 上午11:52
> > > > 寫道:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Hi Jun,
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Relying on both creation times will create an
> inconsistent
> > > > >>>>>> scenario. A
> > > > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep
> will
> > > > seek
> > > > >>>>>> to the
> > > > >>>>>>>>>>>>>>>> beginning for the partitions created later than the
> group.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix
> the
> > > > >>>>>> inconsistency
> > > > >>>>>>>>>>>>>>> using a
> > > > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> > > > consensus,
> > > > >>>>>> KIP-1327
> > > > >>>>>>>>>>>>>>> goes
> > > > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent
> users
> > > > from
> > > > >>>>>> falling
> > > > >>>>>>>>>>>>>> into
> > > > >>>>>>>>>>>>>>>> that pitfall.
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> 2026年6月13日週六
> > > > >>>>>> 上午6:49寫道:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > >>>>>>>
> > > > >>>>>>>>>>>>>> still feels weird to
> > > > >>>>>>>>>>>>>>> me.
> > > > >>>>>>>>>>>>>>>>> It
> > > > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on
> the
> > > > >>>>>> partition
> > > > >>>>>>>>>>>>>>> creation
> > > > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be
> based on
> > > > the
> > > > >>>>>> group
> > > > >>>>>>>>>>>>>>> creation
> > > > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is
> created
> > > > are
> > > > >>>>>> existing
> > > > >>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>> all partitions created after the group creation
> are new
> > > > >>>>>> partitions.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <
> s7133700@gmail.com>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no
> further
> > > > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common
> operational
> > > > task,
> > > > >>>>>> not an
> > > > >>>>>>>>>>>>>> edge
> > > > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section
> accordingly.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日
> 下午5:49 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful
> to
> > > > clarify
> > > > >>>>>> that
> > > > >>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>> is a
> > > > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which
> > > > further
> > > > >>>>>>>>>>>>>> demonstrates
> > > > >>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > > > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational
> task in
> > > > >>>>>> Kafka: To
> > > > >>>>>>>>>>>>>>>>> balance
> > > > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are
> typically
> > > > >>>>>> created with a
> > > > >>>>>>>>>>>>>>>>>> moderate
> > > > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic
> grows
> > > > over
> > > > >>>>>> time, it
> > > > >>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>> often
> > > > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions
> to
> > > > >>>>>> accommodate the
> > > > >>>>>>>>>>>>>>>>> higher
> > > > >>>>>>>>>>>>>>>>>>>> workload.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Regards
> > > > >>>>>>>>>>>>>>>>>>>> Jian
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the
> KIP!
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> 2026年5月30日
> > > > 晚上8:29
> > > > >>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and
> "cold"
> > > > when
> > > > >>>>>>>>>>>>>> describing
> > > > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are
> using
> > > > them
> > > > >>>> to
> > > > >>>>>>>>>>>>>> describe
> > > > >>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the
> records, but
> > > > >>>>>> applying
> > > > >>>>>>>>>>>>>> these
> > > > >>>>>>>>>>>>>>>>>> terms
> > > > >>>>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't
> really care
> > > > >>>>>> whether a
> > > > >>>>>>>>>>>>>>>>>> partition
> > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation
> is
> > > > that
> > > > >>>>>> they won't
> > > > >>>>>>>>>>>>>>>>>> silently
> > > > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic
> during
> > > > their
> > > > >>>>>> active
> > > > >>>>>>>>>>>>>>>>>>>>> consumption.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六
> 下午12:30寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the
> KIP
> > > > >>>>>> motivation
> > > > >>>>>>>>>>>>>>> section.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > > 2026年5月30日
> > > > >>>>>> 凌晨1:12
> > > > >>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a
> stronger
> > > > >>>>>> motivation
> > > > >>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>> KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not
> all
> > > > >>>>>> partitions
> > > > >>>>>>>>>>>>>> without
> > > > >>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly
> expanded
> > > > >>>>>> partition
> > > > >>>>>>>>>>>>>> (hot)
> > > > >>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> > > > consumer
> > > > >>>>>> has
> > > > >>>>>>>>>>>>>> never
> > > > >>>>>>>>>>>>>>>>> seen
> > > > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why
> is the
> > > > hot
> > > > >>>>>>>>>>>>>> partition
> > > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration
> policy is
> > > > also
> > > > >>>>>>>>>>>>>>> insufficient
> > > > >>>>>>>>>>>>>>>>>>>>>>> because:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() -
> duration)
> > > > varies
> > > > >>>>>> across
> > > > >>>>>>>>>>>>>> nodes
> > > > >>>>>>>>>>>>>>>>>> due
> > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an
> overly
> > > > large
> > > > >>>>>>>>>>>>>> duration,
> > > > >>>>>>>>>>>>>>>>>>>>>>> causing
> > > > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client
> recalculates the
> > > > seek
> > > > >>>>>> time on
> > > > >>>>>>>>>>>>>>>>> retry,
> > > > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and
> risking
> > > > data
> > > > >>>>>> loss."
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare.
> If
> > > > these
> > > > >>>>>> issues
> > > > >>>>>>>>>>>>>>>>> persist,
> > > > >>>>>>>>>>>>>>>>>>>>> more
> > > > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > > > >>>>>> situations don't
> > > > >>>>>>>>>>>>>>>>> need a
> > > > >>>>>>>>>>>>>>>>>>>>>>> common
> > > > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > > situations,
> > > > >>>>>> they can
> > > > >>>>>>>>>>>>>>>>>> implement
> > > > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > > > >>>>>>>>>>>>>>>>>>>>>
> ConsumerRebalanceListener.onPartitionsAssigned().
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > > >>>>>> s7133700@gmail.com>
> > > > >>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned
> value
> > > > >>>>>> should
> > > > >>>>>>>>>>>>>> always
> > > > >>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the
> KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> positive
> > > > >>>>>> any
> > > > >>>>>>>>>>>>>>>>>> field
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect
> the
> > > > version
> > > > >>>>>>>>>>>>>> mismatch
> > > > >>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>> during
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment,
> which
> > > > occurs
> > > > >>>>>> inside
> > > > >>>>>>>>>>>>>>>>> poll().
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > > UnsupportedVersionException
> > > > >>>>>> from
> > > > >>>>>>>>>>>>>> poll().
> > > > >>>>>>>>>>>>>>>>>> I’ll
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > 2026年5月17日
> > > > >>>>>> 下午4:50
> > > > >>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The
> age of
> > > > this
> > > > >>>>>>>>>>>>>> partition
> > > > >>>>>>>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the
> broker
> > > > as
> > > > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > > > >>>>>>>>>>>>>>>>>>>>>>> -
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the
> broker
> > > > >>>>>> does not
> > > > >>>>>>>>>>>>>>>>> support
> > > > >>>>>>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is
> > > > unknown.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned
> value
> > > > >>>>>> should
> > > > >>>>>>>>>>>>>> always
> > > > >>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)
> positive
> > > > >>>>>> any
> > > > >>>>>>>>>>>>>>>>>> field
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter
> UnsupportedVersionException
> > > > when
> > > > >>>>>> calling
> > > > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design
> change
> > > > based on
> > > > >>>>>> the
> > > > >>>>>>>>>>>>>>>>>> discussion
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the
> new-partition
> > > > >>>> reset
> > > > >>>>>>>>>>>>>> behavior
> > > > >>>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > > >>>>>>>
> > > > >>>>>>>>>>>>>> now applies to all
> > > > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the
> consumer
> > > > resets
> > > > >>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>>
> > > > >>>>>>>>>>>>>> .partitions
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the
> base
> > > > >>>>>>>>>>>>>> auto.offset.reset
> > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > > represented
> > > > >>>>>> by a
> > > > >>>>>>>>>>>>>>> separate
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>> .partitions,
> > > > >>>>>>>>>>>>>> currently fixed to
> > > > >>>>>>>>>>>>>>>>> earliest).
> > > > >>>>>>>>>>>>>>>>>>>>> This
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the
> behavior to a
> > > > >>>>>> public
> > > > >>>>>>>>>>>>>>>>>> user-facing
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > > 2026年5月16日
> > > > >>>>>> 清晨7:46
> > > > >>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal
> from me
> > > > is
> > > > >>>>>> listed
> > > > >>>>>>>>>>>>>>> below:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>> .partitions
> > > > >>>>>>>>>>>>>> with a default value
> > > > >>>>>>>>>>>>>>>>> of
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > > > >>>>>> by_duration and
> > > > >>>>>>>>>>>>>>>>> latest,
> > > > >>>>>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > > >>>>>> auto.offset.reset=earliest.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>> .partitions
> > > > >>>>>>>>>>>>>> as an internal
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>>
> > > > >>>>>>>>>>>>>> .partitions=earliest
> > > > >>>>>>>>>>>>>>> already
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss
> the use
> > > > cases
> > > > >>>>>> of
> > > > >>>>>>>>>>>>>> other
> > > > >>>>>>>>>>>>>>>>>> values
> > > > >>>>>>>>>>>>>>>>>>>>>>> in a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>> .partitions
> > > > >>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > > > >>>>>>> ,
> > > > >>>>>>>>>>>>>> will be applied to all for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev
> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP
> says
> > > > "When
> > > > >>>>>> a Kafka
> > > > >>>>>>>>>>>>>>>>> topic
> > > > >>>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using
> the
> > > > latest
> > > > >>>>>> auto
> > > > >>>>>>>>>>>>>> offset
> > > > >>>>>>>>>>>>>>>>>> reset
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records
> produced to
> > > > those
> > > > >>>>>>>>>>>>>> partitions
> > > > >>>>>>>>>>>>>>>>>> before
> > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user
> sets
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec,
> the same
> > > > >>>>>> record loss
> > > > >>>>>>>>>>>>>>>>> issue
> > > > >>>>>>>>>>>>>>>>>>>>> could
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking
> auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>> .partitions
> > > > >>>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>> take
> > > > >>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>> same
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user
> could
> > > > set
> > > > >>>>>> it
> > > > >>>>>>>>>>>>>>>>>> by_duration if
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM
> Chia-Ping
> > > > Tsai <
> > > > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be
> missing
> > > > >>>>>> something
> > > > >>>>>>>>>>>>>>>>>> important
> > > > >>>>>>>>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as
> I try
> > > > to
> > > > >>>>>> clarify
> > > > >>>>>>>>>>>>>> with
> > > > >>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>> few
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for
> extending
> > > > this
> > > > >>>>>> logic
> > > > >>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>> other
> > > > >>>>>>>>>>>>>>>>>>>>>>> reset
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > > > earliest
> > > > >>>>>> or
> > > > >>>>>>>>>>>>>>>>> by_duration
> > > > >>>>>>>>>>>>>>>>>>>>>>> don't
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data
> loss
> > > > issue
> > > > >>>>>> when a
> > > > >>>>>>>>>>>>>>>>>> partition
> > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users
> to
> > > > >>>>>> configure for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>> .partitions?
> > > > >>>>>>>>>>>>>> If they set it to
> > > > >>>>>>>>>>>>>>>>> earliest
> > > > >>>>>>>>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge
> > > > cases. For
> > > > >>>>>>>>>>>>>> example,
> > > > >>>>>>>>>>>>>>>>> if a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new
> partition is
> > > > >>>>>> created
> > > > >>>>>>>>>>>>>> during
> > > > >>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip
> to
> > > > latest
> > > > >>>>>> when
> > > > >>>>>>>>>>>>>>>>> resuming,
> > > > >>>>>>>>>>>>>>>>>>>>>>> rather
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > > > partition
> > > > >>>>>> is
> > > > >>>>>>>>>>>>>>>>>> technically
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > > > introducing a
> > > > >>>>>> max.age
> > > > >>>>>>>>>>>>>>>>>> threshold.
> > > > >>>>>>>>>>>>>>>>>>>>>>> It
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what
> is
> > > > >>>>>> genuinely
> > > > >>>>>>>>>>>>>>> "hot/new"
> > > > >>>>>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't
> seen yet.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev
> > > > wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find
> auto.offset.reset.latest.max.age a
> > > > bit
> > > > >>>>>> weird. It
> > > > >>>>>>>>>>>>>>>>> only
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest.
> However, it
> > > > seems
> > > > >>>>>> that the
> > > > >>>>>>>>>>>>>>>>>>>>> motivation
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is
> set to
> > > > other
> > > > >>>>>> values
> > > > >>>>>>>>>>>>>> like
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a
> > > > separate
> > > > >>>>>> way to
> > > > >>>>>>>>>>>>>>> control
> > > > >>>>>>>>>>>>>>>>>>>>> newly
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions
> when the
> > > > >>>> group
> > > > >>>>>>>>>>>>>> starts.
> > > > >>>>>>>>>>>>>>>>>> Have we
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > > auto.offset.reset.new
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > > >>>>>>>
> > > > >>>>>>>>>>>>>>> .partitions?
> > > > >>>>>>>>>>>>>>>>>> If
> > > > >>>>>>>>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset
> policy
> > > > >>>>>> defaults to
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>> policy
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could
> set it
> > > > >>>>>> explicitly
> > > > >>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>> customize
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > > >>>>>> s7133700@gmail.com
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this
> thread.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於
> 2026年5月1日
> > > > 晚上10:37
> > > > >>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to
> v14.
> > > > The
> > > > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64,
> default -1),
> > > > >>>>>> computed
> > > > >>>>>>>>>>>>>>>>>> server-side
> > > > >>>>>>>>>>>>>>>>>>>>>>> by
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > > >>>> partition_creation_time.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat
> flow.
> > > > when
> > > > >>>>>>>>>>>>>>>>>> MembershipManager
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly
> invalidates the
> > > > >>>>>> metadata for
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>> affected
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset
> decision,
> > > > even
> > > > >>>>>> if the
> > > > >>>>>>>>>>>>>>>>> topic
> > > > >>>>>>>>>>>>>>>>>> ID
> > > > >>>>>>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's
> maximum
> > > > >>>>>> supported
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at
> connection
> > > > time.
> > > > >>>>>> If the
> > > > >>>>>>>>>>>>>>>>>> negotiated
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > > > >>>>>> PartitionAgeMs at
> > > > >>>>>>>>>>>>>> all
> > > > >>>>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently
> falling
> > > > >>>>>> back to
> > > > >>>>>>>>>>>>>> latest
> > > > >>>>>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any
> operator-visible
> > > > signal.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in
> the
> > > > KIP.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> chia7712@gmail.com> 於
> > > > >>>>>> 2026年4月29日
> > > > >>>>>>>>>>>>>>> 下午4:04
> > > > >>>>>>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of
> moving the
> > > > >>>>>> 'age'
> > > > >>>>>>>>>>>>>>>>> resolution
> > > > >>>>>>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API
> to keep
> > > > the
> > > > >>>>>> control
> > > > >>>>>>>>>>>>>>>>> plane
> > > > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > > > introducing
> > > > >>>>>>>>>>>>>>> inter-broker
> > > > >>>>>>>>>>>>>>>>>>>>> clock
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided
> a
> > > > single
> > > > >>>>>> source of
> > > > >>>>>>>>>>>>>>>>> truth
> > > > >>>>>>>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time
> skew
> > > > >>>>>> should be
> > > > >>>>>>>>>>>>>>>>>> negligible.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > > > >>>>>> configured in
> > > > >>>>>>>>>>>>>>> minutes
> > > > >>>>>>>>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds)
> between
> > > > >>>>>> brokers
> > > > >>>>>>>>>>>>>> won't
> > > > >>>>>>>>>>>>>>>>>> impact
> > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > > dev@kafka.apache.org>
> > > > >>>>>> 於
> > > > >>>>>>>>>>>>>>>>> 2026年4月29日
> > > > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed
> the
> > > > >>>>>> previous
> > > > >>>>>>>>>>>>>>>>>> conversation
> > > > >>>>>>>>>>>>>>>>>>>>>>> but I
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand
> the
> > > > flow
> > > > >>>>>> with the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There
> is a
> > > > new
> > > > >>>>>> boolean;
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> KIP
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only
> when
> > > > this
> > > > >>>>>> boolean
> > > > >>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>> set.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer
> receives
> > > > a new
> > > > >>>>>>>>>>>>>> partition,
> > > > >>>>>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set
> to
> > > > receive
> > > > >>>>>> the
> > > > >>>>>>>>>>>>>> ages.
> > > > >>>>>>>>>>>>>>> Is
> > > > >>>>>>>>>>>>>>>>>> my
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps
> clarify the
> > > > flow
> > > > >>>>>> and
> > > > >>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>>>> explain
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > > > offsets,
> > > > >>>>>> fetch
> > > > >>>>>>>>>>>>>>>>> offsets,
> > > > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is
> correct, I
> > > > >>>>>> wonder if
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API
> is the
> > > > right
> > > > >>>>>> place
> > > > >>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>> this
> > > > >>>>>>>>>>>>>>>>>>>>>>> given
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > > Alternatively,
> > > > >>>>>> it could
> > > > >>>>>>>>>>>>>>>>> simply
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be
> > > > rather
> > > > >>>>>> cautious
> > > > >>>>>>>>>>>>>>> about
> > > > >>>>>>>>>>>>>>>>>> not
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API
> with
> > > > >>>>>> unrelated
> > > > >>>>>>>>>>>>>>> concepts.
> > > > >>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>>>> API
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning
> or
> > > > revoking
> > > > >>>>>>>>>>>>>>> partitions.
> > > > >>>>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>>>>>> fact
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > > corresponding
> > > > >>>>>> Streams
> > > > >>>>>>>>>>>>>> API
> > > > >>>>>>>>>>>>>>>>>> also
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right.
> What
> > > > would we
> > > > >>>>>> do if
> > > > >>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>>> want to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > > > >>>>>> Muralidhar Basani
> > > > >>>>>>>>>>>>>>> via
> > > > >>>>>>>>>>>>>>>>>> dev
> > > > >>>>>>>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip.
> Good to
> > > > know
> > > > >>>>>> about
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> gap.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version
> bump for
> > > > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on
> response,
> > > > >>>>>> PartitionAges
> > > > >>>>>>>>>>>>>> on
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version
> bump?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is
> there a
> > > > >>>>>> recommended
> > > > >>>>>>>>>>>>>>>>> value
> > > > >>>>>>>>>>>>>>>>>> or
> > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should
> based
> > > > on the
> > > > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > > >>>>>>>>>>>>>> <
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > > > >>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of
> javadocs I
> > > > >>>>>> guess.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no
> > > > changes to
> > > > >>>>>> Kafka
> > > > >>>>>>>>>>>>>>>>> Streams,
> > > > >>>>>>>>>>>>>>>>>>>>>>> would
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > > >>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users
> configure
> > > > it?
> > > > >>>>>> This is
> > > > >>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>> most
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might
> easily
> > > > >>>>>> mistakenly
> > > > >>>>>>>>>>>>>>>>>> configure
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > > > consumer
> > > > >>>>>> falls
> > > > >>>>>>>>>>>>>> back
> > > > >>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > > > >>>>>> per-partition
> > > > >>>>>>>>>>>>>> which
> > > > >>>>>>>>>>>>>>>>>> isn't
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > > > "consumer
> > > > >>>>>> resolves
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>> initial
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that
> partition" as if
> > > > >>>>>> earliest
> > > > >>>>>>>>>>>>>> was
> > > > >>>>>>>>>>>>>>>>>> applied
> > > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and
> auto.offset.reset
> > > > >>>>>> config is
> > > > >>>>>>>>>>>>>>>>>> unchanged.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at
> 2:48 PM 黃竣陽 <
> > > > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to
> include
> > > > this
> > > > >>>>>> change.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <
> chia7712@apache.org>
> > > > 於
> > > > >>>>>>>>>>>>>> 2026年4月28日
> > > > >>>>>>>>>>>>>>>>>> 晚上8:03
> > > > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > > > partition
> > > > >>>>>> creation
> > > > >>>>>>>>>>>>>>> time
> > > > >>>>>>>>>>>>>>>>>> via
> > > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable
> for
> > > > users
> > > > >>>> to
> > > > >>>>>>>>>>>>>> diagnose
> > > > >>>>>>>>>>>>>>>>> and
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > > >>>>>> auto.offset.reset.latest.max.age
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽
> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a
> discussion
> > > > on
> > > > >>>>>> KIP-1327
> > > > >>>>>>>>>>>>>>>>>> Prevent
> > > > >>>>>>>>>>>>>>>>>>>>> Hot
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest
> Policy
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to
> introduces
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy
> distinguish
> > > > newly
> > > > >>>>>> expanded
> > > > >>>>>>>>>>>>>>>>> (hot)
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones.
> Partitions
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > > > threshold
> > > > >>>>>>>>>>>>>>> automatically
> > > > >>>>>>>>>>>>>>>>>> fall
> > > > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent
> data loss
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion
> without
> > > > forcing
> > > > >>>>>> a full
> > > > >>>>>>>>>>>>>>>>>> historical
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Andrew Schofield <sc...@apache.org>.
Hi Jiunn-Yang et al.,
Some background on the problem I hinted at for the KIP-932 implementation. KIP-932 states "If the number of partitions is increased for a topic with a subscription in a share group, the SPSO for the newly created share-partitions is initialized to 0 (which is of course both the earliest and latest offset for an empty topic-partition). This means there is no doubt about what happens when the number of partitions is increased."

The problem was that we had not implemented the use of 0, but instead were using the share.auto.offset.reset config to choose the initial offset in this case. We write a ShareGroupStatePartitionMetadata record when a topic is first being initialized for consumption by a share group, and this contains a list of the partitions which we are initializing. As a result, I believe we can already tell when partitions are added, and we do know when to initialize to 0 as it says in the KIP. We will put in a patch that uses 0 as intended.

I do not mind whether we want to incorporate share groups into KIP-1327, but I do not think that it is necessary to do so.

Thanks,
Andrew

On 2026/06/26 04:31:28 Chia-Ping Tsai wrote:
> > JR5. Since this changes the on-disk data format for PartitionRecord, we
> > need to gate this by a new MV. It also changes the format of
> > ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if this
> > requires a new MV or a new GV.
> 
> That is a good point. If we want to maintain consistency for the NewPartitions field when the group coordinator migrates between new and old binaries, a version gate is definitely required. For example:
> 
> GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())
> 
> The group coordinator should NOT set the NewPartitions field when the gv is smaller than gv_2
> 
> Best,
> Chia-Ping
> 
> On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> > Hi, Jiunn-Yang,
> > 
> > JR2. We should try to maintain consistency across our APIs. So, it would be
> > useful to understand if the user-facing part of this KIP is what share
> > consumer wants too. It would be problematic if we proceed with this
> > approach only for the share consumer to choose a different one later.
> > 
> > JR3. I was thinking that if we add the NewPartitions field in
> > GroupHeartbeatResponse, there is no need to expose the group/partition
> > creation timestamp to other RPCs and the Cli.
> > 
> > JR5. Since this changes the on-disk data format for PartitionRecord, we
> > need to gate this by a new MV. It also changes the format of
> > ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if this
> > requires a new MV or a new GV.
> > 
> > Thanks,
> > 
> > Jun
> > 
> > On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> > 
> > > Hello Jun, chia,
> > >
> > > Thanks for your feedback.
> > >
> > > JR3 JR4 I have updated the KIP.
> > >
> > > JR2 The new-partition detection mechanism is reusable. The coordinator
> > > already has
> > > access to both timestamps required for classification, so the
> > > determination can be made
> > > server-side without introducing any new RPCs. The main difference is where
> > > the policy is
> > > enforced: modern consumer resolves it on the client, whereas a share group
> > > resolves the
> > > starting offset on the partition leader. As a result, the classification
> > > decision would need to
> > > be propagated through the existing share-state initialization path. We
> > > would also need to persist
> > > the group's creation time as part of the share-group state, since that
> > > information is not stored today.
> > >
> > > None of this is a blocker, but it does require changes to the share-state
> > > format and introduces
> > > a share-specific configuration. Given that additional scope, I would
> > > prefer to keep share groups out
> > > of scope for this KIP and address them in a follow-up effort, while
> > > documenting the relevant reuse
> > > opportunities here.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > > >
> > > >> It inspires me to think about mirror clusters. What if the mirror group
> > > is created 'after' the expanded partitions? In this scenario, the new
> > > configuration 'max.age.ms' could handle it if users are aware of the
> > > migration and set a suitable time interval. Of course, this would require
> > > both creation times to be exposed to clients.
> > > >
> > > > Thinking more about the mirror cluster scenario we discussed, I realize
> > > it introduces significant complexity. Handling it properly would require a
> > > tolerance time window rather than a strict time interval to account for
> > > replication lag. To prevent scope creep and keep KIP-1327 focused on its
> > > primary goal, I think it would be best to defer the multi-cluster/mirroring
> > > scenario and explore those approaches in a separate KIP
> > > >
> > > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > > >> hi all,
> > > >>
> > > >>> JR3. The KIP exposes the group/partition creation time to the client.
> > > I am
> > > >>> not sure if there are other use cases for those timestamps. An
> > > alternative
> > > >>> is to avoid exposing those timestamps to the client. The group
> > > coordinator
> > > >>> will categorize the partitions using those timestamps and include the
> > > >>> categorization in the assigned partitions in GroupHeartbeatResponse.
> > > >>
> > > >> It inspires me to think about mirror clusters. What if the mirror group
> > > is created 'after' the expanded partitions? In this scenario, the new
> > > configuration 'max.age.ms' could handle it if users are aware of the
> > > migration and set a suitable time interval. Of course, this would require
> > > both creation times to be exposed to clients.
> > > >>
> > > >> Best,
> > > >> Chia-Ping
> > > >>
> > > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > > >>> Hi, Jiunn-Yang,
> > > >>>
> > > >>> Thanks for the updated KIP.
> > > >>>
> > > >>> JR2. Share consumer (KIP-932) states: "If the number of partitions is
> > > >>> increased for a topic with a subscription in a share group, the SPSO
> > > for
> > > >>> the newly created share-partitions is initialized to 0 ". However, this
> > > >>> part has not been implemented yet. Currently, all partitions without
> > > offset
> > > >>> follows the strategy given by share.auto.offset.reset. So, we have an
> > > >>> opportunity to see if it's better to align this KIP's changes with the
> > > >>> share consumer.
> > > >>> JR2.1 The first part is whether customizing the offset for new
> > > partitions
> > > >>> to something other than 0 is useful in the share consumer. It seems
> > > that
> > > >>> the motivation in this KIP applies to share consumer too.
> > > >>> JR2.2 The second part is how new partitions are detected. The mechanism
> > > >>> introduced in this KIP seems quite effective. A group coordinator could
> > > >>> pick up newly created partitions for an existing topic or newly created
> > > >>> topics matching the regex. Those partitions are categorized as new
> > > >>> partitions and will follow the strategy given by
> > > >>> auto.offset.reset.new.partitions.
> > > >>> This makes sense since those partitions typically don't have a large
> > > >>> backlog. A consumer could also additionally subscribe to an existing
> > > topic
> > > >>> in an existing group. Those partitions will be categorized as existing
> > > >>> partitions. This also makes sense since those partitions could have a
> > > large
> > > >>> backlog and it is better to follow the strategy given by
> > > auto.offset.reset. It
> > > >>> will be useful to think through whether the logic for categorizing new
> > > >>> partitions should also be reused in the share consumer.
> > > >>>
> > > >>> JR3. The KIP exposes the group/partition creation time to the client.
> > > I am
> > > >>> not sure if there are other use cases for those timestamps. An
> > > alternative
> > > >>> is to avoid exposing those timestamps to the client. The group
> > > coordinator
> > > >>> will categorize the partitions using those timestamps and include the
> > > >>> categorization in the assigned partitions in GroupHeartbeatResponse.
> > > >>>
> > > >>> JR4. It would be useful to support this feature on existing groups as
> > > well.
> > > >>> For example, any partition without a creation timestamp can always be
> > > >>> categorized as an existing partition.
> > > >>>
> > > >>> Jun
> > > >>>
> > > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>>
> > > >>>> Hello Andrew,
> > > >>>>
> > > >>>> Thanks for your feedback,
> > > >>>>>> I've just been re-reading this KIP to ensure that share group
> > > support is
> > > >>>>>> not needed. I've discovered that we made a mistake implementing
> > > KIP-932
> > > >>>> in
> > > >>>>>> this area, so we'll rectify that and check whether it's really tight
> > > >>>> enough
> > > >>>>>> not to require KIP-1327.
> > > >>>>
> > > >>>> Could you walk us through the specific area where the mistake was
> > > found?
> > > >>>> That would
> > > >>>> help us assess whether it naturally fits within the scope of KIP-1327
> > > or
> > > >>>> should be addressed
> > > >>>> separately.
> > > >>>>
> > > >>>> AS1 I have updated the KIP
> > > >>>>
> > > >>>> Best Regards,
> > > >>>> Jiunn-Yang
> > > >>>>
> > > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> > > >>>>>
> > > >>>>> hi Andrew
> > > >>>>>
> > > >>>>>> I've discovered that we made a mistake implementing KIP-932 in this
> > > >>>> area,
> > > >>>>> so we'll rectify that and check whether it's really tight enough not
> > > to
> > > >>>>> require KIP-1327.
> > > >>>>>
> > > >>>>> Would you mind sharing more details about that? If it's not anything
> > > >>>> close
> > > >>>>> to NP-hard, we could definitely look into covering it within KIP-1327
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Chia-Ping
> > > >>>>>
> > > >>>>>
> > > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
> > > >>>>>
> > > >>>>>> Hi Jiunn-Yang,
> > > >>>>>> I've just been re-reading this KIP to ensure that share group
> > > support is
> > > >>>>>> not needed. I've discovered that we made a mistake implementing
> > > KIP-932
> > > >>>> in
> > > >>>>>> this area, so we'll rectify that and check whether it's really tight
> > > >>>> enough
> > > >>>>>> not to require KIP-1327.
> > > >>>>>>
> > > >>>>>> One trivial comments for consistency.
> > > >>>>>>
> > > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > > represented by
> > > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Andrew
> > > >>>>>>
> > > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > > >>>>>>> Hi all,
> > > >>>>>>>
> > > >>>>>>> Manually bumping this thread.
> > > >>>>>>>
> > > >>>>>>> Best Regards,
> > > >>>>>>> Jiunn-Yang
> > > >>>>>>>
> > > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > > >>>>>>>>
> > > >>>>>>>> Hello chia,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > > >>>>>>>>
> > > >>>>>>>> Best Regards,
> > > >>>>>>>> Jiunn-Yang
> > > >>>>>>>>
> > > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> > > >>>>>>>>>
> > > >>>>>>>>> hi Jiunn-Yang
> > > >>>>>>>>>
> > > >>>>>>>>>> When the config is set on a cluster that has not yet been
> > > >>>>>> upgraded... classification cannot occur... the consumer falls back
> > > to
> > > >>>> the
> > > >>>>>> base auto.offset.reset for the affected partitions. No exception is
> > > >>>> thrown,
> > > >>>>>> and no operational disruption results.
> > > >>>>>>>>>
> > > >>>>>>>>> Existing group can't take advantage of this excellent new
> > > >>>>>> configuration. Allowing users to modify the group creation time
> > > might be
> > > >>>>>> overkill. Instead, we could print a useful warning message to guide
> > > >>>> users.
> > > >>>>>> For example, we can suggest that they re-create the group with their
> > > >>>>>> existing committed offsets
> > > >>>>>>>>>
> > > >>>>>>>>>> Protocol changes
> > > >>>>>>>>>
> > > >>>>>>>>> Would you mind listing those RPC changes in a table format?
> > > >>>>>>>>>
> > > >>>>>>>>>> The full interaction matrix between the base policy and the
> > > >>>>>> new-partition policy is:
> > > >>>>>>>>>
> > > >>>>>>>>> Please add a filed to describe the target scenario when using
> > > these
> > > >>>>>> policies
> > > >>>>>>>>>
> > > >>>>>>>>> Best,
> > > >>>>>>>>> Chia-Ping
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > > >>>>>>>>>> Hello Jun, chia,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for the new
> > > >>>>>>>>>> approach, PTAL
> > > >>>>>>>>>>
> > > >>>>>>>>>> Best Regards,
> > > >>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Yes, your approach is great. I think the combination of latest
> > > (for
> > > >>>>>> existing partitions) and by_duration (for new partitions) can
> > > address
> > > >>>> 99%
> > > >>>>>> of the complaints I have heard regarding this issue.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Also, leveraging the group creation time here opens the door to
> > > >>>>>> implementing a new policy based on timestamp seek in the future,
> > > should
> > > >>>> the
> > > >>>>>> community want to pursue that.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks for your patience and constructive feedback. We will
> > > update
> > > >>>>>> the KIP accordingly.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best, Chia-Ping
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11
> > > 寫道:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Hi, Chia-Ping,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > > configure a
> > > >>>>>> different
> > > >>>>>>>>>>>> offset policy for existing partitions vs new partitions.
> > > However,
> > > >>>>>> using
> > > >>>>>>>>>>>> group creation time to capture that seems more intuitive.
> > > Here is
> > > >>>>>> another
> > > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize
> > > new
> > > >>>>>> partitions
> > > >>>>>>>>>>>> based on group creation time. Introduce
> > > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose values
> > > can be
> > > >>>>>> earliest,
> > > >>>>>>>>>>>> latest and by_duration, the same as auto.offset.reset. Users
> > > can
> > > >>>>>> set
> > > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they
> > > want to
> > > >>>>>> guarantee
> > > >>>>>>>>>>>> no data loss on new partitions. They can also use by_duration
> > > to
> > > >>>>>> set an
> > > >>>>>>>>>>>> upper bound on the backlog replayed, which can be different
> > > from
> > > >>>>>> that of
> > > >>>>>>>>>>>> the existing partitions. This will address your concern about
> > > too
> > > >>>>>> much
> > > >>>>>>>>>>>> backlog being replayed when the offsets are lost. What do you
> > > >>>>>> think?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > > >>>>>> chia7712@gmail.com> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> The most important part of this story is how users should
> > > expect
> > > >>>>>> the data
> > > >>>>>>>>>>>>> they can see when using the latest or by_duration policy with
> > > >>>>>> expanded
> > > >>>>>>>>>>>>> partitions.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > > >>>>>>>>>>>>> non-deterministic, which means users will either read too
> > > many
> > > >>>>>> historical
> > > >>>>>>>>>>>>> records from existing partitions or lose some records from
> > > >>>>>> expanded
> > > >>>>>>>>>>>>> partitions.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > > >>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > >>>>>>>
> > > >>>>>>>>>>>>> is a bit hard to understand, and that is why I preferred
> > > having a
> > > >>>>>> whole new
> > > >>>>>>>>>>>>> policy based entirely on group creation time (KIP-1282)
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > > 上午1:08寫道:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand the
> > > value
> > > >>>>>> of the new
> > > >>>>>>>>>>>>>> configs with the KIP.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want to
> > > miss
> > > >>>>>> the data if
> > > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing partition
> > > is
> > > >>>>>> easy to
> > > >>>>>>>>>>>>>> understand because it relates to retention time. The
> > > backlog for
> > > >>>>>> the new
> > > >>>>>>>>>>>>>> partition is a bit subtle to understand since it depends on
> > > the
> > > >>>>>> metadata
> > > >>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > >>>>>>> ,
> > > >>>>>>>>>>>>>> the user needs to
> > > >>>>>>>>>>>>>> understand the metadata refresh delay on the consumer side
> > > and
> > > >>>>>> use it to
> > > >>>>>>>>>>>>>> set the config.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the same value
> > > for
> > > >>>>>> the
> > > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> > > >>>> this
> > > >>>>>>>>>>>>>> approach.
> > > >>>>>>>>>>>>>> 1. It computes the seek target client-side as now() -
> > > duration,
> > > >>>>>> which
> > > >>>>>>>>>>>>>> introduces clock skew across consumers and forces operators
> > > to
> > > >>>>>> choose
> > > >>>>>>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> > > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
> > > failed
> > > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> > > >>>>>> potentially
> > > >>>>>>>>>>>>>> miss records produced between attempts.
> > > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without committed
> > > >>>>>> offsets, and
> > > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > > long-existing
> > > >>>>>> partitions
> > > >>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding
> > > a bit
> > > >>>>>> buffer to
> > > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > > improving the
> > > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> > > >>>>>> typically low
> > > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer and tens
> > > of
> > > >>>>>> seconds
> > > >>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading that much
> > > >>>>>> backlog for new
> > > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the same for
> > > existing
> > > >>>>>>>>>>>>>> partitions.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> So, instead of introducing a new config, could we just
> > > reuse the
> > > >>>>>> existing
> > > >>>>>>>>>>>>>> config with better documentation and/or implementation?
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> You're right that group creation time is the more intuitive
> > > >>>>>> answer at
> > > >>>>>>>>>>>>>>> first glance,
> > > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that
> > > "predate
> > > >>>>>> the group"
> > > >>>>>>>>>>>>>>> vs partitions
> > > >>>>>>>>>>>>>>> "created during group runtime," which directly points to a
> > > >>>>>>>>>>>>>> group-lifecycle
> > > >>>>>>>>>>>>>>> classifier.
> > > >>>>>>>>>>>>>>> I'd like to walk through why we landed on partition age,
> > > and
> > > >>>> the
> > > >>>>>>>>>>>>>>> trade-offs we considered.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> This covers the metadata blindness window, but has issues
> > > the
> > > >>>>>> KIP
> > > >>>>>>>>>>>>>>> currently documents
> > > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
> > > across
> > > >>>>>> consumers.
> > > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > > potentially
> > > >>>>>> missing
> > > >>>>>>>>>>>>>>> records produced between
> > > >>>>>>>>>>>>>>> attempts.
> > > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without committed
> > > >>>>>> offsets,
> > > >>>>>>>>>>>>>>> including pre-existing partitions
> > > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively running.
> > > Our
> > > >>>>>> concern
> > > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> T=0:         Group created.
> > > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > >>>>>>>>>>>>>>> T=50:        Partition added during the idle window.
> > > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Under group creation time, the new partition is classified
> > > as
> > > >>>>>> new
> > > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
> > > from
> > > >>>>>> T=50.
> > > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> > > >>>>>> data that
> > > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the
> > > contract
> > > >>>> of
> > > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason
> > > to
> > > >>>>>> treat
> > > >>>>>>>>>>>>>>> the new partition differently; both contain backlog
> > > accumulated
> > > >>>>>> during
> > > >>>>>>>>>>>>>>> the same idle window.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle you
> > > raised
> > > >>>>>> in
> > > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
> > > backlog
> > > >>>>>> on
> > > >>>>>>>>>>>>>>> every other partition during the same idle period; forcing
> > > >>>>>> 0-backlog
> > > >>>>>>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> > > >>>> that
> > > >>>>>>>>>>>>>>> tolerance.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent data loss
> > > >>>> window,
> > > >>>>>>>>>>>>>>> the gap between partition creation and the consumer’s
> > > metadata
> > > >>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely
> > > silent: the
> > > >>>>>>>>>>>>>>> consumer had no opportunity to know about the partition.
> > > >>>>>> Outside this
> > > >>>>>>>>>>>>>>> window, missing data reflects either:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> > > consumers,
> > > >>>>>> or
> > > >>>>>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
> > > via
> > > >>>>>> reset
> > > >>>>>>>>>>>>>> policy.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> We did not choose partition age because it is more elegant
> > > than
> > > >>>>>> group
> > > >>>>>>>>>>>>>>> creation time — we chose it because its failure mode
> > > (requires
> > > >>>> a
> > > >>>>>>>>>>>>>>> threshold) is
> > > >>>>>>>>>>>>>>> less invasive than the failure mode of group creation time
> > > >>>>>> (overrides
> > > >>>>>>>>>>>>>>> user-stated
> > > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> > > 寫道:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi Jun,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Relying on both creation times will create an inconsistent
> > > >>>>>> scenario. A
> > > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
> > > seek
> > > >>>>>> to the
> > > >>>>>>>>>>>>>>>> beginning for the partitions created later than the group.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> > > >>>>>> inconsistency
> > > >>>>>>>>>>>>>>> using a
> > > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> > > consensus,
> > > >>>>>> KIP-1327
> > > >>>>>>>>>>>>>>> goes
> > > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent users
> > > from
> > > >>>>>> falling
> > > >>>>>>>>>>>>>> into
> > > >>>>>>>>>>>>>>>> that pitfall.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > > >>>>>> 上午6:49寫道:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > >>>>>>>
> > > >>>>>>>>>>>>>> still feels weird to
> > > >>>>>>>>>>>>>>> me.
> > > >>>>>>>>>>>>>>>>> It
> > > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on the
> > > >>>>>> partition
> > > >>>>>>>>>>>>>>> creation
> > > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be based on
> > > the
> > > >>>>>> group
> > > >>>>>>>>>>>>>>> creation
> > > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is created
> > > are
> > > >>>>>> existing
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> all partitions created after the group creation are new
> > > >>>>>> partitions.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > > >>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> > > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
> > > task,
> > > >>>>>> not an
> > > >>>>>>>>>>>>>> edge
> > > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
> > > clarify
> > > >>>>>> that
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>> is a
> > > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which
> > > further
> > > >>>>>>>>>>>>>> demonstrates
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> > > >>>>>> Kafka: To
> > > >>>>>>>>>>>>>>>>> balance
> > > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> > > >>>>>> created with a
> > > >>>>>>>>>>>>>>>>>> moderate
> > > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic grows
> > > over
> > > >>>>>> time, it
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>> often
> > > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> > > >>>>>> accommodate the
> > > >>>>>>>>>>>>>>>>> higher
> > > >>>>>>>>>>>>>>>>>>>> workload.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Regards
> > > >>>>>>>>>>>>>>>>>>>> Jian
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> > > 晚上8:29
> > > >>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> > > when
> > > >>>>>>>>>>>>>> describing
> > > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
> > > them
> > > >>>> to
> > > >>>>>>>>>>>>>> describe
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> > > >>>>>> applying
> > > >>>>>>>>>>>>>> these
> > > >>>>>>>>>>>>>>>>>> terms
> > > >>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> > > >>>>>> whether a
> > > >>>>>>>>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is
> > > that
> > > >>>>>> they won't
> > > >>>>>>>>>>>>>>>>>> silently
> > > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
> > > their
> > > >>>>>> active
> > > >>>>>>>>>>>>>>>>>>>>> consumption.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > > >>>>>> motivation
> > > >>>>>>>>>>>>>>> section.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > > 2026年5月30日
> > > >>>>>> 凌晨1:12
> > > >>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > > >>>>>> motivation
> > > >>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> > > >>>>>> partitions
> > > >>>>>>>>>>>>>> without
> > > >>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> > > >>>>>> partition
> > > >>>>>>>>>>>>>> (hot)
> > > >>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> > > consumer
> > > >>>>>> has
> > > >>>>>>>>>>>>>> never
> > > >>>>>>>>>>>>>>>>> seen
> > > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
> > > hot
> > > >>>>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> > > also
> > > >>>>>>>>>>>>>>> insufficient
> > > >>>>>>>>>>>>>>>>>>>>>>> because:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
> > > varies
> > > >>>>>> across
> > > >>>>>>>>>>>>>> nodes
> > > >>>>>>>>>>>>>>>>>> due
> > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> > > large
> > > >>>>>>>>>>>>>> duration,
> > > >>>>>>>>>>>>>>>>>>>>>>> causing
> > > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
> > > seek
> > > >>>>>> time on
> > > >>>>>>>>>>>>>>>>> retry,
> > > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
> > > data
> > > >>>>>> loss."
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
> > > these
> > > >>>>>> issues
> > > >>>>>>>>>>>>>>>>> persist,
> > > >>>>>>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > > >>>>>> situations don't
> > > >>>>>>>>>>>>>>>>> need a
> > > >>>>>>>>>>>>>>>>>>>>>>> common
> > > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > > situations,
> > > >>>>>> they can
> > > >>>>>>>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > > >>>>>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > >>>>>> s7133700@gmail.com>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > >>>>>> should
> > > >>>>>>>>>>>>>> always
> > > >>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > >>>>>> any
> > > >>>>>>>>>>>>>>>>>> field
> > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
> > > version
> > > >>>>>>>>>>>>>> mismatch
> > > >>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>> during
> > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
> > > occurs
> > > >>>>>> inside
> > > >>>>>>>>>>>>>>>>> poll().
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > > UnsupportedVersionException
> > > >>>>>> from
> > > >>>>>>>>>>>>>> poll().
> > > >>>>>>>>>>>>>>>>>> I’ll
> > > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > 2026年5月17日
> > > >>>>>> 下午4:50
> > > >>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> > > this
> > > >>>>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> > > as
> > > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > > >>>>>>>>>>>>>>>>>>>>>>> -
> > > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > > >>>>>> does not
> > > >>>>>>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is
> > > unknown.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > >>>>>> should
> > > >>>>>>>>>>>>>> always
> > > >>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>> greater
> > > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > >>>>>> any
> > > >>>>>>>>>>>>>>>>>> field
> > > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> > > when
> > > >>>>>> calling
> > > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> > > based on
> > > >>>>>> the
> > > >>>>>>>>>>>>>>>>>> discussion
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > > >>>> reset
> > > >>>>>>>>>>>>>> behavior
> > > >>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > >>>>>>>
> > > >>>>>>>>>>>>>> now applies to all
> > > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> > > resets
> > > >>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>>
> > > >>>>>>>>>>>>>> .partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > >>>>>>>>>>>>>> auto.offset.reset
> > > >>>>>>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > > represented
> > > >>>>>> by a
> > > >>>>>>>>>>>>>>> separate
> > > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>> .partitions,
> > > >>>>>>>>>>>>>> currently fixed to
> > > >>>>>>>>>>>>>>>>> earliest).
> > > >>>>>>>>>>>>>>>>>>>>> This
> > > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > > >>>>>> public
> > > >>>>>>>>>>>>>>>>>> user-facing
> > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > 2026年5月16日
> > > >>>>>> 清晨7:46
> > > >>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> > > is
> > > >>>>>> listed
> > > >>>>>>>>>>>>>>> below:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>> .partitions
> > > >>>>>>>>>>>>>> with a default value
> > > >>>>>>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > > >>>>>> by_duration and
> > > >>>>>>>>>>>>>>>>> latest,
> > > >>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > >>>>>> auto.offset.reset=earliest.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>> .partitions
> > > >>>>>>>>>>>>>> as an internal
> > > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>>
> > > >>>>>>>>>>>>>> .partitions=earliest
> > > >>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> > > cases
> > > >>>>>> of
> > > >>>>>>>>>>>>>> other
> > > >>>>>>>>>>>>>>>>>> values
> > > >>>>>>>>>>>>>>>>>>>>>>> in a
> > > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>> .partitions
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > > >>>>>>> ,
> > > >>>>>>>>>>>>>> will be applied to all for
> > > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> > > "When
> > > >>>>>> a Kafka
> > > >>>>>>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> > > latest
> > > >>>>>> auto
> > > >>>>>>>>>>>>>> offset
> > > >>>>>>>>>>>>>>>>>> reset
> > > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
> > > those
> > > >>>>>>>>>>>>>> partitions
> > > >>>>>>>>>>>>>>>>>> before
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > > >>>>>> record loss
> > > >>>>>>>>>>>>>>>>> issue
> > > >>>>>>>>>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>> .partitions
> > > >>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>> take
> > > >>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> > > set
> > > >>>>>> it
> > > >>>>>>>>>>>>>>>>>> by_duration if
> > > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> > > Tsai <
> > > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > > >>>>>> something
> > > >>>>>>>>>>>>>>>>>> important
> > > >>>>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> > > to
> > > >>>>>> clarify
> > > >>>>>>>>>>>>>> with
> > > >>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>> few
> > > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> > > this
> > > >>>>>> logic
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>> other
> > > >>>>>>>>>>>>>>>>>>>>>>> reset
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > > earliest
> > > >>>>>> or
> > > >>>>>>>>>>>>>>>>> by_duration
> > > >>>>>>>>>>>>>>>>>>>>>>> don't
> > > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> > > issue
> > > >>>>>> when a
> > > >>>>>>>>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > > >>>>>> configure for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>> .partitions?
> > > >>>>>>>>>>>>>> If they set it to
> > > >>>>>>>>>>>>>>>>> earliest
> > > >>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge
> > > cases. For
> > > >>>>>>>>>>>>>> example,
> > > >>>>>>>>>>>>>>>>> if a
> > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > > >>>>>> created
> > > >>>>>>>>>>>>>> during
> > > >>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> > > latest
> > > >>>>>> when
> > > >>>>>>>>>>>>>>>>> resuming,
> > > >>>>>>>>>>>>>>>>>>>>>>> rather
> > > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > > partition
> > > >>>>>> is
> > > >>>>>>>>>>>>>>>>>> technically
> > > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > > introducing a
> > > >>>>>> max.age
> > > >>>>>>>>>>>>>>>>>> threshold.
> > > >>>>>>>>>>>>>>>>>>>>>>> It
> > > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > > >>>>>> genuinely
> > > >>>>>>>>>>>>>>> "hot/new"
> > > >>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev
> > > wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> > > bit
> > > >>>>>> weird. It
> > > >>>>>>>>>>>>>>>>> only
> > > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> > > seems
> > > >>>>>> that the
> > > >>>>>>>>>>>>>>>>>>>>> motivation
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> > > other
> > > >>>>>> values
> > > >>>>>>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a
> > > separate
> > > >>>>>> way to
> > > >>>>>>>>>>>>>>> control
> > > >>>>>>>>>>>>>>>>>>>>> newly
> > > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > > >>>> group
> > > >>>>>>>>>>>>>> starts.
> > > >>>>>>>>>>>>>>>>>> Have we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > > auto.offset.reset.new
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>>>>>
> > > >>>>>>>>>>>>>>> .partitions?
> > > >>>>>>>>>>>>>>>>>> If
> > > >>>>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > > >>>>>> defaults to
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > > >>>>>> explicitly
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>> customize
> > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > >>>>>> s7133700@gmail.com
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> > > 晚上10:37
> > > >>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> > > The
> > > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > > >>>>>> computed
> > > >>>>>>>>>>>>>>>>>> server-side
> > > >>>>>>>>>>>>>>>>>>>>>>> by
> > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > >>>> partition_creation_time.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow.
> > > when
> > > >>>>>>>>>>>>>>>>>> MembershipManager
> > > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > > >>>>>> metadata for
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> affected
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> > > even
> > > >>>>>> if the
> > > >>>>>>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>>>>>> ID
> > > >>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > > >>>>>> supported
> > > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> > > time.
> > > >>>>>> If the
> > > >>>>>>>>>>>>>>>>>> negotiated
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > > >>>>>> PartitionAgeMs at
> > > >>>>>>>>>>>>>> all
> > > >>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > > >>>>>> back to
> > > >>>>>>>>>>>>>> latest
> > > >>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> > > signal.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the
> > > KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > >>>>>> 2026年4月29日
> > > >>>>>>>>>>>>>>> 下午4:04
> > > >>>>>>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > > >>>>>> 'age'
> > > >>>>>>>>>>>>>>>>> resolution
> > > >>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> > > the
> > > >>>>>> control
> > > >>>>>>>>>>>>>>>>> plane
> > > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > > introducing
> > > >>>>>>>>>>>>>>> inter-broker
> > > >>>>>>>>>>>>>>>>>>>>> clock
> > > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> > > single
> > > >>>>>> source of
> > > >>>>>>>>>>>>>>>>> truth
> > > >>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > > >>>>>> should be
> > > >>>>>>>>>>>>>>>>>> negligible.
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > > >>>>>> configured in
> > > >>>>>>>>>>>>>>> minutes
> > > >>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > > >>>>>> brokers
> > > >>>>>>>>>>>>>> won't
> > > >>>>>>>>>>>>>>>>>> impact
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > > dev@kafka.apache.org>
> > > >>>>>> 於
> > > >>>>>>>>>>>>>>>>> 2026年4月29日
> > > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > > >>>>>> previous
> > > >>>>>>>>>>>>>>>>>> conversation
> > > >>>>>>>>>>>>>>>>>>>>>>> but I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> > > flow
> > > >>>>>> with the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> > > new
> > > >>>>>> boolean;
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> KIP
> > > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> > > this
> > > >>>>>> boolean
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>> set.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives
> > > a new
> > > >>>>>>>>>>>>>> partition,
> > > >>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> > > receive
> > > >>>>>> the
> > > >>>>>>>>>>>>>> ages.
> > > >>>>>>>>>>>>>>> Is
> > > >>>>>>>>>>>>>>>>>> my
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> > > flow
> > > >>>>>> and
> > > >>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>> explain
> > > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > > offsets,
> > > >>>>>> fetch
> > > >>>>>>>>>>>>>>>>> offsets,
> > > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > > >>>>>> wonder if
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> > > right
> > > >>>>>> place
> > > >>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>> given
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > > Alternatively,
> > > >>>>>> it could
> > > >>>>>>>>>>>>>>>>> simply
> > > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be
> > > rather
> > > >>>>>> cautious
> > > >>>>>>>>>>>>>>> about
> > > >>>>>>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > > >>>>>> unrelated
> > > >>>>>>>>>>>>>>> concepts.
> > > >>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>> API
> > > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> > > revoking
> > > >>>>>>>>>>>>>>> partitions.
> > > >>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>> fact
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > > corresponding
> > > >>>>>> Streams
> > > >>>>>>>>>>>>>> API
> > > >>>>>>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> > > would we
> > > >>>>>> do if
> > > >>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>> want to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > > >>>>>> Muralidhar Basani
> > > >>>>>>>>>>>>>>> via
> > > >>>>>>>>>>>>>>>>>> dev
> > > >>>>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> > > know
> > > >>>>>> about
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> gap.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > > >>>>>> PartitionAges
> > > >>>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > > >>>>>> recommended
> > > >>>>>>>>>>>>>>>>> value
> > > >>>>>>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based
> > > on the
> > > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > >>>>>>>>>>>>>> <
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > > >>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > > >>>>>> guess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no
> > > changes to
> > > >>>>>> Kafka
> > > >>>>>>>>>>>>>>>>> Streams,
> > > >>>>>>>>>>>>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > >>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> > > it?
> > > >>>>>> This is
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> most
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > > >>>>>> mistakenly
> > > >>>>>>>>>>>>>>>>>> configure
> > > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > > consumer
> > > >>>>>> falls
> > > >>>>>>>>>>>>>> back
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > > >>>>>> per-partition
> > > >>>>>>>>>>>>>> which
> > > >>>>>>>>>>>>>>>>>> isn't
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > > "consumer
> > > >>>>>> resolves
> > > >>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>> initial
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > > >>>>>> earliest
> > > >>>>>>>>>>>>>> was
> > > >>>>>>>>>>>>>>>>>> applied
> > > >>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > > >>>>>> config is
> > > >>>>>>>>>>>>>>>>>> unchanged.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include
> > > this
> > > >>>>>> change.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > > 於
> > > >>>>>>>>>>>>>> 2026年4月28日
> > > >>>>>>>>>>>>>>>>>> 晚上8:03
> > > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > > partition
> > > >>>>>> creation
> > > >>>>>>>>>>>>>>> time
> > > >>>>>>>>>>>>>>>>>> via
> > > >>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> > > users
> > > >>>> to
> > > >>>>>>>>>>>>>> diagnose
> > > >>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > >>>>>> auto.offset.reset.latest.max.age
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> > > on
> > > >>>>>> KIP-1327
> > > >>>>>>>>>>>>>>>>>> Prevent
> > > >>>>>>>>>>>>>>>>>>>>> Hot
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>
> > > >>>>
> > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> > > newly
> > > >>>>>> expanded
> > > >>>>>>>>>>>>>>>>> (hot)
> > > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > > threshold
> > > >>>>>>>>>>>>>>> automatically
> > > >>>>>>>>>>>>>>>>>> fall
> > > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> > > forcing
> > > >>>>>> a full
> > > >>>>>>>>>>>>>>>>>> historical
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> > 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
> JR5. Since this changes the on-disk data format for PartitionRecord, we
> need to gate this by a new MV. It also changes the format of
> ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if this
> requires a new MV or a new GV.

That is a good point. If we want to maintain consistency for the NewPartitions field when the group coordinator migrates between new and old binaries, a version gate is definitely required. For example:

GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())

The group coordinator should NOT set the NewPartitions field when the gv is smaller than gv_2

Best,
Chia-Ping

On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> Hi, Jiunn-Yang,
> 
> JR2. We should try to maintain consistency across our APIs. So, it would be
> useful to understand if the user-facing part of this KIP is what share
> consumer wants too. It would be problematic if we proceed with this
> approach only for the share consumer to choose a different one later.
> 
> JR3. I was thinking that if we add the NewPartitions field in
> GroupHeartbeatResponse, there is no need to expose the group/partition
> creation timestamp to other RPCs and the Cli.
> 
> JR5. Since this changes the on-disk data format for PartitionRecord, we
> need to gate this by a new MV. It also changes the format of
> ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if this
> requires a new MV or a new GV.
> 
> Thanks,
> 
> Jun
> 
> On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
> > Hello Jun, chia,
> >
> > Thanks for your feedback.
> >
> > JR3 JR4 I have updated the KIP.
> >
> > JR2 The new-partition detection mechanism is reusable. The coordinator
> > already has
> > access to both timestamps required for classification, so the
> > determination can be made
> > server-side without introducing any new RPCs. The main difference is where
> > the policy is
> > enforced: modern consumer resolves it on the client, whereas a share group
> > resolves the
> > starting offset on the partition leader. As a result, the classification
> > decision would need to
> > be propagated through the existing share-state initialization path. We
> > would also need to persist
> > the group's creation time as part of the share-group state, since that
> > information is not stored today.
> >
> > None of this is a blocker, but it does require changes to the share-state
> > format and introduces
> > a share-specific configuration. Given that additional scope, I would
> > prefer to keep share groups out
> > of scope for this KIP and address them in a follow-up effort, while
> > documenting the relevant reuse
> > opportunities here.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > >
> > >> It inspires me to think about mirror clusters. What if the mirror group
> > is created 'after' the expanded partitions? In this scenario, the new
> > configuration 'max.age.ms' could handle it if users are aware of the
> > migration and set a suitable time interval. Of course, this would require
> > both creation times to be exposed to clients.
> > >
> > > Thinking more about the mirror cluster scenario we discussed, I realize
> > it introduces significant complexity. Handling it properly would require a
> > tolerance time window rather than a strict time interval to account for
> > replication lag. To prevent scope creep and keep KIP-1327 focused on its
> > primary goal, I think it would be best to defer the multi-cluster/mirroring
> > scenario and explore those approaches in a separate KIP
> > >
> > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > >> hi all,
> > >>
> > >>> JR3. The KIP exposes the group/partition creation time to the client.
> > I am
> > >>> not sure if there are other use cases for those timestamps. An
> > alternative
> > >>> is to avoid exposing those timestamps to the client. The group
> > coordinator
> > >>> will categorize the partitions using those timestamps and include the
> > >>> categorization in the assigned partitions in GroupHeartbeatResponse.
> > >>
> > >> It inspires me to think about mirror clusters. What if the mirror group
> > is created 'after' the expanded partitions? In this scenario, the new
> > configuration 'max.age.ms' could handle it if users are aware of the
> > migration and set a suitable time interval. Of course, this would require
> > both creation times to be exposed to clients.
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > >>> Hi, Jiunn-Yang,
> > >>>
> > >>> Thanks for the updated KIP.
> > >>>
> > >>> JR2. Share consumer (KIP-932) states: "If the number of partitions is
> > >>> increased for a topic with a subscription in a share group, the SPSO
> > for
> > >>> the newly created share-partitions is initialized to 0 ". However, this
> > >>> part has not been implemented yet. Currently, all partitions without
> > offset
> > >>> follows the strategy given by share.auto.offset.reset. So, we have an
> > >>> opportunity to see if it's better to align this KIP's changes with the
> > >>> share consumer.
> > >>> JR2.1 The first part is whether customizing the offset for new
> > partitions
> > >>> to something other than 0 is useful in the share consumer. It seems
> > that
> > >>> the motivation in this KIP applies to share consumer too.
> > >>> JR2.2 The second part is how new partitions are detected. The mechanism
> > >>> introduced in this KIP seems quite effective. A group coordinator could
> > >>> pick up newly created partitions for an existing topic or newly created
> > >>> topics matching the regex. Those partitions are categorized as new
> > >>> partitions and will follow the strategy given by
> > >>> auto.offset.reset.new.partitions.
> > >>> This makes sense since those partitions typically don't have a large
> > >>> backlog. A consumer could also additionally subscribe to an existing
> > topic
> > >>> in an existing group. Those partitions will be categorized as existing
> > >>> partitions. This also makes sense since those partitions could have a
> > large
> > >>> backlog and it is better to follow the strategy given by
> > auto.offset.reset. It
> > >>> will be useful to think through whether the logic for categorizing new
> > >>> partitions should also be reused in the share consumer.
> > >>>
> > >>> JR3. The KIP exposes the group/partition creation time to the client.
> > I am
> > >>> not sure if there are other use cases for those timestamps. An
> > alternative
> > >>> is to avoid exposing those timestamps to the client. The group
> > coordinator
> > >>> will categorize the partitions using those timestamps and include the
> > >>> categorization in the assigned partitions in GroupHeartbeatResponse.
> > >>>
> > >>> JR4. It would be useful to support this feature on existing groups as
> > well.
> > >>> For example, any partition without a creation timestamp can always be
> > >>> categorized as an existing partition.
> > >>>
> > >>> Jun
> > >>>
> > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>
> > >>>> Hello Andrew,
> > >>>>
> > >>>> Thanks for your feedback,
> > >>>>>> I've just been re-reading this KIP to ensure that share group
> > support is
> > >>>>>> not needed. I've discovered that we made a mistake implementing
> > KIP-932
> > >>>> in
> > >>>>>> this area, so we'll rectify that and check whether it's really tight
> > >>>> enough
> > >>>>>> not to require KIP-1327.
> > >>>>
> > >>>> Could you walk us through the specific area where the mistake was
> > found?
> > >>>> That would
> > >>>> help us assess whether it naturally fits within the scope of KIP-1327
> > or
> > >>>> should be addressed
> > >>>> separately.
> > >>>>
> > >>>> AS1 I have updated the KIP
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> > >>>>>
> > >>>>> hi Andrew
> > >>>>>
> > >>>>>> I've discovered that we made a mistake implementing KIP-932 in this
> > >>>> area,
> > >>>>> so we'll rectify that and check whether it's really tight enough not
> > to
> > >>>>> require KIP-1327.
> > >>>>>
> > >>>>> Would you mind sharing more details about that? If it's not anything
> > >>>> close
> > >>>>> to NP-hard, we could definitely look into covering it within KIP-1327
> > >>>>>
> > >>>>> Best,
> > >>>>> Chia-Ping
> > >>>>>
> > >>>>>
> > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
> > >>>>>
> > >>>>>> Hi Jiunn-Yang,
> > >>>>>> I've just been re-reading this KIP to ensure that share group
> > support is
> > >>>>>> not needed. I've discovered that we made a mistake implementing
> > KIP-932
> > >>>> in
> > >>>>>> this area, so we'll rectify that and check whether it's really tight
> > >>>> enough
> > >>>>>> not to require KIP-1327.
> > >>>>>>
> > >>>>>> One trivial comments for consistency.
> > >>>>>>
> > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > represented by
> > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Andrew
> > >>>>>>
> > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> Manually bumping this thread.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>>>
> > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > >>>>>>>>
> > >>>>>>>> Hello chia,
> > >>>>>>>>
> > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Jiunn-Yang
> > >>>>>>>>
> > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> > >>>>>>>>>
> > >>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>
> > >>>>>>>>>> When the config is set on a cluster that has not yet been
> > >>>>>> upgraded... classification cannot occur... the consumer falls back
> > to
> > >>>> the
> > >>>>>> base auto.offset.reset for the affected partitions. No exception is
> > >>>> thrown,
> > >>>>>> and no operational disruption results.
> > >>>>>>>>>
> > >>>>>>>>> Existing group can't take advantage of this excellent new
> > >>>>>> configuration. Allowing users to modify the group creation time
> > might be
> > >>>>>> overkill. Instead, we could print a useful warning message to guide
> > >>>> users.
> > >>>>>> For example, we can suggest that they re-create the group with their
> > >>>>>> existing committed offsets
> > >>>>>>>>>
> > >>>>>>>>>> Protocol changes
> > >>>>>>>>>
> > >>>>>>>>> Would you mind listing those RPC changes in a table format?
> > >>>>>>>>>
> > >>>>>>>>>> The full interaction matrix between the base policy and the
> > >>>>>> new-partition policy is:
> > >>>>>>>>>
> > >>>>>>>>> Please add a filed to describe the target scenario when using
> > these
> > >>>>>> policies
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Chia-Ping
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > >>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for the new
> > >>>>>>>>>> approach, PTAL
> > >>>>>>>>>>
> > >>>>>>>>>> Best Regards,
> > >>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>
> > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> > >>>>>>>>>>>
> > >>>>>>>>>>> hi Jun
> > >>>>>>>>>>>
> > >>>>>>>>>>> Yes, your approach is great. I think the combination of latest
> > (for
> > >>>>>> existing partitions) and by_duration (for new partitions) can
> > address
> > >>>> 99%
> > >>>>>> of the complaints I have heard regarding this issue.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also, leveraging the group creation time here opens the door to
> > >>>>>> implementing a new policy based on timestamp seek in the future,
> > should
> > >>>> the
> > >>>>>> community want to pursue that.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for your patience and constructive feedback. We will
> > update
> > >>>>>> the KIP accordingly.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11
> > 寫道:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > configure a
> > >>>>>> different
> > >>>>>>>>>>>> offset policy for existing partitions vs new partitions.
> > However,
> > >>>>>> using
> > >>>>>>>>>>>> group creation time to capture that seems more intuitive.
> > Here is
> > >>>>>> another
> > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize
> > new
> > >>>>>> partitions
> > >>>>>>>>>>>> based on group creation time. Introduce
> > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose values
> > can be
> > >>>>>> earliest,
> > >>>>>>>>>>>> latest and by_duration, the same as auto.offset.reset. Users
> > can
> > >>>>>> set
> > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they
> > want to
> > >>>>>> guarantee
> > >>>>>>>>>>>> no data loss on new partitions. They can also use by_duration
> > to
> > >>>>>> set an
> > >>>>>>>>>>>> upper bound on the backlog replayed, which can be different
> > from
> > >>>>>> that of
> > >>>>>>>>>>>> the existing partitions. This will address your concern about
> > too
> > >>>>>> much
> > >>>>>>>>>>>> backlog being replayed when the offsets are lost. What do you
> > >>>>>> think?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > >>>>>> chia7712@gmail.com> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The most important part of this story is how users should
> > expect
> > >>>>>> the data
> > >>>>>>>>>>>>> they can see when using the latest or by_duration policy with
> > >>>>>> expanded
> > >>>>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > >>>>>>>>>>>>> non-deterministic, which means users will either read too
> > many
> > >>>>>> historical
> > >>>>>>>>>>>>> records from existing partitions or lose some records from
> > >>>>>> expanded
> > >>>>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > >>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>>>>>
> > >>>>>>>>>>>>> is a bit hard to understand, and that is why I preferred
> > having a
> > >>>>>> whole new
> > >>>>>>>>>>>>> policy based entirely on group creation time (KIP-1282)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > 上午1:08寫道:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand the
> > value
> > >>>>>> of the new
> > >>>>>>>>>>>>>> configs with the KIP.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want to
> > miss
> > >>>>>> the data if
> > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing partition
> > is
> > >>>>>> easy to
> > >>>>>>>>>>>>>> understand because it relates to retention time. The
> > backlog for
> > >>>>>> the new
> > >>>>>>>>>>>>>> partition is a bit subtle to understand since it depends on
> > the
> > >>>>>> metadata
> > >>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>>>>> ,
> > >>>>>>>>>>>>>> the user needs to
> > >>>>>>>>>>>>>> understand the metadata refresh delay on the consumer side
> > and
> > >>>>>> use it to
> > >>>>>>>>>>>>>> set the config.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the same value
> > for
> > >>>>>> the
> > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> > >>>> this
> > >>>>>>>>>>>>>> approach.
> > >>>>>>>>>>>>>> 1. It computes the seek target client-side as now() -
> > duration,
> > >>>>>> which
> > >>>>>>>>>>>>>> introduces clock skew across consumers and forces operators
> > to
> > >>>>>> choose
> > >>>>>>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
> > failed
> > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> > >>>>>> potentially
> > >>>>>>>>>>>>>> miss records produced between attempts.
> > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without committed
> > >>>>>> offsets, and
> > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > long-existing
> > >>>>>> partitions
> > >>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding
> > a bit
> > >>>>>> buffer to
> > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > improving the
> > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> > >>>>>> typically low
> > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer and tens
> > of
> > >>>>>> seconds
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading that much
> > >>>>>> backlog for new
> > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the same for
> > existing
> > >>>>>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> So, instead of introducing a new config, could we just
> > reuse the
> > >>>>>> existing
> > >>>>>>>>>>>>>> config with better documentation and/or implementation?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> You're right that group creation time is the more intuitive
> > >>>>>> answer at
> > >>>>>>>>>>>>>>> first glance,
> > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that
> > "predate
> > >>>>>> the group"
> > >>>>>>>>>>>>>>> vs partitions
> > >>>>>>>>>>>>>>> "created during group runtime," which directly points to a
> > >>>>>>>>>>>>>> group-lifecycle
> > >>>>>>>>>>>>>>> classifier.
> > >>>>>>>>>>>>>>> I'd like to walk through why we landed on partition age,
> > and
> > >>>> the
> > >>>>>>>>>>>>>>> trade-offs we considered.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This covers the metadata blindness window, but has issues
> > the
> > >>>>>> KIP
> > >>>>>>>>>>>>>>> currently documents
> > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
> > across
> > >>>>>> consumers.
> > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > potentially
> > >>>>>> missing
> > >>>>>>>>>>>>>>> records produced between
> > >>>>>>>>>>>>>>> attempts.
> > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without committed
> > >>>>>> offsets,
> > >>>>>>>>>>>>>>> including pre-existing partitions
> > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively running.
> > Our
> > >>>>>> concern
> > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> T=0:         Group created.
> > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > >>>>>>>>>>>>>>> T=50:        Partition added during the idle window.
> > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Under group creation time, the new partition is classified
> > as
> > >>>>>> new
> > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
> > from
> > >>>>>> T=50.
> > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> > >>>>>> data that
> > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the
> > contract
> > >>>> of
> > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason
> > to
> > >>>>>> treat
> > >>>>>>>>>>>>>>> the new partition differently; both contain backlog
> > accumulated
> > >>>>>> during
> > >>>>>>>>>>>>>>> the same idle window.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle you
> > raised
> > >>>>>> in
> > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
> > backlog
> > >>>>>> on
> > >>>>>>>>>>>>>>> every other partition during the same idle period; forcing
> > >>>>>> 0-backlog
> > >>>>>>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> > >>>> that
> > >>>>>>>>>>>>>>> tolerance.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent data loss
> > >>>> window,
> > >>>>>>>>>>>>>>> the gap between partition creation and the consumer’s
> > metadata
> > >>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely
> > silent: the
> > >>>>>>>>>>>>>>> consumer had no opportunity to know about the partition.
> > >>>>>> Outside this
> > >>>>>>>>>>>>>>> window, missing data reflects either:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> > consumers,
> > >>>>>> or
> > >>>>>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
> > via
> > >>>>>> reset
> > >>>>>>>>>>>>>> policy.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We did not choose partition age because it is more elegant
> > than
> > >>>>>> group
> > >>>>>>>>>>>>>>> creation time — we chose it because its failure mode
> > (requires
> > >>>> a
> > >>>>>>>>>>>>>>> threshold) is
> > >>>>>>>>>>>>>>> less invasive than the failure mode of group creation time
> > >>>>>> (overrides
> > >>>>>>>>>>>>>>> user-stated
> > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> > 寫道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Relying on both creation times will create an inconsistent
> > >>>>>> scenario. A
> > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
> > seek
> > >>>>>> to the
> > >>>>>>>>>>>>>>>> beginning for the partitions created later than the group.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> > >>>>>> inconsistency
> > >>>>>>>>>>>>>>> using a
> > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> > consensus,
> > >>>>>> KIP-1327
> > >>>>>>>>>>>>>>> goes
> > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent users
> > from
> > >>>>>> falling
> > >>>>>>>>>>>>>> into
> > >>>>>>>>>>>>>>>> that pitfall.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > >>>>>> 上午6:49寫道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>>>>>
> > >>>>>>>>>>>>>> still feels weird to
> > >>>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on the
> > >>>>>> partition
> > >>>>>>>>>>>>>>> creation
> > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be based on
> > the
> > >>>>>> group
> > >>>>>>>>>>>>>>> creation
> > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is created
> > are
> > >>>>>> existing
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> all partitions created after the group creation are new
> > >>>>>> partitions.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
> > task,
> > >>>>>> not an
> > >>>>>>>>>>>>>> edge
> > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
> > clarify
> > >>>>>> that
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which
> > further
> > >>>>>>>>>>>>>> demonstrates
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> > >>>>>> Kafka: To
> > >>>>>>>>>>>>>>>>> balance
> > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> > >>>>>> created with a
> > >>>>>>>>>>>>>>>>>> moderate
> > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic grows
> > over
> > >>>>>> time, it
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> often
> > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> > >>>>>> accommodate the
> > >>>>>>>>>>>>>>>>> higher
> > >>>>>>>>>>>>>>>>>>>> workload.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Regards
> > >>>>>>>>>>>>>>>>>>>> Jian
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> > 晚上8:29
> > >>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> > when
> > >>>>>>>>>>>>>> describing
> > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
> > them
> > >>>> to
> > >>>>>>>>>>>>>> describe
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> > >>>>>> applying
> > >>>>>>>>>>>>>> these
> > >>>>>>>>>>>>>>>>>> terms
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> > >>>>>> whether a
> > >>>>>>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is
> > that
> > >>>>>> they won't
> > >>>>>>>>>>>>>>>>>> silently
> > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
> > their
> > >>>>>> active
> > >>>>>>>>>>>>>>>>>>>>> consumption.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > >>>>>> motivation
> > >>>>>>>>>>>>>>> section.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > 2026年5月30日
> > >>>>>> 凌晨1:12
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > >>>>>> motivation
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> > >>>>>> partitions
> > >>>>>>>>>>>>>> without
> > >>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> > >>>>>> partition
> > >>>>>>>>>>>>>> (hot)
> > >>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> > consumer
> > >>>>>> has
> > >>>>>>>>>>>>>> never
> > >>>>>>>>>>>>>>>>> seen
> > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
> > hot
> > >>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> > also
> > >>>>>>>>>>>>>>> insufficient
> > >>>>>>>>>>>>>>>>>>>>>>> because:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
> > varies
> > >>>>>> across
> > >>>>>>>>>>>>>> nodes
> > >>>>>>>>>>>>>>>>>> due
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> > large
> > >>>>>>>>>>>>>> duration,
> > >>>>>>>>>>>>>>>>>>>>>>> causing
> > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
> > seek
> > >>>>>> time on
> > >>>>>>>>>>>>>>>>> retry,
> > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
> > data
> > >>>>>> loss."
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
> > these
> > >>>>>> issues
> > >>>>>>>>>>>>>>>>> persist,
> > >>>>>>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > >>>>>> situations don't
> > >>>>>>>>>>>>>>>>> need a
> > >>>>>>>>>>>>>>>>>>>>>>> common
> > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > situations,
> > >>>>>> they can
> > >>>>>>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > >>>>>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > >>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > >>>>>> should
> > >>>>>>>>>>>>>> always
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >>>>>> any
> > >>>>>>>>>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
> > version
> > >>>>>>>>>>>>>> mismatch
> > >>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
> > occurs
> > >>>>>> inside
> > >>>>>>>>>>>>>>>>> poll().
> > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > UnsupportedVersionException
> > >>>>>> from
> > >>>>>>>>>>>>>> poll().
> > >>>>>>>>>>>>>>>>>> I’ll
> > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2026年5月17日
> > >>>>>> 下午4:50
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> > this
> > >>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> > as
> > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > >>>>>>>>>>>>>>>>>>>>>>> -
> > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > >>>>>> does not
> > >>>>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is
> > unknown.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > >>>>>> should
> > >>>>>>>>>>>>>> always
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >>>>>> any
> > >>>>>>>>>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> > when
> > >>>>>> calling
> > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> > based on
> > >>>>>> the
> > >>>>>>>>>>>>>>>>>> discussion
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > >>>> reset
> > >>>>>>>>>>>>>> behavior
> > >>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>>>>>
> > >>>>>>>>>>>>>> now applies to all
> > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> > resets
> > >>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>>
> > >>>>>>>>>>>>>> .partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > >>>>>>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > represented
> > >>>>>> by a
> > >>>>>>>>>>>>>>> separate
> > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions,
> > >>>>>>>>>>>>>> currently fixed to
> > >>>>>>>>>>>>>>>>> earliest).
> > >>>>>>>>>>>>>>>>>>>>> This
> > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > >>>>>> public
> > >>>>>>>>>>>>>>>>>> user-facing
> > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2026年5月16日
> > >>>>>> 清晨7:46
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> > is
> > >>>>>> listed
> > >>>>>>>>>>>>>>> below:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>> with a default value
> > >>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > >>>>>> by_duration and
> > >>>>>>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > >>>>>> auto.offset.reset=earliest.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>> as an internal
> > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>>
> > >>>>>>>>>>>>>> .partitions=earliest
> > >>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> > cases
> > >>>>>> of
> > >>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>>>>>> in a
> > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > >>>>>>> ,
> > >>>>>>>>>>>>>> will be applied to all for
> > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> > "When
> > >>>>>> a Kafka
> > >>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> > latest
> > >>>>>> auto
> > >>>>>>>>>>>>>> offset
> > >>>>>>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
> > those
> > >>>>>>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>>> before
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > >>>>>> record loss
> > >>>>>>>>>>>>>>>>> issue
> > >>>>>>>>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>> take
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> > set
> > >>>>>> it
> > >>>>>>>>>>>>>>>>>> by_duration if
> > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> > Tsai <
> > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > >>>>>> something
> > >>>>>>>>>>>>>>>>>> important
> > >>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> > to
> > >>>>>> clarify
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>> few
> > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> > this
> > >>>>>> logic
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > earliest
> > >>>>>> or
> > >>>>>>>>>>>>>>>>> by_duration
> > >>>>>>>>>>>>>>>>>>>>>>> don't
> > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> > issue
> > >>>>>> when a
> > >>>>>>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > >>>>>> configure for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions?
> > >>>>>>>>>>>>>> If they set it to
> > >>>>>>>>>>>>>>>>> earliest
> > >>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge
> > cases. For
> > >>>>>>>>>>>>>> example,
> > >>>>>>>>>>>>>>>>> if a
> > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > >>>>>> created
> > >>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> > latest
> > >>>>>> when
> > >>>>>>>>>>>>>>>>> resuming,
> > >>>>>>>>>>>>>>>>>>>>>>> rather
> > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > partition
> > >>>>>> is
> > >>>>>>>>>>>>>>>>>> technically
> > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > introducing a
> > >>>>>> max.age
> > >>>>>>>>>>>>>>>>>> threshold.
> > >>>>>>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > >>>>>> genuinely
> > >>>>>>>>>>>>>>> "hot/new"
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev
> > wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> > bit
> > >>>>>> weird. It
> > >>>>>>>>>>>>>>>>> only
> > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> > seems
> > >>>>>> that the
> > >>>>>>>>>>>>>>>>>>>>> motivation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> > other
> > >>>>>> values
> > >>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a
> > separate
> > >>>>>> way to
> > >>>>>>>>>>>>>>> control
> > >>>>>>>>>>>>>>>>>>>>> newly
> > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > >>>> group
> > >>>>>>>>>>>>>> starts.
> > >>>>>>>>>>>>>>>>>> Have we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>>
> > >>>>>>>>>>>>>>> .partitions?
> > >>>>>>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > >>>>>> defaults to
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > >>>>>> explicitly
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> customize
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > >>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> > 晚上10:37
> > >>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> > The
> > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > >>>>>> computed
> > >>>>>>>>>>>>>>>>>> server-side
> > >>>>>>>>>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > >>>> partition_creation_time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow.
> > when
> > >>>>>>>>>>>>>>>>>> MembershipManager
> > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > >>>>>> metadata for
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> affected
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> > even
> > >>>>>> if the
> > >>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>> ID
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > >>>>>> supported
> > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> > time.
> > >>>>>> If the
> > >>>>>>>>>>>>>>>>>> negotiated
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > >>>>>> PartitionAgeMs at
> > >>>>>>>>>>>>>> all
> > >>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > >>>>>> back to
> > >>>>>>>>>>>>>> latest
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> > signal.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the
> > KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > >>>>>> 2026年4月29日
> > >>>>>>>>>>>>>>> 下午4:04
> > >>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > >>>>>> 'age'
> > >>>>>>>>>>>>>>>>> resolution
> > >>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> > the
> > >>>>>> control
> > >>>>>>>>>>>>>>>>> plane
> > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > introducing
> > >>>>>>>>>>>>>>> inter-broker
> > >>>>>>>>>>>>>>>>>>>>> clock
> > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> > single
> > >>>>>> source of
> > >>>>>>>>>>>>>>>>> truth
> > >>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > >>>>>> should be
> > >>>>>>>>>>>>>>>>>> negligible.
> > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > >>>>>> configured in
> > >>>>>>>>>>>>>>> minutes
> > >>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > >>>>>> brokers
> > >>>>>>>>>>>>>> won't
> > >>>>>>>>>>>>>>>>>> impact
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > dev@kafka.apache.org>
> > >>>>>> 於
> > >>>>>>>>>>>>>>>>> 2026年4月29日
> > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > >>>>>> previous
> > >>>>>>>>>>>>>>>>>> conversation
> > >>>>>>>>>>>>>>>>>>>>>>> but I
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> > flow
> > >>>>>> with the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> > new
> > >>>>>> boolean;
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> > this
> > >>>>>> boolean
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> set.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives
> > a new
> > >>>>>>>>>>>>>> partition,
> > >>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> > receive
> > >>>>>> the
> > >>>>>>>>>>>>>> ages.
> > >>>>>>>>>>>>>>> Is
> > >>>>>>>>>>>>>>>>>> my
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> > flow
> > >>>>>> and
> > >>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>> explain
> > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > offsets,
> > >>>>>> fetch
> > >>>>>>>>>>>>>>>>> offsets,
> > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > >>>>>> wonder if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> > right
> > >>>>>> place
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>> given
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > Alternatively,
> > >>>>>> it could
> > >>>>>>>>>>>>>>>>> simply
> > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be
> > rather
> > >>>>>> cautious
> > >>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > >>>>>> unrelated
> > >>>>>>>>>>>>>>> concepts.
> > >>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>> API
> > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> > revoking
> > >>>>>>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>> fact
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > corresponding
> > >>>>>> Streams
> > >>>>>>>>>>>>>> API
> > >>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> > would we
> > >>>>>> do if
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>> want to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > >>>>>> Muralidhar Basani
> > >>>>>>>>>>>>>>> via
> > >>>>>>>>>>>>>>>>>> dev
> > >>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> > know
> > >>>>>> about
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> gap.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > >>>>>> PartitionAges
> > >>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > >>>>>> recommended
> > >>>>>>>>>>>>>>>>> value
> > >>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based
> > on the
> > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > >>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > >>>>>> guess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no
> > changes to
> > >>>>>> Kafka
> > >>>>>>>>>>>>>>>>> Streams,
> > >>>>>>>>>>>>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> > it?
> > >>>>>> This is
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > >>>>>> mistakenly
> > >>>>>>>>>>>>>>>>>> configure
> > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > consumer
> > >>>>>> falls
> > >>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > >>>>>> per-partition
> > >>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>> isn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > "consumer
> > >>>>>> resolves
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> initial
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > >>>>>> earliest
> > >>>>>>>>>>>>>> was
> > >>>>>>>>>>>>>>>>>> applied
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > >>>>>> config is
> > >>>>>>>>>>>>>>>>>> unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include
> > this
> > >>>>>> change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > 於
> > >>>>>>>>>>>>>> 2026年4月28日
> > >>>>>>>>>>>>>>>>>> 晚上8:03
> > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > partition
> > >>>>>> creation
> > >>>>>>>>>>>>>>> time
> > >>>>>>>>>>>>>>>>>> via
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> > users
> > >>>> to
> > >>>>>>>>>>>>>> diagnose
> > >>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > >>>>>> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> > on
> > >>>>>> KIP-1327
> > >>>>>>>>>>>>>>>>>> Prevent
> > >>>>>>>>>>>>>>>>>>>>> Hot
> > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> > newly
> > >>>>>> expanded
> > >>>>>>>>>>>>>>>>> (hot)
> > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > threshold
> > >>>>>>>>>>>>>>> automatically
> > >>>>>>>>>>>>>>>>>> fall
> > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> > forcing
> > >>>>>> a full
> > >>>>>>>>>>>>>>>>>> historical
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
> JR5. Since this changes the on-disk data format for PartitionRecord, we
> need to gate this by a new MV. It also changes the format of
> ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if this
> requires a new MV or a new GV.

That is a good point. If we want to maintain consistency for the NewPartitions field when the group coordinator migrates between new and old binaries, a version gate is definitely required. For example:

GV_2(2, MetadataVersion.IBP_4_4_IV0, Map.of())

The group coordinator should NOT set the NewPartitions field when the gv is smaller than gv_2

Best,
Chia-Ping

On 2026/06/25 20:01:39 Jun Rao via dev wrote:
> Hi, Jiunn-Yang,
> 
> JR2. We should try to maintain consistency across our APIs. So, it would be
> useful to understand if the user-facing part of this KIP is what share
> consumer wants too. It would be problematic if we proceed with this
> approach only for the share consumer to choose a different one later.
> 
> JR3. I was thinking that if we add the NewPartitions field in
> GroupHeartbeatResponse, there is no need to expose the group/partition
> creation timestamp to other RPCs and the Cli.
> 
> JR5. Since this changes the on-disk data format for PartitionRecord, we
> need to gate this by a new MV. It also changes the format of
> ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if this
> requires a new MV or a new GV.
> 
> Thanks,
> 
> Jun
> 
> On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
> > Hello Jun, chia,
> >
> > Thanks for your feedback.
> >
> > JR3 JR4 I have updated the KIP.
> >
> > JR2 The new-partition detection mechanism is reusable. The coordinator
> > already has
> > access to both timestamps required for classification, so the
> > determination can be made
> > server-side without introducing any new RPCs. The main difference is where
> > the policy is
> > enforced: modern consumer resolves it on the client, whereas a share group
> > resolves the
> > starting offset on the partition leader. As a result, the classification
> > decision would need to
> > be propagated through the existing share-state initialization path. We
> > would also need to persist
> > the group's creation time as part of the share-group state, since that
> > information is not stored today.
> >
> > None of this is a blocker, but it does require changes to the share-state
> > format and introduces
> > a share-specific configuration. Given that additional scope, I would
> > prefer to keep share groups out
> > of scope for this KIP and address them in a follow-up effort, while
> > documenting the relevant reuse
> > opportunities here.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> > >
> > >> It inspires me to think about mirror clusters. What if the mirror group
> > is created 'after' the expanded partitions? In this scenario, the new
> > configuration 'max.age.ms' could handle it if users are aware of the
> > migration and set a suitable time interval. Of course, this would require
> > both creation times to be exposed to clients.
> > >
> > > Thinking more about the mirror cluster scenario we discussed, I realize
> > it introduces significant complexity. Handling it properly would require a
> > tolerance time window rather than a strict time interval to account for
> > replication lag. To prevent scope creep and keep KIP-1327 focused on its
> > primary goal, I think it would be best to defer the multi-cluster/mirroring
> > scenario and explore those approaches in a separate KIP
> > >
> > > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> > >> hi all,
> > >>
> > >>> JR3. The KIP exposes the group/partition creation time to the client.
> > I am
> > >>> not sure if there are other use cases for those timestamps. An
> > alternative
> > >>> is to avoid exposing those timestamps to the client. The group
> > coordinator
> > >>> will categorize the partitions using those timestamps and include the
> > >>> categorization in the assigned partitions in GroupHeartbeatResponse.
> > >>
> > >> It inspires me to think about mirror clusters. What if the mirror group
> > is created 'after' the expanded partitions? In this scenario, the new
> > configuration 'max.age.ms' could handle it if users are aware of the
> > migration and set a suitable time interval. Of course, this would require
> > both creation times to be exposed to clients.
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > >>> Hi, Jiunn-Yang,
> > >>>
> > >>> Thanks for the updated KIP.
> > >>>
> > >>> JR2. Share consumer (KIP-932) states: "If the number of partitions is
> > >>> increased for a topic with a subscription in a share group, the SPSO
> > for
> > >>> the newly created share-partitions is initialized to 0 ". However, this
> > >>> part has not been implemented yet. Currently, all partitions without
> > offset
> > >>> follows the strategy given by share.auto.offset.reset. So, we have an
> > >>> opportunity to see if it's better to align this KIP's changes with the
> > >>> share consumer.
> > >>> JR2.1 The first part is whether customizing the offset for new
> > partitions
> > >>> to something other than 0 is useful in the share consumer. It seems
> > that
> > >>> the motivation in this KIP applies to share consumer too.
> > >>> JR2.2 The second part is how new partitions are detected. The mechanism
> > >>> introduced in this KIP seems quite effective. A group coordinator could
> > >>> pick up newly created partitions for an existing topic or newly created
> > >>> topics matching the regex. Those partitions are categorized as new
> > >>> partitions and will follow the strategy given by
> > >>> auto.offset.reset.new.partitions.
> > >>> This makes sense since those partitions typically don't have a large
> > >>> backlog. A consumer could also additionally subscribe to an existing
> > topic
> > >>> in an existing group. Those partitions will be categorized as existing
> > >>> partitions. This also makes sense since those partitions could have a
> > large
> > >>> backlog and it is better to follow the strategy given by
> > auto.offset.reset. It
> > >>> will be useful to think through whether the logic for categorizing new
> > >>> partitions should also be reused in the share consumer.
> > >>>
> > >>> JR3. The KIP exposes the group/partition creation time to the client.
> > I am
> > >>> not sure if there are other use cases for those timestamps. An
> > alternative
> > >>> is to avoid exposing those timestamps to the client. The group
> > coordinator
> > >>> will categorize the partitions using those timestamps and include the
> > >>> categorization in the assigned partitions in GroupHeartbeatResponse.
> > >>>
> > >>> JR4. It would be useful to support this feature on existing groups as
> > well.
> > >>> For example, any partition without a creation timestamp can always be
> > >>> categorized as an existing partition.
> > >>>
> > >>> Jun
> > >>>
> > >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>
> > >>>> Hello Andrew,
> > >>>>
> > >>>> Thanks for your feedback,
> > >>>>>> I've just been re-reading this KIP to ensure that share group
> > support is
> > >>>>>> not needed. I've discovered that we made a mistake implementing
> > KIP-932
> > >>>> in
> > >>>>>> this area, so we'll rectify that and check whether it's really tight
> > >>>> enough
> > >>>>>> not to require KIP-1327.
> > >>>>
> > >>>> Could you walk us through the specific area where the mistake was
> > found?
> > >>>> That would
> > >>>> help us assess whether it naturally fits within the scope of KIP-1327
> > or
> > >>>> should be addressed
> > >>>> separately.
> > >>>>
> > >>>> AS1 I have updated the KIP
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> > >>>>>
> > >>>>> hi Andrew
> > >>>>>
> > >>>>>> I've discovered that we made a mistake implementing KIP-932 in this
> > >>>> area,
> > >>>>> so we'll rectify that and check whether it's really tight enough not
> > to
> > >>>>> require KIP-1327.
> > >>>>>
> > >>>>> Would you mind sharing more details about that? If it's not anything
> > >>>> close
> > >>>>> to NP-hard, we could definitely look into covering it within KIP-1327
> > >>>>>
> > >>>>> Best,
> > >>>>> Chia-Ping
> > >>>>>
> > >>>>>
> > >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
> > >>>>>
> > >>>>>> Hi Jiunn-Yang,
> > >>>>>> I've just been re-reading this KIP to ensure that share group
> > support is
> > >>>>>> not needed. I've discovered that we made a mistake implementing
> > KIP-932
> > >>>> in
> > >>>>>> this area, so we'll rectify that and check whether it's really tight
> > >>>> enough
> > >>>>>> not to require KIP-1327.
> > >>>>>>
> > >>>>>> One trivial comments for consistency.
> > >>>>>>
> > >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> > represented by
> > >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Andrew
> > >>>>>>
> > >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> Manually bumping this thread.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>>>
> > >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > >>>>>>>>
> > >>>>>>>> Hello chia,
> > >>>>>>>>
> > >>>>>>>> Thanks for the feedback, I have updated the KIP.
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Jiunn-Yang
> > >>>>>>>>
> > >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> > >>>>>>>>>
> > >>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>
> > >>>>>>>>>> When the config is set on a cluster that has not yet been
> > >>>>>> upgraded... classification cannot occur... the consumer falls back
> > to
> > >>>> the
> > >>>>>> base auto.offset.reset for the affected partitions. No exception is
> > >>>> thrown,
> > >>>>>> and no operational disruption results.
> > >>>>>>>>>
> > >>>>>>>>> Existing group can't take advantage of this excellent new
> > >>>>>> configuration. Allowing users to modify the group creation time
> > might be
> > >>>>>> overkill. Instead, we could print a useful warning message to guide
> > >>>> users.
> > >>>>>> For example, we can suggest that they re-create the group with their
> > >>>>>> existing committed offsets
> > >>>>>>>>>
> > >>>>>>>>>> Protocol changes
> > >>>>>>>>>
> > >>>>>>>>> Would you mind listing those RPC changes in a table format?
> > >>>>>>>>>
> > >>>>>>>>>> The full interaction matrix between the base policy and the
> > >>>>>> new-partition policy is:
> > >>>>>>>>>
> > >>>>>>>>> Please add a filed to describe the target scenario when using
> > these
> > >>>>>> policies
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Chia-Ping
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > >>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the feedback, I have updated the KIP for the new
> > >>>>>>>>>> approach, PTAL
> > >>>>>>>>>>
> > >>>>>>>>>> Best Regards,
> > >>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>
> > >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> > >>>>>>>>>>>
> > >>>>>>>>>>> hi Jun
> > >>>>>>>>>>>
> > >>>>>>>>>>> Yes, your approach is great. I think the combination of latest
> > (for
> > >>>>>> existing partitions) and by_duration (for new partitions) can
> > address
> > >>>> 99%
> > >>>>>> of the complaints I have heard regarding this issue.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also, leveraging the group creation time here opens the door to
> > >>>>>> implementing a new policy based on timestamp seek in the future,
> > should
> > >>>> the
> > >>>>>> community want to pursue that.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for your patience and constructive feedback. We will
> > update
> > >>>>>> the KIP accordingly.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11
> > 寫道:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> > configure a
> > >>>>>> different
> > >>>>>>>>>>>> offset policy for existing partitions vs new partitions.
> > However,
> > >>>>>> using
> > >>>>>>>>>>>> group creation time to capture that seems more intuitive.
> > Here is
> > >>>>>> another
> > >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize
> > new
> > >>>>>> partitions
> > >>>>>>>>>>>> based on group creation time. Introduce
> > >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose values
> > can be
> > >>>>>> earliest,
> > >>>>>>>>>>>> latest and by_duration, the same as auto.offset.reset. Users
> > can
> > >>>>>> set
> > >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they
> > want to
> > >>>>>> guarantee
> > >>>>>>>>>>>> no data loss on new partitions. They can also use by_duration
> > to
> > >>>>>> set an
> > >>>>>>>>>>>> upper bound on the backlog replayed, which can be different
> > from
> > >>>>>> that of
> > >>>>>>>>>>>> the existing partitions. This will address your concern about
> > too
> > >>>>>> much
> > >>>>>>>>>>>> backlog being replayed when the offsets are lost. What do you
> > >>>>>> think?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > >>>>>> chia7712@gmail.com> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> The most important part of this story is how users should
> > expect
> > >>>>>> the data
> > >>>>>>>>>>>>> they can see when using the latest or by_duration policy with
> > >>>>>> expanded
> > >>>>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > >>>>>>>>>>>>> non-deterministic, which means users will either read too
> > many
> > >>>>>> historical
> > >>>>>>>>>>>>> records from existing partitions or lose some records from
> > >>>>>> expanded
> > >>>>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > >>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>>>>>
> > >>>>>>>>>>>>> is a bit hard to understand, and that is why I preferred
> > having a
> > >>>>>> whole new
> > >>>>>>>>>>>>> policy based entirely on group creation time (KIP-1282)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> > 上午1:08寫道:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand the
> > value
> > >>>>>> of the new
> > >>>>>>>>>>>>>> configs with the KIP.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want to
> > miss
> > >>>>>> the data if
> > >>>>>>>>>>>>>> the backlog is small. The backlog of the existing partition
> > is
> > >>>>>> easy to
> > >>>>>>>>>>>>>> understand because it relates to retention time. The
> > backlog for
> > >>>>>> the new
> > >>>>>>>>>>>>>> partition is a bit subtle to understand since it depends on
> > the
> > >>>>>> metadata
> > >>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>>>>> ,
> > >>>>>>>>>>>>>> the user needs to
> > >>>>>>>>>>>>>> understand the metadata refresh delay on the consumer side
> > and
> > >>>>>> use it to
> > >>>>>>>>>>>>>> set the config.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Now, let's consider the alternative: setting the same value
> > for
> > >>>>>> the
> > >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> > >>>> this
> > >>>>>>>>>>>>>> approach.
> > >>>>>>>>>>>>>> 1. It computes the seek target client-side as now() -
> > duration,
> > >>>>>> which
> > >>>>>>>>>>>>>> introduces clock skew across consumers and forces operators
> > to
> > >>>>>> choose
> > >>>>>>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> > >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
> > failed
> > >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> > >>>>>> potentially
> > >>>>>>>>>>>>>> miss records produced between attempts.
> > >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without committed
> > >>>>>> offsets, and
> > >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> > long-existing
> > >>>>>> partitions
> > >>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding
> > a bit
> > >>>>>> buffer to
> > >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> > improving the
> > >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> > >>>>>> typically low
> > >>>>>>>>>>>>>> (in the order of minutes with the classic consumer and tens
> > of
> > >>>>>> seconds
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>> the new consumer). If a user is ok with reading that much
> > >>>>>> backlog for new
> > >>>>>>>>>>>>>> partitions, it seems they will be ok doing the same for
> > existing
> > >>>>>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> So, instead of introducing a new config, could we just
> > reuse the
> > >>>>>> existing
> > >>>>>>>>>>>>>> config with better documentation and/or implementation?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> You're right that group creation time is the more intuitive
> > >>>>>> answer at
> > >>>>>>>>>>>>>>> first glance,
> > >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that
> > "predate
> > >>>>>> the group"
> > >>>>>>>>>>>>>>> vs partitions
> > >>>>>>>>>>>>>>> "created during group runtime," which directly points to a
> > >>>>>>>>>>>>>> group-lifecycle
> > >>>>>>>>>>>>>>> classifier.
> > >>>>>>>>>>>>>>> I'd like to walk through why we landed on partition age,
> > and
> > >>>> the
> > >>>>>>>>>>>>>>> trade-offs we considered.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We evaluated three candidate signals:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This covers the metadata blindness window, but has issues
> > the
> > >>>>>> KIP
> > >>>>>>>>>>>>>>> currently documents
> > >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
> > across
> > >>>>>> consumers.
> > >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> > potentially
> > >>>>>> missing
> > >>>>>>>>>>>>>>> records produced between
> > >>>>>>>>>>>>>>> attempts.
> > >>>>>>>>>>>>>>> - It applies uniformly to all partitions without committed
> > >>>>>> offsets,
> > >>>>>>>>>>>>>>> including pre-existing partitions
> > >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2. Group creation time as classifier
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This works cleanly when the consumer is actively running.
> > Our
> > >>>>>> concern
> > >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> T=0:         Group created.
> > >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > >>>>>>>>>>>>>>> T=50:        Partition added during the idle window.
> > >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Under group creation time, the new partition is classified
> > as
> > >>>>>> new
> > >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
> > from
> > >>>>>> T=50.
> > >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> > >>>>>> data that
> > >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the
> > contract
> > >>>> of
> > >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason
> > to
> > >>>>>> treat
> > >>>>>>>>>>>>>>> the new partition differently; both contain backlog
> > accumulated
> > >>>>>> during
> > >>>>>>>>>>>>>>> the same idle window.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle you
> > raised
> > >>>>>> in
> > >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
> > backlog
> > >>>>>> on
> > >>>>>>>>>>>>>>> every other partition during the same idle period; forcing
> > >>>>>> 0-backlog
> > >>>>>>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> > >>>> that
> > >>>>>>>>>>>>>>> tolerance.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 3. Partition age vs threshold
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Partition age corresponds to the actual silent data loss
> > >>>> window,
> > >>>>>>>>>>>>>>> the gap between partition creation and the consumer’s
> > metadata
> > >>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely
> > silent: the
> > >>>>>>>>>>>>>>> consumer had no opportunity to know about the partition.
> > >>>>>> Outside this
> > >>>>>>>>>>>>>>> window, missing data reflects either:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> > consumers,
> > >>>>>> or
> > >>>>>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
> > via
> > >>>>>> reset
> > >>>>>>>>>>>>>> policy.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> We did not choose partition age because it is more elegant
> > than
> > >>>>>> group
> > >>>>>>>>>>>>>>> creation time — we chose it because its failure mode
> > (requires
> > >>>> a
> > >>>>>>>>>>>>>>> threshold) is
> > >>>>>>>>>>>>>>> less invasive than the failure mode of group creation time
> > >>>>>> (overrides
> > >>>>>>>>>>>>>>> user-stated
> > >>>>>>>>>>>>>>> `latest` intent during idle periods).
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> > 寫道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Relying on both creation times will create an inconsistent
> > >>>>>> scenario. A
> > >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
> > seek
> > >>>>>> to the
> > >>>>>>>>>>>>>>>> beginning for the partitions created later than the group.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> > >>>>>> inconsistency
> > >>>>>>>>>>>>>>> using a
> > >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> > consensus,
> > >>>>>> KIP-1327
> > >>>>>>>>>>>>>>> goes
> > >>>>>>>>>>>>>>>> back to using flexible configurations to prevent users
> > from
> > >>>>>> falling
> > >>>>>>>>>>>>>> into
> > >>>>>>>>>>>>>>>> that pitfall.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > >>>>>> 上午6:49寫道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>>>>>
> > >>>>>>>>>>>>>> still feels weird to
> > >>>>>>>>>>>>>>> me.
> > >>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on the
> > >>>>>> partition
> > >>>>>>>>>>>>>>> creation
> > >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be based on
> > the
> > >>>>>> group
> > >>>>>>>>>>>>>>> creation
> > >>>>>>>>>>>>>>>>> time: all partitions existing when the group is created
> > are
> > >>>>>> existing
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> all partitions created after the group creation are new
> > >>>>>> partitions.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> > >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hello Jian,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
> > task,
> > >>>>>> not an
> > >>>>>>>>>>>>>> edge
> > >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
> > clarify
> > >>>>>> that
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which
> > further
> > >>>>>>>>>>>>>> demonstrates
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> > >>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> > >>>>>> Kafka: To
> > >>>>>>>>>>>>>>>>> balance
> > >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> > >>>>>> created with a
> > >>>>>>>>>>>>>>>>>> moderate
> > >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic grows
> > over
> > >>>>>> time, it
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> often
> > >>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> > >>>>>> accommodate the
> > >>>>>>>>>>>>>>>>> higher
> > >>>>>>>>>>>>>>>>>>>> workload.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Regards
> > >>>>>>>>>>>>>>>>>>>> Jian
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> > 晚上8:29
> > >>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> > when
> > >>>>>>>>>>>>>> describing
> > >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
> > them
> > >>>> to
> > >>>>>>>>>>>>>> describe
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> > >>>>>> applying
> > >>>>>>>>>>>>>> these
> > >>>>>>>>>>>>>>>>>> terms
> > >>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> > >>>>>> whether a
> > >>>>>>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is
> > that
> > >>>>>> they won't
> > >>>>>>>>>>>>>>>>>> silently
> > >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
> > their
> > >>>>>> active
> > >>>>>>>>>>>>>>>>>>>>> consumption.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > >>>>>> motivation
> > >>>>>>>>>>>>>>> section.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> > 2026年5月30日
> > >>>>>> 凌晨1:12
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > >>>>>> motivation
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> > >>>>>> partitions
> > >>>>>>>>>>>>>> without
> > >>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> > >>>>>> partition
> > >>>>>>>>>>>>>> (hot)
> > >>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> > consumer
> > >>>>>> has
> > >>>>>>>>>>>>>> never
> > >>>>>>>>>>>>>>>>> seen
> > >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
> > hot
> > >>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> > also
> > >>>>>>>>>>>>>>> insufficient
> > >>>>>>>>>>>>>>>>>>>>>>> because:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
> > varies
> > >>>>>> across
> > >>>>>>>>>>>>>> nodes
> > >>>>>>>>>>>>>>>>>> due
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> > large
> > >>>>>>>>>>>>>> duration,
> > >>>>>>>>>>>>>>>>>>>>>>> causing
> > >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
> > seek
> > >>>>>> time on
> > >>>>>>>>>>>>>>>>> retry,
> > >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
> > data
> > >>>>>> loss."
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
> > these
> > >>>>>> issues
> > >>>>>>>>>>>>>>>>> persist,
> > >>>>>>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > >>>>>> situations don't
> > >>>>>>>>>>>>>>>>> need a
> > >>>>>>>>>>>>>>>>>>>>>>> common
> > >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> > situations,
> > >>>>>> they can
> > >>>>>>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> > >>>>>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > >>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > >>>>>> should
> > >>>>>>>>>>>>>> always
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >>>>>> any
> > >>>>>>>>>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
> > version
> > >>>>>>>>>>>>>> mismatch
> > >>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
> > occurs
> > >>>>>> inside
> > >>>>>>>>>>>>>>>>> poll().
> > >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> > >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> > UnsupportedVersionException
> > >>>>>> from
> > >>>>>>>>>>>>>> poll().
> > >>>>>>>>>>>>>>>>>> I’ll
> > >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2026年5月17日
> > >>>>>> 下午4:50
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> > this
> > >>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> > as
> > >>>>>>>>>>>>>>>>>>>>> broker_current_time
> > >>>>>>>>>>>>>>>>>>>>>>> -
> > >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > >>>>>> does not
> > >>>>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is
> > unknown.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > >>>>>> should
> > >>>>>>>>>>>>>> always
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >>>>>> any
> > >>>>>>>>>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> > when
> > >>>>>> calling
> > >>>>>>>>>>>>>>>>>>>>> `poll()`?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> > based on
> > >>>>>> the
> > >>>>>>>>>>>>>>>>>> discussion
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > >>>> reset
> > >>>>>>>>>>>>>> behavior
> > >>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>>>>>
> > >>>>>>>>>>>>>> now applies to all
> > >>>>>>>>>>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> > resets
> > >>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>>
> > >>>>>>>>>>>>>> .partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > >>>>>>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>>>>> continues
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> > represented
> > >>>>>> by a
> > >>>>>>>>>>>>>>> separate
> > >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions,
> > >>>>>>>>>>>>>> currently fixed to
> > >>>>>>>>>>>>>>>>> earliest).
> > >>>>>>>>>>>>>>>>>>>>> This
> > >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > >>>>>> public
> > >>>>>>>>>>>>>>>>>> user-facing
> > >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > 2026年5月16日
> > >>>>>> 清晨7:46
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> > is
> > >>>>>> listed
> > >>>>>>>>>>>>>>> below:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>> with a default value
> > >>>>>>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > >>>>>> by_duration and
> > >>>>>>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > >>>>>> auto.offset.reset=earliest.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>> as an internal
> > >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>>
> > >>>>>>>>>>>>>> .partitions=earliest
> > >>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> > cases
> > >>>>>> of
> > >>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>>>>>> in a
> > >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > >>>>>>> ,
> > >>>>>>>>>>>>>> will be applied to all for
> > >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> > "When
> > >>>>>> a Kafka
> > >>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> > latest
> > >>>>>> auto
> > >>>>>>>>>>>>>> offset
> > >>>>>>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
> > those
> > >>>>>>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>>> before
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > >>>>>> record loss
> > >>>>>>>>>>>>>>>>> issue
> > >>>>>>>>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>> take
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> > set
> > >>>>>> it
> > >>>>>>>>>>>>>>>>>> by_duration if
> > >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> > Tsai <
> > >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > >>>>>> something
> > >>>>>>>>>>>>>>>>>> important
> > >>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> > to
> > >>>>>> clarify
> > >>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>> few
> > >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> > this
> > >>>>>> logic
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> > earliest
> > >>>>>> or
> > >>>>>>>>>>>>>>>>> by_duration
> > >>>>>>>>>>>>>>>>>>>>>>> don't
> > >>>>>>>>>>>>>>>>>>>>>>>>> seem
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> > issue
> > >>>>>> when a
> > >>>>>>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > >>>>>> configure for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>> .partitions?
> > >>>>>>>>>>>>>> If they set it to
> > >>>>>>>>>>>>>>>>> earliest
> > >>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge
> > cases. For
> > >>>>>>>>>>>>>> example,
> > >>>>>>>>>>>>>>>>> if a
> > >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > >>>>>> created
> > >>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> > latest
> > >>>>>> when
> > >>>>>>>>>>>>>>>>> resuming,
> > >>>>>>>>>>>>>>>>>>>>>>> rather
> > >>>>>>>>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> > partition
> > >>>>>> is
> > >>>>>>>>>>>>>>>>>> technically
> > >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> > introducing a
> > >>>>>> max.age
> > >>>>>>>>>>>>>>>>>> threshold.
> > >>>>>>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>>>>>> gives
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > >>>>>> genuinely
> > >>>>>>>>>>>>>>> "hot/new"
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>> what is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev
> > wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> > bit
> > >>>>>> weird. It
> > >>>>>>>>>>>>>>>>> only
> > >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> > seems
> > >>>>>> that the
> > >>>>>>>>>>>>>>>>>>>>> motivation
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> > other
> > >>>>>> values
> > >>>>>>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a
> > separate
> > >>>>>> way to
> > >>>>>>>>>>>>>>> control
> > >>>>>>>>>>>>>>>>>>>>> newly
> > >>>>>>>>>>>>>>>>>>>>>>>>> created
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > >>>> group
> > >>>>>>>>>>>>>> starts.
> > >>>>>>>>>>>>>>>>>> Have we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> > auto.offset.reset.new
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>>>>>
> > >>>>>>>>>>>>>>> .partitions?
> > >>>>>>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > >>>>>> defaults to
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > >>>>>> explicitly
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> customize
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > >>>>>> s7133700@gmail.com
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> > 晚上10:37
> > >>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> > The
> > >>>>>>>>>>>>>>>>>> PartitionMetadata
> > >>>>>>>>>>>>>>>>>>>>>>>>> struct
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > >>>>>> computed
> > >>>>>>>>>>>>>>>>>> server-side
> > >>>>>>>>>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > >>>> partition_creation_time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow.
> > when
> > >>>>>>>>>>>>>>>>>> MembershipManager
> > >>>>>>>>>>>>>>>>>>>>>>>>> detects
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > >>>>>> metadata for
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> affected
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> > even
> > >>>>>> if the
> > >>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>> ID
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > >>>>>> supported
> > >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> > time.
> > >>>>>> If the
> > >>>>>>>>>>>>>>>>>> negotiated
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > >>>>>> PartitionAgeMs at
> > >>>>>>>>>>>>>> all
> > >>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > >>>>>> back to
> > >>>>>>>>>>>>>> latest
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>> risking
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> > signal.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the
> > KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > >>>>>> 2026年4月29日
> > >>>>>>>>>>>>>>> 下午4:04
> > >>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > >>>>>> 'age'
> > >>>>>>>>>>>>>>>>> resolution
> > >>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> > the
> > >>>>>> control
> > >>>>>>>>>>>>>>>>> plane
> > >>>>>>>>>>>>>>>>>>>>>>> clean.
> > >>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> > introducing
> > >>>>>>>>>>>>>>> inter-broker
> > >>>>>>>>>>>>>>>>>>>>> clock
> > >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> > single
> > >>>>>> source of
> > >>>>>>>>>>>>>>>>> truth
> > >>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>> time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > >>>>>> should be
> > >>>>>>>>>>>>>>>>>> negligible.
> > >>>>>>>>>>>>>>>>>>>>>>>>> Given
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > >>>>>> configured in
> > >>>>>>>>>>>>>>> minutes
> > >>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > >>>>>> brokers
> > >>>>>>>>>>>>>> won't
> > >>>>>>>>>>>>>>>>>> impact
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> > dev@kafka.apache.org>
> > >>>>>> 於
> > >>>>>>>>>>>>>>>>> 2026年4月29日
> > >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> > >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > >>>>>> previous
> > >>>>>>>>>>>>>>>>>> conversation
> > >>>>>>>>>>>>>>>>>>>>>>> but I
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> > flow
> > >>>>>> with the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> > new
> > >>>>>> boolean;
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>>>>>>>> states
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> > this
> > >>>>>> boolean
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> set.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives
> > a new
> > >>>>>>>>>>>>>> partition,
> > >>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> > receive
> > >>>>>> the
> > >>>>>>>>>>>>>> ages.
> > >>>>>>>>>>>>>>> Is
> > >>>>>>>>>>>>>>>>>> my
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> > flow
> > >>>>>> and
> > >>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>> explain
> > >>>>>>>>>>>>>>>>>>>>>>>>> how it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> > offsets,
> > >>>>>> fetch
> > >>>>>>>>>>>>>>>>> offsets,
> > >>>>>>>>>>>>>>>>>>>>>>> etc.).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > >>>>>> wonder if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> > right
> > >>>>>> place
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>> given
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> > Alternatively,
> > >>>>>> it could
> > >>>>>>>>>>>>>>>>> simply
> > >>>>>>>>>>>>>>>>>>>>>>>>> include
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be
> > rather
> > >>>>>> cautious
> > >>>>>>>>>>>>>>> about
> > >>>>>>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > >>>>>> unrelated
> > >>>>>>>>>>>>>>> concepts.
> > >>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>> API
> > >>>>>>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> > revoking
> > >>>>>>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>> fact
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> > corresponding
> > >>>>>> Streams
> > >>>>>>>>>>>>>> API
> > >>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> > would we
> > >>>>>> do if
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>> want to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > >>>>>> Muralidhar Basani
> > >>>>>>>>>>>>>>> via
> > >>>>>>>>>>>>>>>>>> dev
> > >>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> > know
> > >>>>>> about
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> gap.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > >>>>>>>>>>>>>>>>>> RequestPartitionAges
> > >>>>>>>>>>>>>>>>>>>>>>>>> field.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > >>>>>> PartitionAges
> > >>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > >>>>>> recommended
> > >>>>>>>>>>>>>>>>> value
> > >>>>>>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based
> > on the
> > >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > >>>>>>>>>>>>>> <
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > >>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > >>>>>> guess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no
> > changes to
> > >>>>>> Kafka
> > >>>>>>>>>>>>>>>>> Streams,
> > >>>>>>>>>>>>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> > it?
> > >>>>>> This is
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > >>>>>> mistakenly
> > >>>>>>>>>>>>>>>>>> configure
> > >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> > consumer
> > >>>>>> falls
> > >>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > >>>>>> per-partition
> > >>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>>> isn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> > "consumer
> > >>>>>> resolves
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> initial
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > >>>>>> earliest
> > >>>>>>>>>>>>>> was
> > >>>>>>>>>>>>>>>>>> applied
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > >>>>>> config is
> > >>>>>>>>>>>>>>>>>> unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include
> > this
> > >>>>>> change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> > 於
> > >>>>>>>>>>>>>> 2026年4月28日
> > >>>>>>>>>>>>>>>>>> 晚上8:03
> > >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> > partition
> > >>>>>> creation
> > >>>>>>>>>>>>>>> time
> > >>>>>>>>>>>>>>>>>> via
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> > users
> > >>>> to
> > >>>>>>>>>>>>>> diagnose
> > >>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > >>>>>> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> > on
> > >>>>>> KIP-1327
> > >>>>>>>>>>>>>>>>>> Prevent
> > >>>>>>>>>>>>>>>>>>>>> Hot
> > >>>>>>>>>>>>>>>>>>>>>>>>> Data
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>
> > >>>>
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> > newly
> > >>>>>> expanded
> > >>>>>>>>>>>>>>>>> (hot)
> > >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> > threshold
> > >>>>>>>>>>>>>>> automatically
> > >>>>>>>>>>>>>>>>>> fall
> > >>>>>>>>>>>>>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> > forcing
> > >>>>>> a full
> > >>>>>>>>>>>>>>>>>> historical
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Jiunn-Yang,

JR2. We should try to maintain consistency across our APIs. So, it would be
useful to understand if the user-facing part of this KIP is what share
consumer wants too. It would be problematic if we proceed with this
approach only for the share consumer to choose a different one later.

JR3. I was thinking that if we add the NewPartitions field in
GroupHeartbeatResponse, there is no need to expose the group/partition
creation timestamp to other RPCs and the Cli.

JR5. Since this changes the on-disk data format for PartitionRecord, we
need to gate this by a new MV. It also changes the format of
ConsumerGroupMetadataValue, so we need to gate it too. I'm not sure if this
requires a new MV or a new GV.

Thanks,

Jun

On Thu, Jun 25, 2026 at 5:52 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun, chia,
>
> Thanks for your feedback.
>
> JR3 JR4 I have updated the KIP.
>
> JR2 The new-partition detection mechanism is reusable. The coordinator
> already has
> access to both timestamps required for classification, so the
> determination can be made
> server-side without introducing any new RPCs. The main difference is where
> the policy is
> enforced: modern consumer resolves it on the client, whereas a share group
> resolves the
> starting offset on the partition leader. As a result, the classification
> decision would need to
> be propagated through the existing share-state initialization path. We
> would also need to persist
> the group's creation time as part of the share-group state, since that
> information is not stored today.
>
> None of this is a blocker, but it does require changes to the share-state
> format and introduces
> a share-specific configuration. Given that additional scope, I would
> prefer to keep share groups out
> of scope for this KIP and address them in a follow-up effort, while
> documenting the relevant reuse
> opportunities here.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> >
> >> It inspires me to think about mirror clusters. What if the mirror group
> is created 'after' the expanded partitions? In this scenario, the new
> configuration 'max.age.ms' could handle it if users are aware of the
> migration and set a suitable time interval. Of course, this would require
> both creation times to be exposed to clients.
> >
> > Thinking more about the mirror cluster scenario we discussed, I realize
> it introduces significant complexity. Handling it properly would require a
> tolerance time window rather than a strict time interval to account for
> replication lag. To prevent scope creep and keep KIP-1327 focused on its
> primary goal, I think it would be best to defer the multi-cluster/mirroring
> scenario and explore those approaches in a separate KIP
> >
> > On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> >> hi all,
> >>
> >>> JR3. The KIP exposes the group/partition creation time to the client.
> I am
> >>> not sure if there are other use cases for those timestamps. An
> alternative
> >>> is to avoid exposing those timestamps to the client. The group
> coordinator
> >>> will categorize the partitions using those timestamps and include the
> >>> categorization in the assigned partitions in GroupHeartbeatResponse.
> >>
> >> It inspires me to think about mirror clusters. What if the mirror group
> is created 'after' the expanded partitions? In this scenario, the new
> configuration 'max.age.ms' could handle it if users are aware of the
> migration and set a suitable time interval. Of course, this would require
> both creation times to be exposed to clients.
> >>
> >> Best,
> >> Chia-Ping
> >>
> >> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> >>> Hi, Jiunn-Yang,
> >>>
> >>> Thanks for the updated KIP.
> >>>
> >>> JR2. Share consumer (KIP-932) states: "If the number of partitions is
> >>> increased for a topic with a subscription in a share group, the SPSO
> for
> >>> the newly created share-partitions is initialized to 0 ". However, this
> >>> part has not been implemented yet. Currently, all partitions without
> offset
> >>> follows the strategy given by share.auto.offset.reset. So, we have an
> >>> opportunity to see if it's better to align this KIP's changes with the
> >>> share consumer.
> >>> JR2.1 The first part is whether customizing the offset for new
> partitions
> >>> to something other than 0 is useful in the share consumer. It seems
> that
> >>> the motivation in this KIP applies to share consumer too.
> >>> JR2.2 The second part is how new partitions are detected. The mechanism
> >>> introduced in this KIP seems quite effective. A group coordinator could
> >>> pick up newly created partitions for an existing topic or newly created
> >>> topics matching the regex. Those partitions are categorized as new
> >>> partitions and will follow the strategy given by
> >>> auto.offset.reset.new.partitions.
> >>> This makes sense since those partitions typically don't have a large
> >>> backlog. A consumer could also additionally subscribe to an existing
> topic
> >>> in an existing group. Those partitions will be categorized as existing
> >>> partitions. This also makes sense since those partitions could have a
> large
> >>> backlog and it is better to follow the strategy given by
> auto.offset.reset. It
> >>> will be useful to think through whether the logic for categorizing new
> >>> partitions should also be reused in the share consumer.
> >>>
> >>> JR3. The KIP exposes the group/partition creation time to the client.
> I am
> >>> not sure if there are other use cases for those timestamps. An
> alternative
> >>> is to avoid exposing those timestamps to the client. The group
> coordinator
> >>> will categorize the partitions using those timestamps and include the
> >>> categorization in the assigned partitions in GroupHeartbeatResponse.
> >>>
> >>> JR4. It would be useful to support this feature on existing groups as
> well.
> >>> For example, any partition without a creation timestamp can always be
> >>> categorized as an existing partition.
> >>>
> >>> Jun
> >>>
> >>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>
> >>>> Hello Andrew,
> >>>>
> >>>> Thanks for your feedback,
> >>>>>> I've just been re-reading this KIP to ensure that share group
> support is
> >>>>>> not needed. I've discovered that we made a mistake implementing
> KIP-932
> >>>> in
> >>>>>> this area, so we'll rectify that and check whether it's really tight
> >>>> enough
> >>>>>> not to require KIP-1327.
> >>>>
> >>>> Could you walk us through the specific area where the mistake was
> found?
> >>>> That would
> >>>> help us assess whether it naturally fits within the scope of KIP-1327
> or
> >>>> should be addressed
> >>>> separately.
> >>>>
> >>>> AS1 I have updated the KIP
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> >>>>>
> >>>>> hi Andrew
> >>>>>
> >>>>>> I've discovered that we made a mistake implementing KIP-932 in this
> >>>> area,
> >>>>> so we'll rectify that and check whether it's really tight enough not
> to
> >>>>> require KIP-1327.
> >>>>>
> >>>>> Would you mind sharing more details about that? If it's not anything
> >>>> close
> >>>>> to NP-hard, we could definitely look into covering it within KIP-1327
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>>
> >>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
> >>>>>
> >>>>>> Hi Jiunn-Yang,
> >>>>>> I've just been re-reading this KIP to ensure that share group
> support is
> >>>>>> not needed. I've discovered that we made a mistake implementing
> KIP-932
> >>>> in
> >>>>>> this area, so we'll rectify that and check whether it's really tight
> >>>> enough
> >>>>>> not to require KIP-1327.
> >>>>>>
> >>>>>> One trivial comments for consistency.
> >>>>>>
> >>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is
> represented by
> >>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Andrew
> >>>>>>
> >>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> Manually bumping this thread.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> >>>>>>>>
> >>>>>>>> Hello chia,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback, I have updated the KIP.
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>>
> >>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> >>>>>>>>>
> >>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>> When the config is set on a cluster that has not yet been
> >>>>>> upgraded... classification cannot occur... the consumer falls back
> to
> >>>> the
> >>>>>> base auto.offset.reset for the affected partitions. No exception is
> >>>> thrown,
> >>>>>> and no operational disruption results.
> >>>>>>>>>
> >>>>>>>>> Existing group can't take advantage of this excellent new
> >>>>>> configuration. Allowing users to modify the group creation time
> might be
> >>>>>> overkill. Instead, we could print a useful warning message to guide
> >>>> users.
> >>>>>> For example, we can suggest that they re-create the group with their
> >>>>>> existing committed offsets
> >>>>>>>>>
> >>>>>>>>>> Protocol changes
> >>>>>>>>>
> >>>>>>>>> Would you mind listing those RPC changes in a table format?
> >>>>>>>>>
> >>>>>>>>>> The full interaction matrix between the base policy and the
> >>>>>> new-partition policy is:
> >>>>>>>>>
> >>>>>>>>> Please add a filed to describe the target scenario when using
> these
> >>>>>> policies
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Chia-Ping
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> >>>>>>>>>> Hello Jun, chia,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the feedback, I have updated the KIP for the new
> >>>>>>>>>> approach, PTAL
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>
> >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> >>>>>>>>>>>
> >>>>>>>>>>> hi Jun
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, your approach is great. I think the combination of latest
> (for
> >>>>>> existing partitions) and by_duration (for new partitions) can
> address
> >>>> 99%
> >>>>>> of the complaints I have heard regarding this issue.
> >>>>>>>>>>>
> >>>>>>>>>>> Also, leveraging the group creation time here opens the door to
> >>>>>> implementing a new policy based on timestamp seek in the future,
> should
> >>>> the
> >>>>>> community want to pursue that.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for your patience and constructive feedback. We will
> update
> >>>>>> the KIP accordingly.
> >>>>>>>>>>>
> >>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>
> >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11
> 寫道:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I agree that it's probably useful to allow a user to
> configure a
> >>>>>> different
> >>>>>>>>>>>> offset policy for existing partitions vs new partitions.
> However,
> >>>>>> using
> >>>>>>>>>>>> group creation time to capture that seems more intuitive.
> Here is
> >>>>>> another
> >>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize
> new
> >>>>>> partitions
> >>>>>>>>>>>> based on group creation time. Introduce
> >>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose values
> can be
> >>>>>> earliest,
> >>>>>>>>>>>> latest and by_duration, the same as auto.offset.reset. Users
> can
> >>>>>> set
> >>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they
> want to
> >>>>>> guarantee
> >>>>>>>>>>>> no data loss on new partitions. They can also use by_duration
> to
> >>>>>> set an
> >>>>>>>>>>>> upper bound on the backlog replayed, which can be different
> from
> >>>>>> that of
> >>>>>>>>>>>> the existing partitions. This will address your concern about
> too
> >>>>>> much
> >>>>>>>>>>>> backlog being replayed when the offsets are lost. What do you
> >>>>>> think?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> >>>>>> chia7712@gmail.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The most important part of this story is how users should
> expect
> >>>>>> the data
> >>>>>>>>>>>>> they can see when using the latest or by_duration policy with
> >>>>>> expanded
> >>>>>>>>>>>>> partitions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> >>>>>>>>>>>>> non-deterministic, which means users will either read too
> many
> >>>>>> historical
> >>>>>>>>>>>>> records from existing partitions or lose some records from
> >>>>>> expanded
> >>>>>>>>>>>>> partitions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> >>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >>>>>>>
> >>>>>>>>>>>>> is a bit hard to understand, and that is why I preferred
> having a
> >>>>>> whole new
> >>>>>>>>>>>>> policy based entirely on group creation time (KIP-1282)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二
> 上午1:08寫道:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand the
> value
> >>>>>> of the new
> >>>>>>>>>>>>>> configs with the KIP.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want to
> miss
> >>>>>> the data if
> >>>>>>>>>>>>>> the backlog is small. The backlog of the existing partition
> is
> >>>>>> easy to
> >>>>>>>>>>>>>> understand because it relates to retention time. The
> backlog for
> >>>>>> the new
> >>>>>>>>>>>>>> partition is a bit subtle to understand since it depends on
> the
> >>>>>> metadata
> >>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >>>>>>> ,
> >>>>>>>>>>>>>> the user needs to
> >>>>>>>>>>>>>> understand the metadata refresh delay on the consumer side
> and
> >>>>>> use it to
> >>>>>>>>>>>>>> set the config.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Now, let's consider the alternative: setting the same value
> for
> >>>>>> the
> >>>>>>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> >>>> this
> >>>>>>>>>>>>>> approach.
> >>>>>>>>>>>>>> 1. It computes the seek target client-side as now() -
> duration,
> >>>>>> which
> >>>>>>>>>>>>>> introduces clock skew across consumers and forces operators
> to
> >>>>>> choose
> >>>>>>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> >>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so
> failed
> >>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> >>>>>> potentially
> >>>>>>>>>>>>>> miss records produced between attempts.
> >>>>>>>>>>>>>> 3. It applies uniformly to all partitions without committed
> >>>>>> offsets, and
> >>>>>>>>>>>>>> cannot distinguish newly expanded partitions from
> long-existing
> >>>>>> partitions
> >>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding
> a bit
> >>>>>> buffer to
> >>>>>>>>>>>>>> the metadata refresh delay. We could also consider
> improving the
> >>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> >>>>>> typically low
> >>>>>>>>>>>>>> (in the order of minutes with the classic consumer and tens
> of
> >>>>>> seconds
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>> the new consumer). If a user is ok with reading that much
> >>>>>> backlog for new
> >>>>>>>>>>>>>> partitions, it seems they will be ok doing the same for
> existing
> >>>>>>>>>>>>>> partitions.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So, instead of introducing a new config, could we just
> reuse the
> >>>>>> existing
> >>>>>>>>>>>>>> config with better documentation and/or implementation?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> You're right that group creation time is the more intuitive
> >>>>>> answer at
> >>>>>>>>>>>>>>> first glance,
> >>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that
> "predate
> >>>>>> the group"
> >>>>>>>>>>>>>>> vs partitions
> >>>>>>>>>>>>>>> "created during group runtime," which directly points to a
> >>>>>>>>>>>>>> group-lifecycle
> >>>>>>>>>>>>>>> classifier.
> >>>>>>>>>>>>>>> I'd like to walk through why we landed on partition age,
> and
> >>>> the
> >>>>>>>>>>>>>>> trade-offs we considered.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We evaluated three candidate signals:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. `by_duration:5secs`
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This covers the metadata blindness window, but has issues
> the
> >>>>>> KIP
> >>>>>>>>>>>>>>> currently documents
> >>>>>>>>>>>>>>> under "Why not use `by_duration`?":
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew
> across
> >>>>>> consumers.
> >>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward,
> potentially
> >>>>>> missing
> >>>>>>>>>>>>>>> records produced between
> >>>>>>>>>>>>>>> attempts.
> >>>>>>>>>>>>>>> - It applies uniformly to all partitions without committed
> >>>>>> offsets,
> >>>>>>>>>>>>>>> including pre-existing partitions
> >>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2. Group creation time as classifier
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This works cleanly when the consumer is actively running.
> Our
> >>>>>> concern
> >>>>>>>>>>>>>>> is the idle / late-rejoin case:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> T=0:         Group created.
> >>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> >>>>>>>>>>>>>>> T=50:        Partition added during the idle window.
> >>>>>>>>>>>>>>> T=100:       Consumer resumes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Under group creation time, the new partition is classified
> as
> >>>>>> new
> >>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything
> from
> >>>>>> T=50.
> >>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> >>>>>> data that
> >>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the
> contract
> >>>> of
> >>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason
> to
> >>>>>> treat
> >>>>>>>>>>>>>>> the new partition differently; both contain backlog
> accumulated
> >>>>>> during
> >>>>>>>>>>>>>>> the same idle window.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle you
> raised
> >>>>>> in
> >>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some
> backlog
> >>>>>> on
> >>>>>>>>>>>>>>> every other partition during the same idle period; forcing
> >>>>>> 0-backlog
> >>>>>>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> >>>> that
> >>>>>>>>>>>>>>> tolerance.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 3. Partition age vs threshold
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Partition age corresponds to the actual silent data loss
> >>>> window,
> >>>>>>>>>>>>>>> the gap between partition creation and the consumer’s
> metadata
> >>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely
> silent: the
> >>>>>>>>>>>>>>> consumer had no opportunity to know about the partition.
> >>>>>> Outside this
> >>>>>>>>>>>>>>> window, missing data reflects either:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle
> consumers,
> >>>>>> or
> >>>>>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not
> via
> >>>>>> reset
> >>>>>>>>>>>>>> policy.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> We did not choose partition age because it is more elegant
> than
> >>>>>> group
> >>>>>>>>>>>>>>> creation time — we chose it because its failure mode
> (requires
> >>>> a
> >>>>>>>>>>>>>>> threshold) is
> >>>>>>>>>>>>>>> less invasive than the failure mode of group creation time
> >>>>>> (overrides
> >>>>>>>>>>>>>>> user-stated
> >>>>>>>>>>>>>>> `latest` intent during idle periods).
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52
> 寫道:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Relying on both creation times will create an inconsistent
> >>>>>> scenario. A
> >>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will
> seek
> >>>>>> to the
> >>>>>>>>>>>>>>>> beginning for the partitions created later than the group.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> >>>>>> inconsistency
> >>>>>>>>>>>>>>> using a
> >>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a
> consensus,
> >>>>>> KIP-1327
> >>>>>>>>>>>>>>> goes
> >>>>>>>>>>>>>>>> back to using flexible configurations to prevent users
> from
> >>>>>> falling
> >>>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>> that pitfall.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> >>>>>> 上午6:49寫道:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >>>>>>>
> >>>>>>>>>>>>>> still feels weird to
> >>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on the
> >>>>>> partition
> >>>>>>>>>>>>>>> creation
> >>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be based on
> the
> >>>>>> group
> >>>>>>>>>>>>>>> creation
> >>>>>>>>>>>>>>>>> time: all partitions existing when the group is created
> are
> >>>>>> existing
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> all partitions created after the group creation are new
> >>>>>> partitions.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> >>>>>>>>>>>>>>>>>> discussion, I will close the vote.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hello Jian,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for your feedback,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational
> task,
> >>>>>> not an
> >>>>>>>>>>>>>> edge
> >>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to
> clarify
> >>>>>> that
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which
> further
> >>>>>>>>>>>>>> demonstrates
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
> >>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> >>>>>> Kafka: To
> >>>>>>>>>>>>>>>>> balance
> >>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> >>>>>> created with a
> >>>>>>>>>>>>>>>>>> moderate
> >>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic grows
> over
> >>>>>> time, it
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> often
> >>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> >>>>>> accommodate the
> >>>>>>>>>>>>>>>>> higher
> >>>>>>>>>>>>>>>>>>>> workload.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Regards
> >>>>>>>>>>>>>>>>>>>> Jian
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日
> 晚上8:29
> >>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold"
> when
> >>>>>>>>>>>>>> describing
> >>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using
> them
> >>>> to
> >>>>>>>>>>>>>> describe
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> >>>>>> applying
> >>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>>> terms
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> >>>>>> whether a
> >>>>>>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is
> that
> >>>>>> they won't
> >>>>>>>>>>>>>>>>>> silently
> >>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during
> their
> >>>>>> active
> >>>>>>>>>>>>>>>>>>>>> consumption.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> >>>>>> motivation
> >>>>>>>>>>>>>>> section.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於
> 2026年5月30日
> >>>>>> 凌晨1:12
> >>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> >>>>>> motivation
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> >>>>>> partitions
> >>>>>>>>>>>>>> without
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> >>>>>> partition
> >>>>>>>>>>>>>> (hot)
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the
> consumer
> >>>>>> has
> >>>>>>>>>>>>>> never
> >>>>>>>>>>>>>>>>> seen
> >>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the
> hot
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is
> also
> >>>>>>>>>>>>>>> insufficient
> >>>>>>>>>>>>>>>>>>>>>>> because:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration)
> varies
> >>>>>> across
> >>>>>>>>>>>>>> nodes
> >>>>>>>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly
> large
> >>>>>>>>>>>>>> duration,
> >>>>>>>>>>>>>>>>>>>>>>> causing
> >>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> >>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the
> seek
> >>>>>> time on
> >>>>>>>>>>>>>>>>> retry,
> >>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking
> data
> >>>>>> loss."
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If
> these
> >>>>>> issues
> >>>>>>>>>>>>>>>>> persist,
> >>>>>>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> >>>>>> situations don't
> >>>>>>>>>>>>>>>>> need a
> >>>>>>>>>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare
> situations,
> >>>>>> they can
> >>>>>>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>>>>>> customized logic using
> >>>>>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> >>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> >>>>>> should
> >>>>>>>>>>>>>> always
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> >>>>>> any
> >>>>>>>>>>>>>>>>>> field
> >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the
> version
> >>>>>>>>>>>>>> mismatch
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which
> occurs
> >>>>>> inside
> >>>>>>>>>>>>>>>>> poll().
> >>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
> >>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an
> UnsupportedVersionException
> >>>>>> from
> >>>>>>>>>>>>>> poll().
> >>>>>>>>>>>>>>>>>> I’ll
> >>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> 2026年5月17日
> >>>>>> 下午4:50
> >>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of
> this
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker
> as
> >>>>>>>>>>>>>>>>>>>>> broker_current_time
> >>>>>>>>>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> >>>>>> does not
> >>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is
> unknown.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> >>>>>> should
> >>>>>>>>>>>>>> always
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> >>>>>> any
> >>>>>>>>>>>>>>>>>> field
> >>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException
> when
> >>>>>> calling
> >>>>>>>>>>>>>>>>>>>>> `poll()`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change
> based on
> >>>>>> the
> >>>>>>>>>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> >>>> reset
> >>>>>>>>>>>>>> behavior
> >>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >>>>>>>
> >>>>>>>>>>>>>> now applies to all
> >>>>>>>>>>>>>>>>>> auto.offset.reset
> >>>>>>>>>>>>>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> >>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer
> resets
> >>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>>
> >>>>>>>>>>>>>> .partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
> >>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> >>>>>>>>>>>>>> auto.offset.reset
> >>>>>>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>>>>>> continues
> >>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is
> represented
> >>>>>> by a
> >>>>>>>>>>>>>>> separate
> >>>>>>>>>>>>>>>>>>>>>>>>> internal config
> >>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>> .partitions,
> >>>>>>>>>>>>>> currently fixed to
> >>>>>>>>>>>>>>>>> earliest).
> >>>>>>>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
> >>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> >>>>>> public
> >>>>>>>>>>>>>>>>>> user-facing
> >>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> 2026年5月16日
> >>>>>> 清晨7:46
> >>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me
> is
> >>>>>> listed
> >>>>>>>>>>>>>>> below:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>> .partitions
> >>>>>>>>>>>>>> with a default value
> >>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> >>>>>> by_duration and
> >>>>>>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
> >>>>>> auto.offset.reset=earliest.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>> .partitions
> >>>>>>>>>>>>>> as an internal
> >>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>>
> >>>>>>>>>>>>>> .partitions=earliest
> >>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use
> cases
> >>>>>> of
> >>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>>>>>>> in a
> >>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>> .partitions
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> >>>>>>> ,
> >>>>>>>>>>>>>> will be applied to all for
> >>>>>>>>>>>>>>>>>>>>>>>>> consistency.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says
> "When
> >>>>>> a Kafka
> >>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>> expanded
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the
> latest
> >>>>>> auto
> >>>>>>>>>>>>>> offset
> >>>>>>>>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to
> those
> >>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> >>>>>> record loss
> >>>>>>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>> .partitions
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>> take
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could
> set
> >>>>>> it
> >>>>>>>>>>>>>>>>>> by_duration if
> >>>>>>>>>>>>>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping
> Tsai <
> >>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> >>>>>> something
> >>>>>>>>>>>>>>>>>> important
> >>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try
> to
> >>>>>> clarify
> >>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>> few
> >>>>>>>>>>>>>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending
> this
> >>>>>> logic
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like
> earliest
> >>>>>> or
> >>>>>>>>>>>>>>>>> by_duration
> >>>>>>>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss
> issue
> >>>>>> when a
> >>>>>>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>> expanded.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> >>>>>> configure for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>> .partitions?
> >>>>>>>>>>>>>> If they set it to
> >>>>>>>>>>>>>>>>> earliest
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge
> cases. For
> >>>>>>>>>>>>>> example,
> >>>>>>>>>>>>>>>>> if a
> >>>>>>>>>>>>>>>>>>>>>>>>> consumer is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> >>>>>> created
> >>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> downtime,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to
> latest
> >>>>>> when
> >>>>>>>>>>>>>>>>> resuming,
> >>>>>>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the
> partition
> >>>>>> is
> >>>>>>>>>>>>>>>>>> technically
> >>>>>>>>>>>>>>>>>>>>>>>>> "new" to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for
> introducing a
> >>>>>> max.age
> >>>>>>>>>>>>>>>>>> threshold.
> >>>>>>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>>>>>> gives
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> >>>>>> genuinely
> >>>>>>>>>>>>>>> "hot/new"
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> what is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev
> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a
> bit
> >>>>>> weird. It
> >>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>>>>> applies when
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it
> seems
> >>>>>> that the
> >>>>>>>>>>>>>>>>>>>>> motivation
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to
> other
> >>>>>> values
> >>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a
> separate
> >>>>>> way to
> >>>>>>>>>>>>>>> control
> >>>>>>>>>>>>>>>>>>>>> newly
> >>>>>>>>>>>>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> >>>> group
> >>>>>>>>>>>>>> starts.
> >>>>>>>>>>>>>>>>>> Have we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like
> auto.offset.reset.new
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>>>>>
> >>>>>>>>>>>>>>> .partitions?
> >>>>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> >>>>>> defaults to
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> >>>>>> explicitly
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> customize
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> >>>>>> s7133700@gmail.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日
> 晚上10:37
> >>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14.
> The
> >>>>>>>>>>>>>>>>>> PartitionMetadata
> >>>>>>>>>>>>>>>>>>>>>>>>> struct
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> >>>>>> computed
> >>>>>>>>>>>>>>>>>> server-side
> >>>>>>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> >>>> partition_creation_time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow.
> when
> >>>>>>>>>>>>>>>>>> MembershipManager
> >>>>>>>>>>>>>>>>>>>>>>>>> detects
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> >>>>>> metadata for
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision,
> even
> >>>>>> if the
> >>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>> ID
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> >>>>>> supported
> >>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection
> time.
> >>>>>> If the
> >>>>>>>>>>>>>>>>>> negotiated
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> >>>>>> PartitionAgeMs at
> >>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> >>>>>> back to
> >>>>>>>>>>>>>> latest
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> risking
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible
> signal.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the
> KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> >>>>>> 2026年4月29日
> >>>>>>>>>>>>>>> 下午4:04
> >>>>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> >>>>>> 'age'
> >>>>>>>>>>>>>>>>> resolution
> >>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep
> the
> >>>>>> control
> >>>>>>>>>>>>>>>>> plane
> >>>>>>>>>>>>>>>>>>>>>>> clean.
> >>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is
> introducing
> >>>>>>>>>>>>>>> inter-broker
> >>>>>>>>>>>>>>>>>>>>> clock
> >>>>>>>>>>>>>>>>>>>>>>>>> skew.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a
> single
> >>>>>> source of
> >>>>>>>>>>>>>>>>> truth
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> >>>>>> should be
> >>>>>>>>>>>>>>>>>> negligible.
> >>>>>>>>>>>>>>>>>>>>>>>>> Given
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> >>>>>> configured in
> >>>>>>>>>>>>>>> minutes
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>> hours, a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> >>>>>> brokers
> >>>>>>>>>>>>>> won't
> >>>>>>>>>>>>>>>>>> impact
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <
> dev@kafka.apache.org>
> >>>>>> 於
> >>>>>>>>>>>>>>>>> 2026年4月29日
> >>>>>>>>>>>>>>>>>>>>>>> 下午3:29
> >>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> >>>>>> previous
> >>>>>>>>>>>>>>>>>> conversation
> >>>>>>>>>>>>>>>>>>>>>>> but I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the
> flow
> >>>>>> with the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a
> new
> >>>>>> boolean;
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>>>>>> states
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when
> this
> >>>>>> boolean
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> set.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives
> a new
> >>>>>>>>>>>>>> partition,
> >>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to
> receive
> >>>>>> the
> >>>>>>>>>>>>>> ages.
> >>>>>>>>>>>>>>> Is
> >>>>>>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the
> flow
> >>>>>> and
> >>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>> explain
> >>>>>>>>>>>>>>>>>>>>>>>>> how it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list
> offsets,
> >>>>>> fetch
> >>>>>>>>>>>>>>>>> offsets,
> >>>>>>>>>>>>>>>>>>>>>>> etc.).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> >>>>>> wonder if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the
> right
> >>>>>> place
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>> given
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway.
> Alternatively,
> >>>>>> it could
> >>>>>>>>>>>>>>>>> simply
> >>>>>>>>>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be
> rather
> >>>>>> cautious
> >>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> >>>>>> unrelated
> >>>>>>>>>>>>>>> concepts.
> >>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>> API
> >>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or
> revoking
> >>>>>>>>>>>>>>> partitions.
> >>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>> fact
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the
> corresponding
> >>>>>> Streams
> >>>>>>>>>>>>>> API
> >>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>>>>> suggests
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What
> would we
> >>>>>> do if
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> want to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> >>>>>> Muralidhar Basani
> >>>>>>>>>>>>>>> via
> >>>>>>>>>>>>>>>>>> dev
> >>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to
> know
> >>>>>> about
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> gap.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> >>>>>>>>>>>>>>>>>> RequestPartitionAges
> >>>>>>>>>>>>>>>>>>>>>>>>> field.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> >>>>>> PartitionAges
> >>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> >>>>>> recommended
> >>>>>>>>>>>>>>>>> value
> >>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based
> on the
> >>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> >>>>>>>>>>>>>> <
> >>>>>>
> >>>>
> https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> >>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> >>>>>> guess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no
> changes to
> >>>>>> Kafka
> >>>>>>>>>>>>>>>>> Streams,
> >>>>>>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> >>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure
> it?
> >>>>>> This is
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> >>>>>> mistakenly
> >>>>>>>>>>>>>>>>>> configure
> >>>>>>>>>>>>>>>>>>>>>>>>> it. Or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the
> consumer
> >>>>>> falls
> >>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> earliest"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> >>>>>> per-partition
> >>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>> isn't
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like
> "consumer
> >>>>>> resolves
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> initial
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> >>>>>> earliest
> >>>>>>>>>>>>>> was
> >>>>>>>>>>>>>>>>>> applied
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> >>>>>> config is
> >>>>>>>>>>>>>>>>>> unchanged.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> >>>>>>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include
> this
> >>>>>> change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org>
> 於
> >>>>>>>>>>>>>> 2026年4月28日
> >>>>>>>>>>>>>>>>>> 晚上8:03
> >>>>>>>>>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the
> partition
> >>>>>> creation
> >>>>>>>>>>>>>>> time
> >>>>>>>>>>>>>>>>>> via
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for
> users
> >>>> to
> >>>>>>>>>>>>>> diagnose
> >>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> >>>>>> auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion
> on
> >>>>>> KIP-1327
> >>>>>>>>>>>>>>>>>> Prevent
> >>>>>>>>>>>>>>>>>>>>> Hot
> >>>>>>>>>>>>>>>>>>>>>>>>> Data
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>
> >>>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish
> newly
> >>>>>> expanded
> >>>>>>>>>>>>>>>>> (hot)
> >>>>>>>>>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured
> threshold
> >>>>>>>>>>>>>>> automatically
> >>>>>>>>>>>>>>>>>> fall
> >>>>>>>>>>>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without
> forcing
> >>>>>> a full
> >>>>>>>>>>>>>>>>>> historical
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello Jun, chia,

Thanks for your feedback.

JR3 JR4 I have updated the KIP.

JR2 The new-partition detection mechanism is reusable. The coordinator already has 
access to both timestamps required for classification, so the determination can be made 
server-side without introducing any new RPCs. The main difference is where the policy is 
enforced: modern consumer resolves it on the client, whereas a share group resolves the 
starting offset on the partition leader. As a result, the classification decision would need to 
be propagated through the existing share-state initialization path. We would also need to persist 
the group's creation time as part of the share-group state, since that information is not stored today. 

None of this is a blocker, but it does require changes to the share-state format and introduces 
a share-specific configuration. Given that additional scope, I would prefer to keep share groups out 
of scope for this KIP and address them in a follow-up effort, while documenting the relevant reuse 
opportunities here.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月25日 下午1:31 寫道:
> 
>> It inspires me to think about mirror clusters. What if the mirror group is created 'after' the expanded partitions? In this scenario, the new configuration 'max.age.ms' could handle it if users are aware of the migration and set a suitable time interval. Of course, this would require both creation times to be exposed to clients.
> 
> Thinking more about the mirror cluster scenario we discussed, I realize it introduces significant complexity. Handling it properly would require a tolerance time window rather than a strict time interval to account for replication lag. To prevent scope creep and keep KIP-1327 focused on its primary goal, I think it would be best to defer the multi-cluster/mirroring scenario and explore those approaches in a separate KIP
> 
> On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
>> hi all,
>> 
>>> JR3. The KIP exposes the group/partition creation time to the client. I am
>>> not sure if there are other use cases for those timestamps. An alternative
>>> is to avoid exposing those timestamps to the client. The group coordinator
>>> will categorize the partitions using those timestamps and include the
>>> categorization in the assigned partitions in GroupHeartbeatResponse.
>> 
>> It inspires me to think about mirror clusters. What if the mirror group is created 'after' the expanded partitions? In this scenario, the new configuration 'max.age.ms' could handle it if users are aware of the migration and set a suitable time interval. Of course, this would require both creation times to be exposed to clients.
>> 
>> Best,
>> Chia-Ping
>> 
>> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
>>> Hi, Jiunn-Yang,
>>> 
>>> Thanks for the updated KIP.
>>> 
>>> JR2. Share consumer (KIP-932) states: "If the number of partitions is
>>> increased for a topic with a subscription in a share group, the SPSO for
>>> the newly created share-partitions is initialized to 0 ". However, this
>>> part has not been implemented yet. Currently, all partitions without offset
>>> follows the strategy given by share.auto.offset.reset. So, we have an
>>> opportunity to see if it's better to align this KIP's changes with the
>>> share consumer.
>>> JR2.1 The first part is whether customizing the offset for new partitions
>>> to something other than 0 is useful in the share consumer. It seems that
>>> the motivation in this KIP applies to share consumer too.
>>> JR2.2 The second part is how new partitions are detected. The mechanism
>>> introduced in this KIP seems quite effective. A group coordinator could
>>> pick up newly created partitions for an existing topic or newly created
>>> topics matching the regex. Those partitions are categorized as new
>>> partitions and will follow the strategy given by
>>> auto.offset.reset.new.partitions.
>>> This makes sense since those partitions typically don't have a large
>>> backlog. A consumer could also additionally subscribe to an existing topic
>>> in an existing group. Those partitions will be categorized as existing
>>> partitions. This also makes sense since those partitions could have a large
>>> backlog and it is better to follow the strategy given by auto.offset.reset. It
>>> will be useful to think through whether the logic for categorizing new
>>> partitions should also be reused in the share consumer.
>>> 
>>> JR3. The KIP exposes the group/partition creation time to the client. I am
>>> not sure if there are other use cases for those timestamps. An alternative
>>> is to avoid exposing those timestamps to the client. The group coordinator
>>> will categorize the partitions using those timestamps and include the
>>> categorization in the assigned partitions in GroupHeartbeatResponse.
>>> 
>>> JR4. It would be useful to support this feature on existing groups as well.
>>> For example, any partition without a creation timestamp can always be
>>> categorized as an existing partition.
>>> 
>>> Jun
>>> 
>>> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hello Andrew,
>>>> 
>>>> Thanks for your feedback,
>>>>>> I've just been re-reading this KIP to ensure that share group support is
>>>>>> not needed. I've discovered that we made a mistake implementing KIP-932
>>>> in
>>>>>> this area, so we'll rectify that and check whether it's really tight
>>>> enough
>>>>>> not to require KIP-1327.
>>>> 
>>>> Could you walk us through the specific area where the mistake was found?
>>>> That would
>>>> help us assess whether it naturally fits within the scope of KIP-1327 or
>>>> should be addressed
>>>> separately.
>>>> 
>>>> AS1 I have updated the KIP
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
>>>>> 
>>>>> hi Andrew
>>>>> 
>>>>>> I've discovered that we made a mistake implementing KIP-932 in this
>>>> area,
>>>>> so we'll rectify that and check whether it's really tight enough not to
>>>>> require KIP-1327.
>>>>> 
>>>>> Would you mind sharing more details about that? If it's not anything
>>>> close
>>>>> to NP-hard, we could definitely look into covering it within KIP-1327
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> 
>>>>> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
>>>>> 
>>>>>> Hi Jiunn-Yang,
>>>>>> I've just been re-reading this KIP to ensure that share group support is
>>>>>> not needed. I've discovered that we made a mistake implementing KIP-932
>>>> in
>>>>>> this area, so we'll rectify that and check whether it's really tight
>>>> enough
>>>>>> not to require KIP-1327.
>>>>>> 
>>>>>> One trivial comments for consistency.
>>>>>> 
>>>>>> AS1: In kafka-consumer-groups.sh, unknown column data is represented by
>>>>>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
>>>>>> 
>>>>>> Thanks,
>>>>>> Andrew
>>>>>> 
>>>>>> On 2026/06/23 23:23:38 黃竣陽 wrote:
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> Manually bumping this thread.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
>>>>>>>> 
>>>>>>>> Hello chia,
>>>>>>>> 
>>>>>>>> Thanks for the feedback, I have updated the KIP.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
>>>>>>>>> 
>>>>>>>>> hi Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> When the config is set on a cluster that has not yet been
>>>>>> upgraded... classification cannot occur... the consumer falls back to
>>>> the
>>>>>> base auto.offset.reset for the affected partitions. No exception is
>>>> thrown,
>>>>>> and no operational disruption results.
>>>>>>>>> 
>>>>>>>>> Existing group can't take advantage of this excellent new
>>>>>> configuration. Allowing users to modify the group creation time might be
>>>>>> overkill. Instead, we could print a useful warning message to guide
>>>> users.
>>>>>> For example, we can suggest that they re-create the group with their
>>>>>> existing committed offsets
>>>>>>>>> 
>>>>>>>>>> Protocol changes
>>>>>>>>> 
>>>>>>>>> Would you mind listing those RPC changes in a table format?
>>>>>>>>> 
>>>>>>>>>> The full interaction matrix between the base policy and the
>>>>>> new-partition policy is:
>>>>>>>>> 
>>>>>>>>> Please add a filed to describe the target scenario when using these
>>>>>> policies
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the feedback, I have updated the KIP for the new
>>>>>>>>>> approach, PTAL
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
>>>>>>>>>>> 
>>>>>>>>>>> hi Jun
>>>>>>>>>>> 
>>>>>>>>>>> Yes, your approach is great. I think the combination of latest (for
>>>>>> existing partitions) and by_duration (for new partitions) can address
>>>> 99%
>>>>>> of the complaints I have heard regarding this issue.
>>>>>>>>>>> 
>>>>>>>>>>> Also, leveraging the group creation time here opens the door to
>>>>>> implementing a new policy based on timestamp seek in the future, should
>>>> the
>>>>>> community want to pursue that.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for your patience and constructive feedback. We will update
>>>>>> the KIP accordingly.
>>>>>>>>>>> 
>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>> 
>>>>>>>>>>>> I agree that it's probably useful to allow a user to configure a
>>>>>> different
>>>>>>>>>>>> offset policy for existing partitions vs new partitions. However,
>>>>>> using
>>>>>>>>>>>> group creation time to capture that seems more intuitive. Here is
>>>>>> another
>>>>>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new
>>>>>> partitions
>>>>>>>>>>>> based on group creation time. Introduce
>>>>>>>>>>>> a new config auto.offset.reset.new.partitions whose values can be
>>>>>> earliest,
>>>>>>>>>>>> latest and by_duration, the same as auto.offset.reset. Users can
>>>>>> set
>>>>>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to
>>>>>> guarantee
>>>>>>>>>>>> no data loss on new partitions. They can also use by_duration to
>>>>>> set an
>>>>>>>>>>>> upper bound on the backlog replayed, which can be different from
>>>>>> that of
>>>>>>>>>>>> the existing partitions. This will address your concern about too
>>>>>> much
>>>>>>>>>>>> backlog being replayed when the offsets are lost. What do you
>>>>>> think?
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
>>>>>> chia7712@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The most important part of this story is how users should expect
>>>>>> the data
>>>>>>>>>>>>> they can see when using the latest or by_duration policy with
>>>>>> expanded
>>>>>>>>>>>>> partitions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
>>>>>>>>>>>>> non-deterministic, which means users will either read too many
>>>>>> historical
>>>>>>>>>>>>> records from existing partitions or lose some records from
>>>>>> expanded
>>>>>>>>>>>>> partitions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
>>>>>>> 
>>>>>>>>>>>>> is a bit hard to understand, and that is why I preferred having a
>>>>>> whole new
>>>>>>>>>>>>> policy based entirely on group creation time (KIP-1282)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the reply. I am still trying to understand the value
>>>>>> of the new
>>>>>>>>>>>>>> configs with the KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The motivation of the KIP is that a user doesn't want to miss
>>>>>> the data if
>>>>>>>>>>>>>> the backlog is small. The backlog of the existing partition is
>>>>>> easy to
>>>>>>>>>>>>>> understand because it relates to retention time. The backlog for
>>>>>> the new
>>>>>>>>>>>>>> partition is a bit subtle to understand since it depends on the
>>>>>> metadata
>>>>>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
>>>>>>> ,
>>>>>>>>>>>>>> the user needs to
>>>>>>>>>>>>>> understand the metadata refresh delay on the consumer side and
>>>>>> use it to
>>>>>>>>>>>>>> set the config.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Now, let's consider the alternative: setting the same value for
>>>>>> the
>>>>>>>>>>>>>> existing by_duration policy. The KIP lists three issues with
>>>> this
>>>>>>>>>>>>>> approach.
>>>>>>>>>>>>>> 1. It computes the seek target client-side as now() - duration,
>>>>>> which
>>>>>>>>>>>>>> introduces clock skew across consumers and forces operators to
>>>>>> choose
>>>>>>>>>>>>>> overly large durations, causing unnecessary reprocessing.
>>>>>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so failed
>>>>>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
>>>>>> potentially
>>>>>>>>>>>>>> miss records produced between attempts.
>>>>>>>>>>>>>> 3. It applies uniformly to all partitions without committed
>>>>>> offsets, and
>>>>>>>>>>>>>> cannot distinguish newly expanded partitions from long-existing
>>>>>> partitions
>>>>>>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
>>>>>> buffer to
>>>>>>>>>>>>>> the metadata refresh delay. We could also consider improving the
>>>>>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
>>>>>> typically low
>>>>>>>>>>>>>> (in the order of minutes with the classic consumer and tens of
>>>>>> seconds
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> the new consumer). If a user is ok with reading that much
>>>>>> backlog for new
>>>>>>>>>>>>>> partitions, it seems they will be ok doing the same for existing
>>>>>>>>>>>>>> partitions.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> So, instead of introducing a new config, could we just reuse the
>>>>>> existing
>>>>>>>>>>>>>> config with better documentation and/or implementation?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> You're right that group creation time is the more intuitive
>>>>>> answer at
>>>>>>>>>>>>>>> first glance,
>>>>>>>>>>>>>>> the KIP's own motivation talks about partitions that "predate
>>>>>> the group"
>>>>>>>>>>>>>>> vs partitions
>>>>>>>>>>>>>>> "created during group runtime," which directly points to a
>>>>>>>>>>>>>> group-lifecycle
>>>>>>>>>>>>>>> classifier.
>>>>>>>>>>>>>>> I'd like to walk through why we landed on partition age, and
>>>> the
>>>>>>>>>>>>>>> trade-offs we considered.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We evaluated three candidate signals:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. `by_duration:5secs`
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This covers the metadata blindness window, but has issues the
>>>>>> KIP
>>>>>>>>>>>>>>> currently documents
>>>>>>>>>>>>>>> under "Why not use `by_duration`?":
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew across
>>>>>> consumers.
>>>>>>>>>>>>>>> - `ListOffsets` retries shift the target forward, potentially
>>>>>> missing
>>>>>>>>>>>>>>> records produced between
>>>>>>>>>>>>>>> attempts.
>>>>>>>>>>>>>>> - It applies uniformly to all partitions without committed
>>>>>> offsets,
>>>>>>>>>>>>>>> including pre-existing partitions
>>>>>>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2. Group creation time as classifier
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This works cleanly when the consumer is actively running. Our
>>>>>> concern
>>>>>>>>>>>>>>> is the idle / late-rejoin case:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> T=0:         Group created.
>>>>>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>>>>>>>>>>>>> T=50:        Partition added during the idle window.
>>>>>>>>>>>>>>> T=100:       Consumer resumes.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Under group creation time, the new partition is classified as
>>>>>> new
>>>>>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from
>>>>>> T=50.
>>>>>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
>>>>>> data that
>>>>>>>>>>>>>>> the consumer accepts as lost — that is precisely the contract
>>>> of
>>>>>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
>>>>>> treat
>>>>>>>>>>>>>>> the new partition differently; both contain backlog accumulated
>>>>>> during
>>>>>>>>>>>>>>> the same idle window.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This aligns with the "backlog is backlog” principle you raised
>>>>>> in
>>>>>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog
>>>>>> on
>>>>>>>>>>>>>>> every other partition during the same idle period; forcing
>>>>>> 0-backlog
>>>>>>>>>>>>>>> tolerance only on new partitions would be inconsistent with
>>>> that
>>>>>>>>>>>>>>> tolerance.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 3. Partition age vs threshold
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Partition age corresponds to the actual silent data loss
>>>> window,
>>>>>>>>>>>>>>> the gap between partition creation and the consumer’s metadata
>>>>>>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: the
>>>>>>>>>>>>>>> consumer had no opportunity to know about the partition.
>>>>>> Outside this
>>>>>>>>>>>>>>> window, missing data reflects either:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle consumers,
>>>>>> or
>>>>>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not via
>>>>>> reset
>>>>>>>>>>>>>> policy.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> We did not choose partition age because it is more elegant than
>>>>>> group
>>>>>>>>>>>>>>> creation time — we chose it because its failure mode (requires
>>>> a
>>>>>>>>>>>>>>> threshold) is
>>>>>>>>>>>>>>> less invasive than the failure mode of group creation time
>>>>>> (overrides
>>>>>>>>>>>>>>> user-stated
>>>>>>>>>>>>>>> `latest` intent during idle periods).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Relying on both creation times will create an inconsistent
>>>>>> scenario. A
>>>>>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will seek
>>>>>> to the
>>>>>>>>>>>>>>>> beginning for the partitions created later than the group.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
>>>>>> inconsistency
>>>>>>>>>>>>>>> using a
>>>>>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
>>>>>> KIP-1327
>>>>>>>>>>>>>>> goes
>>>>>>>>>>>>>>>> back to using flexible configurations to prevent users from
>>>>>> falling
>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>> that pitfall.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
>>>>>> 上午6:49寫道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
>>>>>>> 
>>>>>>>>>>>>>> still feels weird to
>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>> categorizes partitions as new or existing based on the
>>>>>> partition
>>>>>>>>>>>>>>> creation
>>>>>>>>>>>>>>>>> time. Intuitively, the categorization should be based on the
>>>>>> group
>>>>>>>>>>>>>>> creation
>>>>>>>>>>>>>>>>> time: all partitions existing when the group is created are
>>>>>> existing
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> all partitions created after the group creation are new
>>>>>> partitions.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Manually bumping this thread. If there is no further
>>>>>>>>>>>>>>>>>> discussion, I will close the vote.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hello Jian,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for your feedback,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational task,
>>>>>> not an
>>>>>>>>>>>>>> edge
>>>>>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify
>>>>>> that
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
>>>>>>>>>>>>>> demonstrates
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> need for this optimization. For example:
>>>>>>>>>>>>>>>>>>>> A partition expansion is a common operational task in
>>>>>> Kafka: To
>>>>>>>>>>>>>>>>> balance
>>>>>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
>>>>>> created with a
>>>>>>>>>>>>>>>>>> moderate
>>>>>>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
>>>>>> time, it
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> often
>>>>>>>>>>>>>>>>>>>> necessary to increase the number of partitions to
>>>>>> accommodate the
>>>>>>>>>>>>>>>>> higher
>>>>>>>>>>>>>>>>>>>> workload.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>> Jian
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29
>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
>>>>>>>>>>>>>> describing
>>>>>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them
>>>> to
>>>>>>>>>>>>>> describe
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
>>>>>> applying
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>>> terms
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
>>>>>> whether a
>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
>>>>>> they won't
>>>>>>>>>>>>>>>>>> silently
>>>>>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during their
>>>>>> active
>>>>>>>>>>>>>>>>>>>>> consumption.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
>>>>>> motivation
>>>>>>>>>>>>>>> section.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
>>>>>> 凌晨1:12
>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
>>>>>> motivation
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
>>>>>> partitions
>>>>>>>>>>>>>> without
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
>>>>>> partition
>>>>>>>>>>>>>> (hot)
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer
>>>>>> has
>>>>>>>>>>>>>> never
>>>>>>>>>>>>>>>>> seen
>>>>>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
>>>>>>>>>>>>>>> insufficient
>>>>>>>>>>>>>>>>>>>>>>> because:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies
>>>>>> across
>>>>>>>>>>>>>> nodes
>>>>>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
>>>>>>>>>>>>>> duration,
>>>>>>>>>>>>>>>>>>>>>>> causing
>>>>>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek
>>>>>> time on
>>>>>>>>>>>>>>>>> retry,
>>>>>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data
>>>>>> loss."
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If these
>>>>>> issues
>>>>>>>>>>>>>>>>> persist,
>>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
>>>>>> situations don't
>>>>>>>>>>>>>>>>> need a
>>>>>>>>>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare situations,
>>>>>> they can
>>>>>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>>>>>>>>> customized logic using
>>>>>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
>>>>>> should
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>>>>>> any
>>>>>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
>>>>>>>>>>>>>> mismatch
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs
>>>>>> inside
>>>>>>>>>>>>>>>>> poll().
>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException
>>>>>> from
>>>>>>>>>>>>>> poll().
>>>>>>>>>>>>>>>>>> I’ll
>>>>>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
>>>>>> 下午4:50
>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>>>>>>>>>>>>>>>>> broker_current_time
>>>>>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
>>>>>> does not
>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
>>>>>> should
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>>>>>> any
>>>>>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
>>>>>> calling
>>>>>>>>>>>>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on
>>>>>> the
>>>>>>>>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
>>>> reset
>>>>>>>>>>>>>> behavior
>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
>>>>>>> 
>>>>>>>>>>>>>> now applies to all
>>>>>>>>>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets
>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> 
>>>>>>>>>>>>>> .partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>>>>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented
>>>>>> by a
>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> .partitions,
>>>>>>>>>>>>>> currently fixed to
>>>>>>>>>>>>>>>>> earliest).
>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
>>>>>> public
>>>>>>>>>>>>>>>>>> user-facing
>>>>>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
>>>>>> 清晨7:46
>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is
>>>>>> listed
>>>>>>>>>>>>>>> below:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> .partitions
>>>>>>>>>>>>>> with a default value
>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
>>>>>> by_duration and
>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>> does not change the logic of
>>>>>> auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> .partitions
>>>>>>>>>>>>>> as an internal
>>>>>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> 
>>>>>>>>>>>>>> .partitions=earliest
>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases
>>>>>> of
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> .partitions
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
>>>>>>> ,
>>>>>>>>>>>>>> will be applied to all for
>>>>>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When
>>>>>> a Kafka
>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest
>>>>>> auto
>>>>>>>>>>>>>> offset
>>>>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
>>>>>> record loss
>>>>>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> .partitions
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> take
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set
>>>>>> it
>>>>>>>>>>>>>>>>>> by_duration if
>>>>>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>>>>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
>>>>>> something
>>>>>>>>>>>>>>>>>> important
>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
>>>>>> clarify
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
>>>>>> logic
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest
>>>>>> or
>>>>>>>>>>>>>>>>> by_duration
>>>>>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue
>>>>>> when a
>>>>>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
>>>>>> configure for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> .partitions?
>>>>>>>>>>>>>> If they set it to
>>>>>>>>>>>>>>>>> earliest
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>>>>> if a
>>>>>>>>>>>>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
>>>>>> created
>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest
>>>>>> when
>>>>>>>>>>>>>>>>> resuming,
>>>>>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition
>>>>>> is
>>>>>>>>>>>>>>>>>> technically
>>>>>>>>>>>>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
>>>>>> max.age
>>>>>>>>>>>>>>>>>> threshold.
>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
>>>>>> genuinely
>>>>>>>>>>>>>>> "hot/new"
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
>>>>>> weird. It
>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems
>>>>>> that the
>>>>>>>>>>>>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
>>>>>> values
>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
>>>>>> way to
>>>>>>>>>>>>>>> control
>>>>>>>>>>>>>>>>>>>>> newly
>>>>>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
>>>> group
>>>>>>>>>>>>>> starts.
>>>>>>>>>>>>>>>>>> Have we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>>>>>> 
>>>>>>>>>>>>>>> .partitions?
>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
>>>>>> defaults to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
>>>>>> explicitly
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> customize
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
>>>>>> s7133700@gmail.com
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37
>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>>>>>>>>>>>>>>>>> PartitionMetadata
>>>>>>>>>>>>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
>>>>>> computed
>>>>>>>>>>>>>>>>>> server-side
>>>>>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
>>>> partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>>>>>>>>>>>>> MembershipManager
>>>>>>>>>>>>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
>>>>>> metadata for
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even
>>>>>> if the
>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>> ID
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
>>>>>> supported
>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time.
>>>>>> If the
>>>>>>>>>>>>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
>>>>>> PartitionAgeMs at
>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
>>>>>> back to
>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>>>>>> 2026年4月29日
>>>>>>>>>>>>>>> 下午4:04
>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
>>>>>> 'age'
>>>>>>>>>>>>>>>>> resolution
>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
>>>>>> control
>>>>>>>>>>>>>>>>> plane
>>>>>>>>>>>>>>>>>>>>>>> clean.
>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>>>>>>>>>>>>>>> inter-broker
>>>>>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
>>>>>> source of
>>>>>>>>>>>>>>>>> truth
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
>>>>>> should be
>>>>>>>>>>>>>>>>>> negligible.
>>>>>>>>>>>>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
>>>>>> configured in
>>>>>>>>>>>>>>> minutes
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
>>>>>> brokers
>>>>>>>>>>>>>> won't
>>>>>>>>>>>>>>>>>> impact
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org>
>>>>>> 於
>>>>>>>>>>>>>>>>> 2026年4月29日
>>>>>>>>>>>>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
>>>>>> previous
>>>>>>>>>>>>>>>>>> conversation
>>>>>>>>>>>>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow
>>>>>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
>>>>>> boolean;
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
>>>>>> boolean
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>>>>>>>>>>>>>> partition,
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive
>>>>>> the
>>>>>>>>>>>>>> ages.
>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow
>>>>>> and
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> explain
>>>>>>>>>>>>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets,
>>>>>> fetch
>>>>>>>>>>>>>>>>> offsets,
>>>>>>>>>>>>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
>>>>>> wonder if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
>>>>>> place
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively,
>>>>>> it could
>>>>>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
>>>>>> cautious
>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
>>>>>> unrelated
>>>>>>>>>>>>>>> concepts.
>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>>>>>>>>>>>>>>> partitions.
>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
>>>>>> Streams
>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we
>>>>>> do if
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
>>>>>> Muralidhar Basani
>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>> dev
>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
>>>>>> about
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>>>>>>>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
>>>>>> PartitionAges
>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
>>>>>> recommended
>>>>>>>>>>>>>>>>> value
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
>>>>>>>>>>>>>> <
>>>>>> 
>>>> https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
>>>>>> guess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to
>>>>>> Kafka
>>>>>>>>>>>>>>>>> Streams,
>>>>>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it?
>>>>>> This is
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
>>>>>> mistakenly
>>>>>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer
>>>>>> falls
>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
>>>>>> per-partition
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
>>>>>> resolves
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
>>>>>> earliest
>>>>>>>>>>>>>> was
>>>>>>>>>>>>>>>>>> applied
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
>>>>>> config is
>>>>>>>>>>>>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
>>>>>> change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>>>>>>>>>>>>> 2026年4月28日
>>>>>>>>>>>>>>>>>> 晚上8:03
>>>>>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
>>>>>> creation
>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users
>>>> to
>>>>>>>>>>>>>> diagnose
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>>>>>> KIP-1327
>>>>>>>>>>>>>>>>>> Prevent
>>>>>>>>>>>>>>>>>>>>> Hot
>>>>>>>>>>>>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>> 
>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
>>>>>> expanded
>>>>>>>>>>>>>>>>> (hot)
>>>>>>>>>>>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>>>>>>>>>>>>>>> automatically
>>>>>>>>>>>>>>>>>> fall
>>>>>>>>>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing
>>>>>> a full
>>>>>>>>>>>>>>>>>> historical
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
> It inspires me to think about mirror clusters. What if the mirror group is created 'after' the expanded partitions? In this scenario, the new configuration 'max.age.ms' could handle it if users are aware of the migration and set a suitable time interval. Of course, this would require both creation times to be exposed to clients.

Thinking more about the mirror cluster scenario we discussed, I realize it introduces significant complexity. Handling it properly would require a tolerance time window rather than a strict time interval to account for replication lag. To prevent scope creep and keep KIP-1327 focused on its primary goal, I think it would be best to defer the multi-cluster/mirroring scenario and explore those approaches in a separate KIP

On 2026/06/24 21:29:19 Chia-Ping Tsai wrote:
> hi all,
> 
> > JR3. The KIP exposes the group/partition creation time to the client. I am
> > not sure if there are other use cases for those timestamps. An alternative
> > is to avoid exposing those timestamps to the client. The group coordinator
> > will categorize the partitions using those timestamps and include the
> > categorization in the assigned partitions in GroupHeartbeatResponse.
> 
> It inspires me to think about mirror clusters. What if the mirror group is created 'after' the expanded partitions? In this scenario, the new configuration 'max.age.ms' could handle it if users are aware of the migration and set a suitable time interval. Of course, this would require both creation times to be exposed to clients.
> 
> Best,
> Chia-Ping
> 
> On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> > Hi, Jiunn-Yang,
> > 
> > Thanks for the updated KIP.
> > 
> > JR2. Share consumer (KIP-932) states: "If the number of partitions is
> > increased for a topic with a subscription in a share group, the SPSO for
> > the newly created share-partitions is initialized to 0 ". However, this
> > part has not been implemented yet. Currently, all partitions without offset
> > follows the strategy given by share.auto.offset.reset. So, we have an
> > opportunity to see if it's better to align this KIP's changes with the
> > share consumer.
> > JR2.1 The first part is whether customizing the offset for new partitions
> > to something other than 0 is useful in the share consumer. It seems that
> > the motivation in this KIP applies to share consumer too.
> > JR2.2 The second part is how new partitions are detected. The mechanism
> > introduced in this KIP seems quite effective. A group coordinator could
> > pick up newly created partitions for an existing topic or newly created
> > topics matching the regex. Those partitions are categorized as new
> > partitions and will follow the strategy given by
> > auto.offset.reset.new.partitions.
> > This makes sense since those partitions typically don't have a large
> > backlog. A consumer could also additionally subscribe to an existing topic
> > in an existing group. Those partitions will be categorized as existing
> > partitions. This also makes sense since those partitions could have a large
> > backlog and it is better to follow the strategy given by auto.offset.reset. It
> > will be useful to think through whether the logic for categorizing new
> > partitions should also be reused in the share consumer.
> > 
> > JR3. The KIP exposes the group/partition creation time to the client. I am
> > not sure if there are other use cases for those timestamps. An alternative
> > is to avoid exposing those timestamps to the client. The group coordinator
> > will categorize the partitions using those timestamps and include the
> > categorization in the assigned partitions in GroupHeartbeatResponse.
> > 
> > JR4. It would be useful to support this feature on existing groups as well.
> > For example, any partition without a creation timestamp can always be
> > categorized as an existing partition.
> > 
> > Jun
> > 
> > On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
> > 
> > > Hello Andrew,
> > >
> > > Thanks for your feedback,
> > > >> I've just been re-reading this KIP to ensure that share group support is
> > > >> not needed. I've discovered that we made a mistake implementing KIP-932
> > > in
> > > >> this area, so we'll rectify that and check whether it's really tight
> > > enough
> > > >> not to require KIP-1327.
> > >
> > > Could you walk us through the specific area where the mistake was found?
> > > That would
> > > help us assess whether it naturally fits within the scope of KIP-1327 or
> > > should be addressed
> > > separately.
> > >
> > > AS1 I have updated the KIP
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > > > Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> > > >
> > > > hi Andrew
> > > >
> > > >> I've discovered that we made a mistake implementing KIP-932 in this
> > > area,
> > > > so we'll rectify that and check whether it's really tight enough not to
> > > > require KIP-1327.
> > > >
> > > > Would you mind sharing more details about that? If it's not anything
> > > close
> > > > to NP-hard, we could definitely look into covering it within KIP-1327
> > > >
> > > > Best,
> > > > Chia-Ping
> > > >
> > > >
> > > > Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
> > > >
> > > >> Hi Jiunn-Yang,
> > > >> I've just been re-reading this KIP to ensure that share group support is
> > > >> not needed. I've discovered that we made a mistake implementing KIP-932
> > > in
> > > >> this area, so we'll rectify that and check whether it's really tight
> > > enough
> > > >> not to require KIP-1327.
> > > >>
> > > >> One trivial comments for consistency.
> > > >>
> > > >> AS1: In kafka-consumer-groups.sh, unknown column data is represented by
> > > >> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > > >>> Hi all,
> > > >>>
> > > >>> Manually bumping this thread.
> > > >>>
> > > >>> Best Regards,
> > > >>> Jiunn-Yang
> > > >>>
> > > >>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > > >>>>
> > > >>>> Hello chia,
> > > >>>>
> > > >>>> Thanks for the feedback, I have updated the KIP.
> > > >>>>
> > > >>>> Best Regards,
> > > >>>> Jiunn-Yang
> > > >>>>
> > > >>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> > > >>>>>
> > > >>>>> hi Jiunn-Yang
> > > >>>>>
> > > >>>>>> When the config is set on a cluster that has not yet been
> > > >> upgraded... classification cannot occur... the consumer falls back to
> > > the
> > > >> base auto.offset.reset for the affected partitions. No exception is
> > > thrown,
> > > >> and no operational disruption results.
> > > >>>>>
> > > >>>>> Existing group can't take advantage of this excellent new
> > > >> configuration. Allowing users to modify the group creation time might be
> > > >> overkill. Instead, we could print a useful warning message to guide
> > > users.
> > > >> For example, we can suggest that they re-create the group with their
> > > >> existing committed offsets
> > > >>>>>
> > > >>>>>> Protocol changes
> > > >>>>>
> > > >>>>> Would you mind listing those RPC changes in a table format?
> > > >>>>>
> > > >>>>>> The full interaction matrix between the base policy and the
> > > >> new-partition policy is:
> > > >>>>>
> > > >>>>> Please add a filed to describe the target scenario when using these
> > > >> policies
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Chia-Ping
> > > >>>>>
> > > >>>>>
> > > >>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > > >>>>>> Hello Jun, chia,
> > > >>>>>>
> > > >>>>>> Thanks for the feedback, I have updated the KIP for the new
> > > >>>>>> approach, PTAL
> > > >>>>>>
> > > >>>>>> Best Regards,
> > > >>>>>> Jiunn-Yang
> > > >>>>>>
> > > >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> > > >>>>>>>
> > > >>>>>>> hi Jun
> > > >>>>>>>
> > > >>>>>>> Yes, your approach is great. I think the combination of latest (for
> > > >> existing partitions) and by_duration (for new partitions) can address
> > > 99%
> > > >> of the complaints I have heard regarding this issue.
> > > >>>>>>>
> > > >>>>>>> Also, leveraging the group creation time here opens the door to
> > > >> implementing a new policy based on timestamp seek in the future, should
> > > the
> > > >> community want to pursue that.
> > > >>>>>>>
> > > >>>>>>> Thanks for your patience and constructive feedback. We will update
> > > >> the KIP accordingly.
> > > >>>>>>>
> > > >>>>>>> Best, Chia-Ping
> > > >>>>>>>
> > > >>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
> > > >>>>>>>>
> > > >>>>>>>> Hi, Chia-Ping,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the reply.
> > > >>>>>>>>
> > > >>>>>>>> I agree that it's probably useful to allow a user to configure a
> > > >> different
> > > >>>>>>>> offset policy for existing partitions vs new partitions. However,
> > > >> using
> > > >>>>>>>> group creation time to capture that seems more intuitive. Here is
> > > >> another
> > > >>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new
> > > >> partitions
> > > >>>>>>>> based on group creation time. Introduce
> > > >>>>>>>> a new config auto.offset.reset.new.partitions whose values can be
> > > >> earliest,
> > > >>>>>>>> latest and by_duration, the same as auto.offset.reset. Users can
> > > >> set
> > > >>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to
> > > >> guarantee
> > > >>>>>>>> no data loss on new partitions. They can also use by_duration to
> > > >> set an
> > > >>>>>>>> upper bound on the backlog replayed, which can be different from
> > > >> that of
> > > >>>>>>>> the existing partitions. This will address your concern about too
> > > >> much
> > > >>>>>>>> backlog being replayed when the offsets are lost. What do you
> > > >> think?
> > > >>>>>>>>
> > > >>>>>>>> Jun
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > > >> chia7712@gmail.com> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> hi Jun
> > > >>>>>>>>>
> > > >>>>>>>>> The most important part of this story is how users should expect
> > > >> the data
> > > >>>>>>>>> they can see when using the latest or by_duration policy with
> > > >> expanded
> > > >>>>>>>>> partitions.
> > > >>>>>>>>>
> > > >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > > >>>>>>>>> non-deterministic, which means users will either read too many
> > > >> historical
> > > >>>>>>>>> records from existing partitions or lose some records from
> > > >> expanded
> > > >>>>>>>>> partitions.
> > > >>>>>>>>>
> > > >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > > >>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > >>>
> > > >>>>>>>>> is a bit hard to understand, and that is why I preferred having a
> > > >> whole new
> > > >>>>>>>>> policy based entirely on group creation time (KIP-1282)
> > > >>>>>>>>>
> > > >>>>>>>>> Best,
> > > >>>>>>>>> Chia-Ping
> > > >>>>>>>>>
> > > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> > > >>>>>>>>>
> > > >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for the reply. I am still trying to understand the value
> > > >> of the new
> > > >>>>>>>>>> configs with the KIP.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The motivation of the KIP is that a user doesn't want to miss
> > > >> the data if
> > > >>>>>>>>>> the backlog is small. The backlog of the existing partition is
> > > >> easy to
> > > >>>>>>>>>> understand because it relates to retention time. The backlog for
> > > >> the new
> > > >>>>>>>>>> partition is a bit subtle to understand since it depends on the
> > > >> metadata
> > > >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > >>> ,
> > > >>>>>>>>>> the user needs to
> > > >>>>>>>>>> understand the metadata refresh delay on the consumer side and
> > > >> use it to
> > > >>>>>>>>>> set the config.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Now, let's consider the alternative: setting the same value for
> > > >> the
> > > >>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> > > this
> > > >>>>>>>>>> approach.
> > > >>>>>>>>>> 1. It computes the seek target client-side as now() - duration,
> > > >> which
> > > >>>>>>>>>> introduces clock skew across consumers and forces operators to
> > > >> choose
> > > >>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> > > >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so failed
> > > >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> > > >> potentially
> > > >>>>>>>>>> miss records produced between attempts.
> > > >>>>>>>>>> 3. It applies uniformly to all partitions without committed
> > > >> offsets, and
> > > >>>>>>>>>> cannot distinguish newly expanded partitions from long-existing
> > > >> partitions
> > > >>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
> > > >> buffer to
> > > >>>>>>>>>> the metadata refresh delay. We could also consider improving the
> > > >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> > > >> typically low
> > > >>>>>>>>>> (in the order of minutes with the classic consumer and tens of
> > > >> seconds
> > > >>>>>>>>>> with
> > > >>>>>>>>>> the new consumer). If a user is ok with reading that much
> > > >> backlog for new
> > > >>>>>>>>>> partitions, it seems they will be ok doing the same for existing
> > > >>>>>>>>>> partitions.
> > > >>>>>>>>>>
> > > >>>>>>>>>> So, instead of introducing a new config, could we just reuse the
> > > >> existing
> > > >>>>>>>>>> config with better documentation and/or implementation?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Jun
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > > >> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> You're right that group creation time is the more intuitive
> > > >> answer at
> > > >>>>>>>>>>> first glance,
> > > >>>>>>>>>>> the KIP's own motivation talks about partitions that "predate
> > > >> the group"
> > > >>>>>>>>>>> vs partitions
> > > >>>>>>>>>>> "created during group runtime," which directly points to a
> > > >>>>>>>>>> group-lifecycle
> > > >>>>>>>>>>> classifier.
> > > >>>>>>>>>>> I'd like to walk through why we landed on partition age, and
> > > the
> > > >>>>>>>>>>> trade-offs we considered.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> We evaluated three candidate signals:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 1. `by_duration:5secs`
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This covers the metadata blindness window, but has issues the
> > > >> KIP
> > > >>>>>>>>>>> currently documents
> > > >>>>>>>>>>> under "Why not use `by_duration`?":
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew across
> > > >> consumers.
> > > >>>>>>>>>>> - `ListOffsets` retries shift the target forward, potentially
> > > >> missing
> > > >>>>>>>>>>> records produced between
> > > >>>>>>>>>>> attempts.
> > > >>>>>>>>>>> - It applies uniformly to all partitions without committed
> > > >> offsets,
> > > >>>>>>>>>>> including pre-existing partitions
> > > >>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 2. Group creation time as classifier
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This works cleanly when the consumer is actively running. Our
> > > >> concern
> > > >>>>>>>>>>> is the idle / late-rejoin case:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> T=0:         Group created.
> > > >>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > > >>>>>>>>>>> T=50:        Partition added during the idle window.
> > > >>>>>>>>>>> T=100:       Consumer resumes.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Under group creation time, the new partition is classified as
> > > >> new
> > > >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from
> > > >> T=50.
> > > >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> > > >> data that
> > > >>>>>>>>>>> the consumer accepts as lost — that is precisely the contract
> > > of
> > > >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
> > > >> treat
> > > >>>>>>>>>>> the new partition differently; both contain backlog accumulated
> > > >> during
> > > >>>>>>>>>>> the same idle window.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> This aligns with the "backlog is backlog” principle you raised
> > > >> in
> > > >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog
> > > >> on
> > > >>>>>>>>>>> every other partition during the same idle period; forcing
> > > >> 0-backlog
> > > >>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> > > that
> > > >>>>>>>>>>> tolerance.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> 3. Partition age vs threshold
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Partition age corresponds to the actual silent data loss
> > > window,
> > > >>>>>>>>>>> the gap between partition creation and the consumer’s metadata
> > > >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: the
> > > >>>>>>>>>>> consumer had no opportunity to know about the partition.
> > > >> Outside this
> > > >>>>>>>>>>> window, missing data reflects either:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle consumers,
> > > >> or
> > > >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not via
> > > >> reset
> > > >>>>>>>>>> policy.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> We did not choose partition age because it is more elegant than
> > > >> group
> > > >>>>>>>>>>> creation time — we chose it because its failure mode (requires
> > > a
> > > >>>>>>>>>>> threshold) is
> > > >>>>>>>>>>> less invasive than the failure mode of group creation time
> > > >> (overrides
> > > >>>>>>>>>>> user-stated
> > > >>>>>>>>>>> `latest` intent during idle periods).
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Hi Jun,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Relying on both creation times will create an inconsistent
> > > >> scenario. A
> > > >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will seek
> > > >> to the
> > > >>>>>>>>>>>> beginning for the partitions created later than the group.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> > > >> inconsistency
> > > >>>>>>>>>>> using a
> > > >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
> > > >> KIP-1327
> > > >>>>>>>>>>> goes
> > > >>>>>>>>>>>> back to using flexible configurations to prevent users from
> > > >> falling
> > > >>>>>>>>>> into
> > > >>>>>>>>>>>> that pitfall.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Best, Chia-Ping
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > > >> 上午6:49寫道:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > >>>
> > > >>>>>>>>>> still feels weird to
> > > >>>>>>>>>>> me.
> > > >>>>>>>>>>>>> It
> > > >>>>>>>>>>>>> categorizes partitions as new or existing based on the
> > > >> partition
> > > >>>>>>>>>>> creation
> > > >>>>>>>>>>>>> time. Intuitively, the categorization should be based on the
> > > >> group
> > > >>>>>>>>>>> creation
> > > >>>>>>>>>>>>> time: all partitions existing when the group is created are
> > > >> existing
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>> all partitions created after the group creation are new
> > > >> partitions.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > > >> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> > > >>>>>>>>>>>>>> discussion, I will close the vote.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Hello Jian,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Thanks for your feedback,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational task,
> > > >> not an
> > > >>>>>>>>>> edge
> > > >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify
> > > >> that
> > > >>>>>>>>>> this
> > > >>>>>>>>>>>>> is a
> > > >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
> > > >>>>>>>>>> demonstrates
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>> need for this optimization. For example:
> > > >>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> > > >> Kafka: To
> > > >>>>>>>>>>>>> balance
> > > >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> > > >> created with a
> > > >>>>>>>>>>>>>> moderate
> > > >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
> > > >> time, it
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>>>> often
> > > >>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> > > >> accommodate the
> > > >>>>>>>>>>>>> higher
> > > >>>>>>>>>>>>>>>> workload.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Regards
> > > >>>>>>>>>>>>>>>> Jian
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hello chia,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29
> > > >> 寫道:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> > > >>>>>>>>>> describing
> > > >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them
> > > to
> > > >>>>>>>>>> describe
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> > > >> applying
> > > >>>>>>>>>> these
> > > >>>>>>>>>>>>>> terms
> > > >>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> > > >> whether a
> > > >>>>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
> > > >> they won't
> > > >>>>>>>>>>>>>> silently
> > > >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during their
> > > >> active
> > > >>>>>>>>>>>>>>>>> consumption.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hello Jun,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > > >> motivation
> > > >>>>>>>>>>> section.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
> > > >> 凌晨1:12
> > > >>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > > >> motivation
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> KIP.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> > > >> partitions
> > > >>>>>>>>>> without
> > > >>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> > > >> partition
> > > >>>>>>>>>> (hot)
> > > >>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer
> > > >> has
> > > >>>>>>>>>> never
> > > >>>>>>>>>>>>> seen
> > > >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> > > >>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> > > >>>>>>>>>>> insufficient
> > > >>>>>>>>>>>>>>>>>>> because:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies
> > > >> across
> > > >>>>>>>>>> nodes
> > > >>>>>>>>>>>>>> due
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> > > >>>>>>>>>> duration,
> > > >>>>>>>>>>>>>>>>>>> causing
> > > >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > > >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek
> > > >> time on
> > > >>>>>>>>>>>>> retry,
> > > >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data
> > > >> loss."
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If these
> > > >> issues
> > > >>>>>>>>>>>>> persist,
> > > >>>>>>>>>>>>>>>>> more
> > > >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > > >> situations don't
> > > >>>>>>>>>>>>> need a
> > > >>>>>>>>>>>>>>>>>>> common
> > > >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare situations,
> > > >> they can
> > > >>>>>>>>>>>>>> implement
> > > >>>>>>>>>>>>>>>>>>>> customized logic using
> > > >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > > >> s7133700@gmail.com>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > >> should
> > > >>>>>>>>>> always
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>> greater
> > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > >>  any
> > > >>>>>>>>>>>>>> field
> > > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
> > > >>>>>>>>>> mismatch
> > > >>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>> during
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs
> > > >> inside
> > > >>>>>>>>>>>>> poll().
> > > >>>>>>>>>>>>>>>>>>>>> Therefore, the
> > > >>>>>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException
> > > >> from
> > > >>>>>>>>>> poll().
> > > >>>>>>>>>>>>>> I’ll
> > > >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
> > > >> 下午4:50
> > > >>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> > > >>>>>>>>>> partition
> > > >>>>>>>>>>>>> in
> > > >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> > > >>>>>>>>>>>>>>>>> broker_current_time
> > > >>>>>>>>>>>>>>>>>>> -
> > > >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > > >> does not
> > > >>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > > >> should
> > > >>>>>>>>>> always
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>> greater
> > > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > > >>  any
> > > >>>>>>>>>>>>>> field
> > > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
> > > >> calling
> > > >>>>>>>>>>>>>>>>> `poll()`?
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on
> > > >> the
> > > >>>>>>>>>>>>>> discussion
> > > >>>>>>>>>>>>>>>>>>>>>>> feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > > reset
> > > >>>>>>>>>> behavior
> > > >>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > > >>>
> > > >>>>>>>>>> now applies to all
> > > >>>>>>>>>>>>>> auto.offset.reset
> > > >>>>>>>>>>>>>>>>>>>>> values
> > > >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > > >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets
> > > to
> > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>
> > > >>>>>>>>>> .partitions
> > > >>>>>>>>>>>>>>>>>>>>>>> config setting
> > > >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > > >>>>>>>>>> auto.offset.reset
> > > >>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>> continues
> > > >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > > >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented
> > > >> by a
> > > >>>>>>>>>>> separate
> > > >>>>>>>>>>>>>>>>>>>>> internal config
> > > >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>> .partitions,
> > > >>>>>>>>>> currently fixed to
> > > >>>>>>>>>>>>> earliest).
> > > >>>>>>>>>>>>>>>>> This
> > > >>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > > >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > > >> public
> > > >>>>>>>>>>>>>> user-facing
> > > >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
> > > >> 清晨7:46
> > > >>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is
> > > >> listed
> > > >>>>>>>>>>> below:
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>> .partitions
> > > >>>>>>>>>> with a default value
> > > >>>>>>>>>>>>> of
> > > >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > > >> by_duration and
> > > >>>>>>>>>>>>> latest,
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > > >> auto.offset.reset=earliest.
> > > >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>> .partitions
> > > >>>>>>>>>> as an internal
> > > >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>
> > > >>>>>>>>>> .partitions=earliest
> > > >>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases
> > > >> of
> > > >>>>>>>>>> other
> > > >>>>>>>>>>>>>> values
> > > >>>>>>>>>>>>>>>>>>> in a
> > > >>>>>>>>>>>>>>>>>>>>> separate KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>> .partitions
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > > >>> ,
> > > >>>>>>>>>> will be applied to all for
> > > >>>>>>>>>>>>>>>>>>>>> consistency.
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > > >>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When
> > > >> a Kafka
> > > >>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>> expanded
> > > >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest
> > > >> auto
> > > >>>>>>>>>> offset
> > > >>>>>>>>>>>>>> reset
> > > >>>>>>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> > > >>>>>>>>>> partitions
> > > >>>>>>>>>>>>>> before
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > > >> record loss
> > > >>>>>>>>>>>>> issue
> > > >>>>>>>>>>>>>>>>> could
> > > >>>>>>>>>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>> .partitions
> > > >>>>>>>>>> will
> > > >>>>>>>>>>> take
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> same
> > > >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set
> > > >> it
> > > >>>>>>>>>>>>>> by_duration if
> > > >>>>>>>>>>>>>>>>>>>>> needed.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > > >>>>>>>>>>>>>>>>> chia7712@apache.org
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > > >> something
> > > >>>>>>>>>>>>>> important
> > > >>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
> > > >> clarify
> > > >>>>>>>>>> with
> > > >>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>> few
> > > >>>>>>>>>>>>>>>>>>>>> questions:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
> > > >> logic
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>> other
> > > >>>>>>>>>>>>>>>>>>> reset
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest
> > > >> or
> > > >>>>>>>>>>>>> by_duration
> > > >>>>>>>>>>>>>>>>>>> don't
> > > >>>>>>>>>>>>>>>>>>>>> seem
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue
> > > >> when a
> > > >>>>>>>>>>>>>> partition
> > > >>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>> expanded.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > > >> configure for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>> .partitions?
> > > >>>>>>>>>> If they set it to
> > > >>>>>>>>>>>>> earliest
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>> latest,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> > > >>>>>>>>>> example,
> > > >>>>>>>>>>>>> if a
> > > >>>>>>>>>>>>>>>>>>>>> consumer is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > > >> created
> > > >>>>>>>>>> during
> > > >>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>> downtime,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest
> > > >> when
> > > >>>>>>>>>>>>> resuming,
> > > >>>>>>>>>>>>>>>>>>> rather
> > > >>>>>>>>>>>>>>>>>>>>> than
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition
> > > >> is
> > > >>>>>>>>>>>>>> technically
> > > >>>>>>>>>>>>>>>>>>>>> "new" to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
> > > >> max.age
> > > >>>>>>>>>>>>>> threshold.
> > > >>>>>>>>>>>>>>>>>>> It
> > > >>>>>>>>>>>>>>>>>>>>> gives
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > > >> genuinely
> > > >>>>>>>>>>> "hot/new"
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> what is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
> > > >> weird. It
> > > >>>>>>>>>>>>> only
> > > >>>>>>>>>>>>>>>>>>>>> applies when
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems
> > > >> that the
> > > >>>>>>>>>>>>>>>>> motivation
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
> > > >> values
> > > >>>>>>>>>> like
> > > >>>>>>>>>>>>>>>>>>>>> by_duration.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> > > >> way to
> > > >>>>>>>>>>> control
> > > >>>>>>>>>>>>>>>>> newly
> > > >>>>>>>>>>>>>>>>>>>>> created
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > > group
> > > >>>>>>>>>> starts.
> > > >>>>>>>>>>>>>> Have we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > > >>>
> > > >>>>>>>>>>> .partitions?
> > > >>>>>>>>>>>>>> If
> > > >>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > > >> defaults to
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> policy
> > > >>>>>>>>>>>>>>>>>>>>> used
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > > >> explicitly
> > > >>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>> customize
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > > >> s7133700@gmail.com
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37
> > > >> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > > >>>>>>>>>>>>>> PartitionMetadata
> > > >>>>>>>>>>>>>>>>>>>>> struct
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > > >> computed
> > > >>>>>>>>>>>>>> server-side
> > > >>>>>>>>>>>>>>>>>>> by
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > > partition_creation_time.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > > >>>>>>>>>>>>>> MembershipManager
> > > >>>>>>>>>>>>>>>>>>>>> detects
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > > >> metadata for
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> affected
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even
> > > >> if the
> > > >>>>>>>>>>>>> topic
> > > >>>>>>>>>>>>>> ID
> > > >>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> already
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > > >> supported
> > > >>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time.
> > > >> If the
> > > >>>>>>>>>>>>>> negotiated
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > > >> PartitionAgeMs at
> > > >>>>>>>>>> all
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>> can
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > > >> back to
> > > >>>>>>>>>> latest
> > > >>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>> risking
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > > >> 2026年4月29日
> > > >>>>>>>>>>> 下午4:04
> > > >>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > > >> 'age'
> > > >>>>>>>>>>>>> resolution
> > > >>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
> > > >> control
> > > >>>>>>>>>>>>> plane
> > > >>>>>>>>>>>>>>>>>>> clean.
> > > >>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> main
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > > >>>>>>>>>>> inter-broker
> > > >>>>>>>>>>>>>>>>> clock
> > > >>>>>>>>>>>>>>>>>>>>> skew.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
> > > >> source of
> > > >>>>>>>>>>>>> truth
> > > >>>>>>>>>>>>>>>>> for
> > > >>>>>>>>>>>>>>>>>>>>> time.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > > >> should be
> > > >>>>>>>>>>>>>> negligible.
> > > >>>>>>>>>>>>>>>>>>>>> Given
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > > >> configured in
> > > >>>>>>>>>>> minutes
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>>>>>> hours, a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > > >> brokers
> > > >>>>>>>>>> won't
> > > >>>>>>>>>>>>>> impact
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org>
> > > >> 於
> > > >>>>>>>>>>>>> 2026年4月29日
> > > >>>>>>>>>>>>>>>>>>> 下午3:29
> > > >>>>>>>>>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > > >> previous
> > > >>>>>>>>>>>>>> conversation
> > > >>>>>>>>>>>>>>>>>>> but I
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow
> > > >> with the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
> > > >> boolean;
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> KIP
> > > >>>>>>>>>>>>>>>>>>>>> states
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
> > > >> boolean
> > > >>>>>>>>>> is
> > > >>>>>>>>>>>>>> set.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> > > >>>>>>>>>> partition,
> > > >>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive
> > > >> the
> > > >>>>>>>>>> ages.
> > > >>>>>>>>>>> Is
> > > >>>>>>>>>>>>>> my
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow
> > > >> and
> > > >>>>>>>>>> also
> > > >>>>>>>>>>>>>> explain
> > > >>>>>>>>>>>>>>>>>>>>> how it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets,
> > > >> fetch
> > > >>>>>>>>>>>>> offsets,
> > > >>>>>>>>>>>>>>>>>>> etc.).
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > > >> wonder if
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
> > > >> place
> > > >>>>>>>>>> for
> > > >>>>>>>>>>>>>> this
> > > >>>>>>>>>>>>>>>>>>> given
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively,
> > > >> it could
> > > >>>>>>>>>>>>> simply
> > > >>>>>>>>>>>>>>>>>>>>> include
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > > >> cautious
> > > >>>>>>>>>>> about
> > > >>>>>>>>>>>>>> not
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > > >> unrelated
> > > >>>>>>>>>>> concepts.
> > > >>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>> API
> > > >>>>>>>>>>>>>>>>>>>>> is
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > > >>>>>>>>>>> partitions.
> > > >>>>>>>>>>>>>> The
> > > >>>>>>>>>>>>>>>>>>> fact
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
> > > >> Streams
> > > >>>>>>>>>> API
> > > >>>>>>>>>>>>>> also
> > > >>>>>>>>>>>>>>>>>>>>> suggests
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we
> > > >> do if
> > > >>>>>>>>>> we
> > > >>>>>>>>>>>>>> want to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> support
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > > >> Muralidhar Basani
> > > >>>>>>>>>>> via
> > > >>>>>>>>>>>>>> dev
> > > >>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
> > > >> about
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> gap.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > > >>>>>>>>>>>>>> RequestPartitionAges
> > > >>>>>>>>>>>>>>>>>>>>> field.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > > >> PartitionAges
> > > >>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > > >> recommended
> > > >>>>>>>>>>>>> value
> > > >>>>>>>>>>>>>> or
> > > >>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > > >>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > > >>>>>>>>>> <
> > > >>
> > > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > > >>>
> > > >>>>>>>>>>>>>>>>>>>>> ?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > > >> guess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to
> > > >> Kafka
> > > >>>>>>>>>>>>> Streams,
> > > >>>>>>>>>>>>>>>>>>> would
> > > >>>>>>>>>>>>>>>>>>>>> it
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > > >>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it?
> > > >> This is
> > > >>>>>>>>>> the
> > > >>>>>>>>>>>>>> most
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > > >> mistakenly
> > > >>>>>>>>>>>>>> configure
> > > >>>>>>>>>>>>>>>>>>>>> it. Or
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer
> > > >> falls
> > > >>>>>>>>>> back
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>> earliest"
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > > >> per-partition
> > > >>>>>>>>>> which
> > > >>>>>>>>>>>>>> isn't
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
> > > >> resolves
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>> initial
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > > >> earliest
> > > >>>>>>>>>> was
> > > >>>>>>>>>>>>>> applied
> > > >>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > > >> config is
> > > >>>>>>>>>>>>>> unchanged.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > > >>>>>>>>>>>>>> s7133700@gmail.com>
> > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> > > >> change.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > > >>>>>>>>>> 2026年4月28日
> > > >>>>>>>>>>>>>> 晚上8:03
> > > >>>>>>>>>>>>>>>>>>> 寫道:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
> > > >> creation
> > > >>>>>>>>>>> time
> > > >>>>>>>>>>>>>> via
> > > >>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users
> > > to
> > > >>>>>>>>>> diagnose
> > > >>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > > >> auto.offset.reset.latest.max.age
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> > > >> KIP-1327
> > > >>>>>>>>>>>>>> Prevent
> > > >>>>>>>>>>>>>>>>> Hot
> > > >>>>>>>>>>>>>>>>>>>>> Data
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>
> > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
> > > >> expanded
> > > >>>>>>>>>>>>> (hot)
> > > >>>>>>>>>>>>>>>>>>>>> partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > > >>>>>>>>>>> automatically
> > > >>>>>>>>>>>>>> fall
> > > >>>>>>>>>>>>>>>>>>>>> back
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing
> > > >> a full
> > > >>>>>>>>>>>>>> historical
> > > >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > >
> > >
> > 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
hi all,

> JR3. The KIP exposes the group/partition creation time to the client. I am
> not sure if there are other use cases for those timestamps. An alternative
> is to avoid exposing those timestamps to the client. The group coordinator
> will categorize the partitions using those timestamps and include the
> categorization in the assigned partitions in GroupHeartbeatResponse.

It inspires me to think about mirror clusters. What if the mirror group is created 'after' the expanded partitions? In this scenario, the new configuration 'max.age.ms' could handle it if users are aware of the migration and set a suitable time interval. Of course, this would require both creation times to be exposed to clients.

Best,
Chia-Ping

On 2026/06/24 20:46:16 Jun Rao via dev wrote:
> Hi, Jiunn-Yang,
> 
> Thanks for the updated KIP.
> 
> JR2. Share consumer (KIP-932) states: "If the number of partitions is
> increased for a topic with a subscription in a share group, the SPSO for
> the newly created share-partitions is initialized to 0 ". However, this
> part has not been implemented yet. Currently, all partitions without offset
> follows the strategy given by share.auto.offset.reset. So, we have an
> opportunity to see if it's better to align this KIP's changes with the
> share consumer.
> JR2.1 The first part is whether customizing the offset for new partitions
> to something other than 0 is useful in the share consumer. It seems that
> the motivation in this KIP applies to share consumer too.
> JR2.2 The second part is how new partitions are detected. The mechanism
> introduced in this KIP seems quite effective. A group coordinator could
> pick up newly created partitions for an existing topic or newly created
> topics matching the regex. Those partitions are categorized as new
> partitions and will follow the strategy given by
> auto.offset.reset.new.partitions.
> This makes sense since those partitions typically don't have a large
> backlog. A consumer could also additionally subscribe to an existing topic
> in an existing group. Those partitions will be categorized as existing
> partitions. This also makes sense since those partitions could have a large
> backlog and it is better to follow the strategy given by auto.offset.reset. It
> will be useful to think through whether the logic for categorizing new
> partitions should also be reused in the share consumer.
> 
> JR3. The KIP exposes the group/partition creation time to the client. I am
> not sure if there are other use cases for those timestamps. An alternative
> is to avoid exposing those timestamps to the client. The group coordinator
> will categorize the partitions using those timestamps and include the
> categorization in the assigned partitions in GroupHeartbeatResponse.
> 
> JR4. It would be useful to support this feature on existing groups as well.
> For example, any partition without a creation timestamp can always be
> categorized as an existing partition.
> 
> Jun
> 
> On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
> > Hello Andrew,
> >
> > Thanks for your feedback,
> > >> I've just been re-reading this KIP to ensure that share group support is
> > >> not needed. I've discovered that we made a mistake implementing KIP-932
> > in
> > >> this area, so we'll rectify that and check whether it's really tight
> > enough
> > >> not to require KIP-1327.
> >
> > Could you walk us through the specific area where the mistake was found?
> > That would
> > help us assess whether it naturally fits within the scope of KIP-1327 or
> > should be addressed
> > separately.
> >
> > AS1 I have updated the KIP
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> > >
> > > hi Andrew
> > >
> > >> I've discovered that we made a mistake implementing KIP-932 in this
> > area,
> > > so we'll rectify that and check whether it's really tight enough not to
> > > require KIP-1327.
> > >
> > > Would you mind sharing more details about that? If it's not anything
> > close
> > > to NP-hard, we could definitely look into covering it within KIP-1327
> > >
> > > Best,
> > > Chia-Ping
> > >
> > >
> > > Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
> > >
> > >> Hi Jiunn-Yang,
> > >> I've just been re-reading this KIP to ensure that share group support is
> > >> not needed. I've discovered that we made a mistake implementing KIP-932
> > in
> > >> this area, so we'll rectify that and check whether it's really tight
> > enough
> > >> not to require KIP-1327.
> > >>
> > >> One trivial comments for consistency.
> > >>
> > >> AS1: In kafka-consumer-groups.sh, unknown column data is represented by
> > >> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > >>> Hi all,
> > >>>
> > >>> Manually bumping this thread.
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > >>>>
> > >>>> Hello chia,
> > >>>>
> > >>>> Thanks for the feedback, I have updated the KIP.
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> > >>>>>
> > >>>>> hi Jiunn-Yang
> > >>>>>
> > >>>>>> When the config is set on a cluster that has not yet been
> > >> upgraded... classification cannot occur... the consumer falls back to
> > the
> > >> base auto.offset.reset for the affected partitions. No exception is
> > thrown,
> > >> and no operational disruption results.
> > >>>>>
> > >>>>> Existing group can't take advantage of this excellent new
> > >> configuration. Allowing users to modify the group creation time might be
> > >> overkill. Instead, we could print a useful warning message to guide
> > users.
> > >> For example, we can suggest that they re-create the group with their
> > >> existing committed offsets
> > >>>>>
> > >>>>>> Protocol changes
> > >>>>>
> > >>>>> Would you mind listing those RPC changes in a table format?
> > >>>>>
> > >>>>>> The full interaction matrix between the base policy and the
> > >> new-partition policy is:
> > >>>>>
> > >>>>> Please add a filed to describe the target scenario when using these
> > >> policies
> > >>>>>
> > >>>>> Best,
> > >>>>> Chia-Ping
> > >>>>>
> > >>>>>
> > >>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > >>>>>> Hello Jun, chia,
> > >>>>>>
> > >>>>>> Thanks for the feedback, I have updated the KIP for the new
> > >>>>>> approach, PTAL
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>> Jiunn-Yang
> > >>>>>>
> > >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> > >>>>>>>
> > >>>>>>> hi Jun
> > >>>>>>>
> > >>>>>>> Yes, your approach is great. I think the combination of latest (for
> > >> existing partitions) and by_duration (for new partitions) can address
> > 99%
> > >> of the complaints I have heard regarding this issue.
> > >>>>>>>
> > >>>>>>> Also, leveraging the group creation time here opens the door to
> > >> implementing a new policy based on timestamp seek in the future, should
> > the
> > >> community want to pursue that.
> > >>>>>>>
> > >>>>>>> Thanks for your patience and constructive feedback. We will update
> > >> the KIP accordingly.
> > >>>>>>>
> > >>>>>>> Best, Chia-Ping
> > >>>>>>>
> > >>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
> > >>>>>>>>
> > >>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>
> > >>>>>>>> Thanks for the reply.
> > >>>>>>>>
> > >>>>>>>> I agree that it's probably useful to allow a user to configure a
> > >> different
> > >>>>>>>> offset policy for existing partitions vs new partitions. However,
> > >> using
> > >>>>>>>> group creation time to capture that seems more intuitive. Here is
> > >> another
> > >>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new
> > >> partitions
> > >>>>>>>> based on group creation time. Introduce
> > >>>>>>>> a new config auto.offset.reset.new.partitions whose values can be
> > >> earliest,
> > >>>>>>>> latest and by_duration, the same as auto.offset.reset. Users can
> > >> set
> > >>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to
> > >> guarantee
> > >>>>>>>> no data loss on new partitions. They can also use by_duration to
> > >> set an
> > >>>>>>>> upper bound on the backlog replayed, which can be different from
> > >> that of
> > >>>>>>>> the existing partitions. This will address your concern about too
> > >> much
> > >>>>>>>> backlog being replayed when the offsets are lost. What do you
> > >> think?
> > >>>>>>>>
> > >>>>>>>> Jun
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> > >> chia7712@gmail.com> wrote:
> > >>>>>>>>>
> > >>>>>>>>> hi Jun
> > >>>>>>>>>
> > >>>>>>>>> The most important part of this story is how users should expect
> > >> the data
> > >>>>>>>>> they can see when using the latest or by_duration policy with
> > >> expanded
> > >>>>>>>>> partitions.
> > >>>>>>>>>
> > >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > >>>>>>>>> non-deterministic, which means users will either read too many
> > >> historical
> > >>>>>>>>> records from existing partitions or lose some records from
> > >> expanded
> > >>>>>>>>> partitions.
> > >>>>>>>>>
> > >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > >>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>
> > >>>>>>>>> is a bit hard to understand, and that is why I preferred having a
> > >> whole new
> > >>>>>>>>> policy based entirely on group creation time (KIP-1282)
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> Chia-Ping
> > >>>>>>>>>
> > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the reply. I am still trying to understand the value
> > >> of the new
> > >>>>>>>>>> configs with the KIP.
> > >>>>>>>>>>
> > >>>>>>>>>> The motivation of the KIP is that a user doesn't want to miss
> > >> the data if
> > >>>>>>>>>> the backlog is small. The backlog of the existing partition is
> > >> easy to
> > >>>>>>>>>> understand because it relates to retention time. The backlog for
> > >> the new
> > >>>>>>>>>> partition is a bit subtle to understand since it depends on the
> > >> metadata
> > >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>> ,
> > >>>>>>>>>> the user needs to
> > >>>>>>>>>> understand the metadata refresh delay on the consumer side and
> > >> use it to
> > >>>>>>>>>> set the config.
> > >>>>>>>>>>
> > >>>>>>>>>> Now, let's consider the alternative: setting the same value for
> > >> the
> > >>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> > this
> > >>>>>>>>>> approach.
> > >>>>>>>>>> 1. It computes the seek target client-side as now() - duration,
> > >> which
> > >>>>>>>>>> introduces clock skew across consumers and forces operators to
> > >> choose
> > >>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> > >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so failed
> > >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> > >> potentially
> > >>>>>>>>>> miss records produced between attempts.
> > >>>>>>>>>> 3. It applies uniformly to all partitions without committed
> > >> offsets, and
> > >>>>>>>>>> cannot distinguish newly expanded partitions from long-existing
> > >> partitions
> > >>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> > >>>>>>>>>>
> > >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
> > >> buffer to
> > >>>>>>>>>> the metadata refresh delay. We could also consider improving the
> > >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> > >> typically low
> > >>>>>>>>>> (in the order of minutes with the classic consumer and tens of
> > >> seconds
> > >>>>>>>>>> with
> > >>>>>>>>>> the new consumer). If a user is ok with reading that much
> > >> backlog for new
> > >>>>>>>>>> partitions, it seems they will be ok doing the same for existing
> > >>>>>>>>>> partitions.
> > >>>>>>>>>>
> > >>>>>>>>>> So, instead of introducing a new config, could we just reuse the
> > >> existing
> > >>>>>>>>>> config with better documentation and/or implementation?
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> > >> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>
> > >>>>>>>>>>> You're right that group creation time is the more intuitive
> > >> answer at
> > >>>>>>>>>>> first glance,
> > >>>>>>>>>>> the KIP's own motivation talks about partitions that "predate
> > >> the group"
> > >>>>>>>>>>> vs partitions
> > >>>>>>>>>>> "created during group runtime," which directly points to a
> > >>>>>>>>>> group-lifecycle
> > >>>>>>>>>>> classifier.
> > >>>>>>>>>>> I'd like to walk through why we landed on partition age, and
> > the
> > >>>>>>>>>>> trade-offs we considered.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We evaluated three candidate signals:
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. `by_duration:5secs`
> > >>>>>>>>>>>
> > >>>>>>>>>>> This covers the metadata blindness window, but has issues the
> > >> KIP
> > >>>>>>>>>>> currently documents
> > >>>>>>>>>>> under "Why not use `by_duration`?":
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew across
> > >> consumers.
> > >>>>>>>>>>> - `ListOffsets` retries shift the target forward, potentially
> > >> missing
> > >>>>>>>>>>> records produced between
> > >>>>>>>>>>> attempts.
> > >>>>>>>>>>> - It applies uniformly to all partitions without committed
> > >> offsets,
> > >>>>>>>>>>> including pre-existing partitions
> > >>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. Group creation time as classifier
> > >>>>>>>>>>>
> > >>>>>>>>>>> This works cleanly when the consumer is actively running. Our
> > >> concern
> > >>>>>>>>>>> is the idle / late-rejoin case:
> > >>>>>>>>>>>
> > >>>>>>>>>>> T=0:         Group created.
> > >>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > >>>>>>>>>>> T=50:        Partition added during the idle window.
> > >>>>>>>>>>> T=100:       Consumer resumes.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Under group creation time, the new partition is classified as
> > >> new
> > >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from
> > >> T=50.
> > >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> > >> data that
> > >>>>>>>>>>> the consumer accepts as lost — that is precisely the contract
> > of
> > >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
> > >> treat
> > >>>>>>>>>>> the new partition differently; both contain backlog accumulated
> > >> during
> > >>>>>>>>>>> the same idle window.
> > >>>>>>>>>>>
> > >>>>>>>>>>> This aligns with the "backlog is backlog” principle you raised
> > >> in
> > >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog
> > >> on
> > >>>>>>>>>>> every other partition during the same idle period; forcing
> > >> 0-backlog
> > >>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> > that
> > >>>>>>>>>>> tolerance.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 3. Partition age vs threshold
> > >>>>>>>>>>>
> > >>>>>>>>>>> Partition age corresponds to the actual silent data loss
> > window,
> > >>>>>>>>>>> the gap between partition creation and the consumer’s metadata
> > >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: the
> > >>>>>>>>>>> consumer had no opportunity to know about the partition.
> > >> Outside this
> > >>>>>>>>>>> window, missing data reflects either:
> > >>>>>>>>>>>
> > >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle consumers,
> > >> or
> > >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not via
> > >> reset
> > >>>>>>>>>> policy.
> > >>>>>>>>>>>
> > >>>>>>>>>>> We did not choose partition age because it is more elegant than
> > >> group
> > >>>>>>>>>>> creation time — we chose it because its failure mode (requires
> > a
> > >>>>>>>>>>> threshold) is
> > >>>>>>>>>>> less invasive than the failure mode of group creation time
> > >> (overrides
> > >>>>>>>>>>> user-stated
> > >>>>>>>>>>> `latest` intent during idle periods).
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best Regards,
> > >>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi Jun,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Relying on both creation times will create an inconsistent
> > >> scenario. A
> > >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will seek
> > >> to the
> > >>>>>>>>>>>> beginning for the partitions created later than the group.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> > >> inconsistency
> > >>>>>>>>>>> using a
> > >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
> > >> KIP-1327
> > >>>>>>>>>>> goes
> > >>>>>>>>>>>> back to using flexible configurations to prevent users from
> > >> falling
> > >>>>>>>>>> into
> > >>>>>>>>>>>> that pitfall.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> > >> 上午6:49寫道:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>
> > >>>>>>>>>> still feels weird to
> > >>>>>>>>>>> me.
> > >>>>>>>>>>>>> It
> > >>>>>>>>>>>>> categorizes partitions as new or existing based on the
> > >> partition
> > >>>>>>>>>>> creation
> > >>>>>>>>>>>>> time. Intuitively, the categorization should be based on the
> > >> group
> > >>>>>>>>>>> creation
> > >>>>>>>>>>>>> time: all partitions existing when the group is created are
> > >> existing
> > >>>>>>>>>> and
> > >>>>>>>>>>>>> all partitions created after the group creation are new
> > >> partitions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> > >> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> > >>>>>>>>>>>>>> discussion, I will close the vote.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hello Jian,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for your feedback,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational task,
> > >> not an
> > >>>>>>>>>> edge
> > >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify
> > >> that
> > >>>>>>>>>> this
> > >>>>>>>>>>>>> is a
> > >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
> > >>>>>>>>>> demonstrates
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> need for this optimization. For example:
> > >>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> > >> Kafka: To
> > >>>>>>>>>>>>> balance
> > >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> > >> created with a
> > >>>>>>>>>>>>>> moderate
> > >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
> > >> time, it
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>> often
> > >>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> > >> accommodate the
> > >>>>>>>>>>>>> higher
> > >>>>>>>>>>>>>>>> workload.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Regards
> > >>>>>>>>>>>>>>>> Jian
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29
> > >> 寫道:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> > >>>>>>>>>> describing
> > >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them
> > to
> > >>>>>>>>>> describe
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> > >> applying
> > >>>>>>>>>> these
> > >>>>>>>>>>>>>> terms
> > >>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> > >> whether a
> > >>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
> > >> they won't
> > >>>>>>>>>>>>>> silently
> > >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during their
> > >> active
> > >>>>>>>>>>>>>>>>> consumption.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> > >> motivation
> > >>>>>>>>>>> section.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
> > >> 凌晨1:12
> > >>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> > >> motivation
> > >>>>>>>>>> for
> > >>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> > >> partitions
> > >>>>>>>>>> without
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> > >> partition
> > >>>>>>>>>> (hot)
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer
> > >> has
> > >>>>>>>>>> never
> > >>>>>>>>>>>>> seen
> > >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> > >>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> > >>>>>>>>>>> insufficient
> > >>>>>>>>>>>>>>>>>>> because:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies
> > >> across
> > >>>>>>>>>> nodes
> > >>>>>>>>>>>>>> due
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> > >>>>>>>>>> duration,
> > >>>>>>>>>>>>>>>>>>> causing
> > >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek
> > >> time on
> > >>>>>>>>>>>>> retry,
> > >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data
> > >> loss."
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If these
> > >> issues
> > >>>>>>>>>>>>> persist,
> > >>>>>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> > >> situations don't
> > >>>>>>>>>>>>> need a
> > >>>>>>>>>>>>>>>>>>> common
> > >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare situations,
> > >> they can
> > >>>>>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>>>>> customized logic using
> > >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> > >> s7133700@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > >> should
> > >>>>>>>>>> always
> > >>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >>  any
> > >>>>>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
> > >>>>>>>>>> mismatch
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs
> > >> inside
> > >>>>>>>>>>>>> poll().
> > >>>>>>>>>>>>>>>>>>>>> Therefore, the
> > >>>>>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException
> > >> from
> > >>>>>>>>>> poll().
> > >>>>>>>>>>>>>> I’ll
> > >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
> > >> 下午4:50
> > >>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> > >>>>>>>>>> partition
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> > >>>>>>>>>>>>>>>>> broker_current_time
> > >>>>>>>>>>>>>>>>>>> -
> > >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> > >> does not
> > >>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> > >> should
> > >>>>>>>>>> always
> > >>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> > >>  any
> > >>>>>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
> > >> calling
> > >>>>>>>>>>>>>>>>> `poll()`?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on
> > >> the
> > >>>>>>>>>>>>>> discussion
> > >>>>>>>>>>>>>>>>>>>>>>> feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> > reset
> > >>>>>>>>>> behavior
> > >>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> > >>>
> > >>>>>>>>>> now applies to all
> > >>>>>>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets
> > to
> > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>
> > >>>>>>>>>> .partitions
> > >>>>>>>>>>>>>>>>>>>>>>> config setting
> > >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > >>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>> continues
> > >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented
> > >> by a
> > >>>>>>>>>>> separate
> > >>>>>>>>>>>>>>>>>>>>> internal config
> > >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>> .partitions,
> > >>>>>>>>>> currently fixed to
> > >>>>>>>>>>>>> earliest).
> > >>>>>>>>>>>>>>>>> This
> > >>>>>>>>>>>>>>>>>>>>> decoupled design makes
> > >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> > >> public
> > >>>>>>>>>>>>>> user-facing
> > >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
> > >> 清晨7:46
> > >>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is
> > >> listed
> > >>>>>>>>>>> below:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>> .partitions
> > >>>>>>>>>> with a default value
> > >>>>>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> > >> by_duration and
> > >>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>> does not change the logic of
> > >> auto.offset.reset=earliest.
> > >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>> .partitions
> > >>>>>>>>>> as an internal
> > >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>
> > >>>>>>>>>> .partitions=earliest
> > >>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases
> > >> of
> > >>>>>>>>>> other
> > >>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>> in a
> > >>>>>>>>>>>>>>>>>>>>> separate KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>> .partitions
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> > >>> ,
> > >>>>>>>>>> will be applied to all for
> > >>>>>>>>>>>>>>>>>>>>> consistency.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When
> > >> a Kafka
> > >>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> expanded
> > >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest
> > >> auto
> > >>>>>>>>>> offset
> > >>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> > >>>>>>>>>> partitions
> > >>>>>>>>>>>>>> before
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> > >> record loss
> > >>>>>>>>>>>>> issue
> > >>>>>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>> .partitions
> > >>>>>>>>>> will
> > >>>>>>>>>>> take
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set
> > >> it
> > >>>>>>>>>>>>>> by_duration if
> > >>>>>>>>>>>>>>>>>>>>> needed.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > >>>>>>>>>>>>>>>>> chia7712@apache.org
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> > >> something
> > >>>>>>>>>>>>>> important
> > >>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
> > >> clarify
> > >>>>>>>>>> with
> > >>>>>>>>>>>>> a
> > >>>>>>>>>>>>>> few
> > >>>>>>>>>>>>>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
> > >> logic
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest
> > >> or
> > >>>>>>>>>>>>> by_duration
> > >>>>>>>>>>>>>>>>>>> don't
> > >>>>>>>>>>>>>>>>>>>>> seem
> > >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue
> > >> when a
> > >>>>>>>>>>>>>> partition
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>> expanded.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> > >> configure for
> > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>> .partitions?
> > >>>>>>>>>> If they set it to
> > >>>>>>>>>>>>> earliest
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> > >>>>>>>>>> example,
> > >>>>>>>>>>>>> if a
> > >>>>>>>>>>>>>>>>>>>>> consumer is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> > >> created
> > >>>>>>>>>> during
> > >>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> downtime,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest
> > >> when
> > >>>>>>>>>>>>> resuming,
> > >>>>>>>>>>>>>>>>>>> rather
> > >>>>>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition
> > >> is
> > >>>>>>>>>>>>>> technically
> > >>>>>>>>>>>>>>>>>>>>> "new" to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
> > >> max.age
> > >>>>>>>>>>>>>> threshold.
> > >>>>>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>>>>> gives
> > >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> > >> genuinely
> > >>>>>>>>>>> "hot/new"
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> what is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
> > >> weird. It
> > >>>>>>>>>>>>> only
> > >>>>>>>>>>>>>>>>>>>>> applies when
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems
> > >> that the
> > >>>>>>>>>>>>>>>>> motivation
> > >>>>>>>>>>>>>>>>>>>>>>>>>> equally
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
> > >> values
> > >>>>>>>>>> like
> > >>>>>>>>>>>>>>>>>>>>> by_duration.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> > >> way to
> > >>>>>>>>>>> control
> > >>>>>>>>>>>>>>>>> newly
> > >>>>>>>>>>>>>>>>>>>>> created
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> > group
> > >>>>>>>>>> starts.
> > >>>>>>>>>>>>>> Have we
> > >>>>>>>>>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> > >>>
> > >>>>>>>>>>> .partitions?
> > >>>>>>>>>>>>>> If
> > >>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> > >> defaults to
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> > >> explicitly
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> customize
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> > >> s7133700@gmail.com
> > >>>>>>>>>>>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37
> > >> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > >>>>>>>>>>>>>> PartitionMetadata
> > >>>>>>>>>>>>>>>>>>>>> struct
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> > >> computed
> > >>>>>>>>>>>>>> server-side
> > >>>>>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> > partition_creation_time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > >>>>>>>>>>>>>> MembershipManager
> > >>>>>>>>>>>>>>>>>>>>> detects
> > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> > >> metadata for
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> affected
> > >>>>>>>>>>>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even
> > >> if the
> > >>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>> ID
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> > >> supported
> > >>>>>>>>>>>>>>>>>>>>> MetadataResponse
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time.
> > >> If the
> > >>>>>>>>>>>>>> negotiated
> > >>>>>>>>>>>>>>>>>>>>>>>>>> version is
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> > >> PartitionAgeMs at
> > >>>>>>>>>> all
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> > >> back to
> > >>>>>>>>>> latest
> > >>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>> risking
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> > >> 2026年4月29日
> > >>>>>>>>>>> 下午4:04
> > >>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> > >> 'age'
> > >>>>>>>>>>>>> resolution
> > >>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
> > >> control
> > >>>>>>>>>>>>> plane
> > >>>>>>>>>>>>>>>>>>> clean.
> > >>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>> main
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > >>>>>>>>>>> inter-broker
> > >>>>>>>>>>>>>>>>> clock
> > >>>>>>>>>>>>>>>>>>>>> skew.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
> > >> source of
> > >>>>>>>>>>>>> truth
> > >>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>> time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> > >> should be
> > >>>>>>>>>>>>>> negligible.
> > >>>>>>>>>>>>>>>>>>>>> Given
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> > >> configured in
> > >>>>>>>>>>> minutes
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>>>>> hours, a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> > >> brokers
> > >>>>>>>>>> won't
> > >>>>>>>>>>>>>> impact
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org>
> > >> 於
> > >>>>>>>>>>>>> 2026年4月29日
> > >>>>>>>>>>>>>>>>>>> 下午3:29
> > >>>>>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> > >> previous
> > >>>>>>>>>>>>>> conversation
> > >>>>>>>>>>>>>>>>>>> but I
> > >>>>>>>>>>>>>>>>>>>>>>>>>> took a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow
> > >> with the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
> > >> boolean;
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>>>>> states
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
> > >> boolean
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>> set.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> > >>>>>>>>>> partition,
> > >>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive
> > >> the
> > >>>>>>>>>> ages.
> > >>>>>>>>>>> Is
> > >>>>>>>>>>>>>> my
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow
> > >> and
> > >>>>>>>>>> also
> > >>>>>>>>>>>>>> explain
> > >>>>>>>>>>>>>>>>>>>>> how it
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets,
> > >> fetch
> > >>>>>>>>>>>>> offsets,
> > >>>>>>>>>>>>>>>>>>> etc.).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> > >> wonder if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
> > >> place
> > >>>>>>>>>> for
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>> given
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively,
> > >> it could
> > >>>>>>>>>>>>> simply
> > >>>>>>>>>>>>>>>>>>>>> include
> > >>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> > >> cautious
> > >>>>>>>>>>> about
> > >>>>>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> > >> unrelated
> > >>>>>>>>>>> concepts.
> > >>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>> API
> > >>>>>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > >>>>>>>>>>> partitions.
> > >>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>> fact
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
> > >> Streams
> > >>>>>>>>>> API
> > >>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>> suggests
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we
> > >> do if
> > >>>>>>>>>> we
> > >>>>>>>>>>>>>> want to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> > >> Muralidhar Basani
> > >>>>>>>>>>> via
> > >>>>>>>>>>>>>> dev
> > >>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
> > >> about
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> gap.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > >>>>>>>>>>>>>> RequestPartitionAges
> > >>>>>>>>>>>>>>>>>>>>> field.
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> > >> PartitionAges
> > >>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> > >> recommended
> > >>>>>>>>>>>>> value
> > >>>>>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > >>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> > >>>>>>>>>> <
> > >>
> > https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> > >>>
> > >>>>>>>>>>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> > >> guess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to
> > >> Kafka
> > >>>>>>>>>>>>> Streams,
> > >>>>>>>>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > >>>>>>>>>>>>> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it?
> > >> This is
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> > >> mistakenly
> > >>>>>>>>>>>>>> configure
> > >>>>>>>>>>>>>>>>>>>>> it. Or
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer
> > >> falls
> > >>>>>>>>>> back
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>> earliest"
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> > >> per-partition
> > >>>>>>>>>> which
> > >>>>>>>>>>>>>> isn't
> > >>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
> > >> resolves
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> initial
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> > >> earliest
> > >>>>>>>>>> was
> > >>>>>>>>>>>>>> applied
> > >>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> > >> config is
> > >>>>>>>>>>>>>> unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > >>>>>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> > >> change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > >>>>>>>>>> 2026年4月28日
> > >>>>>>>>>>>>>> 晚上8:03
> > >>>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
> > >> creation
> > >>>>>>>>>>> time
> > >>>>>>>>>>>>>> via
> > >>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users
> > to
> > >>>>>>>>>> diagnose
> > >>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> > >> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> > >> KIP-1327
> > >>>>>>>>>>>>>> Prevent
> > >>>>>>>>>>>>>>>>> Hot
> > >>>>>>>>>>>>>>>>>>>>> Data
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
> > >> expanded
> > >>>>>>>>>>>>> (hot)
> > >>>>>>>>>>>>>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > >>>>>>>>>>> automatically
> > >>>>>>>>>>>>>> fall
> > >>>>>>>>>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing
> > >> a full
> > >>>>>>>>>>>>>> historical
> > >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>
> >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Jiunn-Yang,

Thanks for the updated KIP.

JR2. Share consumer (KIP-932) states: "If the number of partitions is
increased for a topic with a subscription in a share group, the SPSO for
the newly created share-partitions is initialized to 0 ". However, this
part has not been implemented yet. Currently, all partitions without offset
follows the strategy given by share.auto.offset.reset. So, we have an
opportunity to see if it's better to align this KIP's changes with the
share consumer.
JR2.1 The first part is whether customizing the offset for new partitions
to something other than 0 is useful in the share consumer. It seems that
the motivation in this KIP applies to share consumer too.
JR2.2 The second part is how new partitions are detected. The mechanism
introduced in this KIP seems quite effective. A group coordinator could
pick up newly created partitions for an existing topic or newly created
topics matching the regex. Those partitions are categorized as new
partitions and will follow the strategy given by
auto.offset.reset.new.partitions.
This makes sense since those partitions typically don't have a large
backlog. A consumer could also additionally subscribe to an existing topic
in an existing group. Those partitions will be categorized as existing
partitions. This also makes sense since those partitions could have a large
backlog and it is better to follow the strategy given by auto.offset.reset. It
will be useful to think through whether the logic for categorizing new
partitions should also be reused in the share consumer.

JR3. The KIP exposes the group/partition creation time to the client. I am
not sure if there are other use cases for those timestamps. An alternative
is to avoid exposing those timestamps to the client. The group coordinator
will categorize the partitions using those timestamps and include the
categorization in the assigned partitions in GroupHeartbeatResponse.

JR4. It would be useful to support this feature on existing groups as well.
For example, any partition without a creation timestamp can always be
categorized as an existing partition.

Jun

On Wed, Jun 24, 2026 at 6:12 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Andrew,
>
> Thanks for your feedback,
> >> I've just been re-reading this KIP to ensure that share group support is
> >> not needed. I've discovered that we made a mistake implementing KIP-932
> in
> >> this area, so we'll rectify that and check whether it's really tight
> enough
> >> not to require KIP-1327.
>
> Could you walk us through the specific area where the mistake was found?
> That would
> help us assess whether it naturally fits within the scope of KIP-1327 or
> should be addressed
> separately.
>
> AS1 I have updated the KIP
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> >
> > hi Andrew
> >
> >> I've discovered that we made a mistake implementing KIP-932 in this
> area,
> > so we'll rectify that and check whether it's really tight enough not to
> > require KIP-1327.
> >
> > Would you mind sharing more details about that? If it's not anything
> close
> > to NP-hard, we could definitely look into covering it within KIP-1327
> >
> > Best,
> > Chia-Ping
> >
> >
> > Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
> >
> >> Hi Jiunn-Yang,
> >> I've just been re-reading this KIP to ensure that share group support is
> >> not needed. I've discovered that we made a mistake implementing KIP-932
> in
> >> this area, so we'll rectify that and check whether it's really tight
> enough
> >> not to require KIP-1327.
> >>
> >> One trivial comments for consistency.
> >>
> >> AS1: In kafka-consumer-groups.sh, unknown column data is represented by
> >> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
> >>
> >> Thanks,
> >> Andrew
> >>
> >> On 2026/06/23 23:23:38 黃竣陽 wrote:
> >>> Hi all,
> >>>
> >>> Manually bumping this thread.
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> >>>>
> >>>> Hello chia,
> >>>>
> >>>> Thanks for the feedback, I have updated the KIP.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> >>>>>
> >>>>> hi Jiunn-Yang
> >>>>>
> >>>>>> When the config is set on a cluster that has not yet been
> >> upgraded... classification cannot occur... the consumer falls back to
> the
> >> base auto.offset.reset for the affected partitions. No exception is
> thrown,
> >> and no operational disruption results.
> >>>>>
> >>>>> Existing group can't take advantage of this excellent new
> >> configuration. Allowing users to modify the group creation time might be
> >> overkill. Instead, we could print a useful warning message to guide
> users.
> >> For example, we can suggest that they re-create the group with their
> >> existing committed offsets
> >>>>>
> >>>>>> Protocol changes
> >>>>>
> >>>>> Would you mind listing those RPC changes in a table format?
> >>>>>
> >>>>>> The full interaction matrix between the base policy and the
> >> new-partition policy is:
> >>>>>
> >>>>> Please add a filed to describe the target scenario when using these
> >> policies
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>>
> >>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
> >>>>>> Hello Jun, chia,
> >>>>>>
> >>>>>> Thanks for the feedback, I have updated the KIP for the new
> >>>>>> approach, PTAL
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> >>>>>>>
> >>>>>>> hi Jun
> >>>>>>>
> >>>>>>> Yes, your approach is great. I think the combination of latest (for
> >> existing partitions) and by_duration (for new partitions) can address
> 99%
> >> of the complaints I have heard regarding this issue.
> >>>>>>>
> >>>>>>> Also, leveraging the group creation time here opens the door to
> >> implementing a new policy based on timestamp seek in the future, should
> the
> >> community want to pursue that.
> >>>>>>>
> >>>>>>> Thanks for your patience and constructive feedback. We will update
> >> the KIP accordingly.
> >>>>>>>
> >>>>>>> Best, Chia-Ping
> >>>>>>>
> >>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
> >>>>>>>>
> >>>>>>>> Hi, Chia-Ping,
> >>>>>>>>
> >>>>>>>> Thanks for the reply.
> >>>>>>>>
> >>>>>>>> I agree that it's probably useful to allow a user to configure a
> >> different
> >>>>>>>> offset policy for existing partitions vs new partitions. However,
> >> using
> >>>>>>>> group creation time to capture that seems more intuitive. Here is
> >> another
> >>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new
> >> partitions
> >>>>>>>> based on group creation time. Introduce
> >>>>>>>> a new config auto.offset.reset.new.partitions whose values can be
> >> earliest,
> >>>>>>>> latest and by_duration, the same as auto.offset.reset. Users can
> >> set
> >>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to
> >> guarantee
> >>>>>>>> no data loss on new partitions. They can also use by_duration to
> >> set an
> >>>>>>>> upper bound on the backlog replayed, which can be different from
> >> that of
> >>>>>>>> the existing partitions. This will address your concern about too
> >> much
> >>>>>>>> backlog being replayed when the offsets are lost. What do you
> >> think?
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> >> chia7712@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> hi Jun
> >>>>>>>>>
> >>>>>>>>> The most important part of this story is how users should expect
> >> the data
> >>>>>>>>> they can see when using the latest or by_duration policy with
> >> expanded
> >>>>>>>>> partitions.
> >>>>>>>>>
> >>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
> >>>>>>>>> non-deterministic, which means users will either read too many
> >> historical
> >>>>>>>>> records from existing partitions or lose some records from
> >> expanded
> >>>>>>>>> partitions.
> >>>>>>>>>
> >>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
> >>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >>>
> >>>>>>>>> is a bit hard to understand, and that is why I preferred having a
> >> whole new
> >>>>>>>>> policy based entirely on group creation time (KIP-1282)
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Chia-Ping
> >>>>>>>>>
> >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> >>>>>>>>>
> >>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the reply. I am still trying to understand the value
> >> of the new
> >>>>>>>>>> configs with the KIP.
> >>>>>>>>>>
> >>>>>>>>>> The motivation of the KIP is that a user doesn't want to miss
> >> the data if
> >>>>>>>>>> the backlog is small. The backlog of the existing partition is
> >> easy to
> >>>>>>>>>> understand because it relates to retention time. The backlog for
> >> the new
> >>>>>>>>>> partition is a bit subtle to understand since it depends on the
> >> metadata
> >>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >>> ,
> >>>>>>>>>> the user needs to
> >>>>>>>>>> understand the metadata refresh delay on the consumer side and
> >> use it to
> >>>>>>>>>> set the config.
> >>>>>>>>>>
> >>>>>>>>>> Now, let's consider the alternative: setting the same value for
> >> the
> >>>>>>>>>> existing by_duration policy. The KIP lists three issues with
> this
> >>>>>>>>>> approach.
> >>>>>>>>>> 1. It computes the seek target client-side as now() - duration,
> >> which
> >>>>>>>>>> introduces clock skew across consumers and forces operators to
> >> choose
> >>>>>>>>>> overly large durations, causing unnecessary reprocessing.
> >>>>>>>>>> 2. The target timestamp is recomputed on each retry, so failed
> >>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
> >> potentially
> >>>>>>>>>> miss records produced between attempts.
> >>>>>>>>>> 3. It applies uniformly to all partitions without committed
> >> offsets, and
> >>>>>>>>>> cannot distinguish newly expanded partitions from long-existing
> >> partitions
> >>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
> >>>>>>>>>>
> >>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
> >> buffer to
> >>>>>>>>>> the metadata refresh delay. We could also consider improving the
> >>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
> >> typically low
> >>>>>>>>>> (in the order of minutes with the classic consumer and tens of
> >> seconds
> >>>>>>>>>> with
> >>>>>>>>>> the new consumer). If a user is ok with reading that much
> >> backlog for new
> >>>>>>>>>> partitions, it seems they will be ok doing the same for existing
> >>>>>>>>>> partitions.
> >>>>>>>>>>
> >>>>>>>>>> So, instead of introducing a new config, could we just reuse the
> >> existing
> >>>>>>>>>> config with better documentation and/or implementation?
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> >> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>
> >>>>>>>>>>> You're right that group creation time is the more intuitive
> >> answer at
> >>>>>>>>>>> first glance,
> >>>>>>>>>>> the KIP's own motivation talks about partitions that "predate
> >> the group"
> >>>>>>>>>>> vs partitions
> >>>>>>>>>>> "created during group runtime," which directly points to a
> >>>>>>>>>> group-lifecycle
> >>>>>>>>>>> classifier.
> >>>>>>>>>>> I'd like to walk through why we landed on partition age, and
> the
> >>>>>>>>>>> trade-offs we considered.
> >>>>>>>>>>>
> >>>>>>>>>>> We evaluated three candidate signals:
> >>>>>>>>>>>
> >>>>>>>>>>> 1. `by_duration:5secs`
> >>>>>>>>>>>
> >>>>>>>>>>> This covers the metadata blindness window, but has issues the
> >> KIP
> >>>>>>>>>>> currently documents
> >>>>>>>>>>> under "Why not use `by_duration`?":
> >>>>>>>>>>>
> >>>>>>>>>>> - Client-side `now() - duration` introduces clock skew across
> >> consumers.
> >>>>>>>>>>> - `ListOffsets` retries shift the target forward, potentially
> >> missing
> >>>>>>>>>>> records produced between
> >>>>>>>>>>> attempts.
> >>>>>>>>>>> - It applies uniformly to all partitions without committed
> >> offsets,
> >>>>>>>>>>> including pre-existing partitions
> >>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
> >>>>>>>>>>>
> >>>>>>>>>>> 2. Group creation time as classifier
> >>>>>>>>>>>
> >>>>>>>>>>> This works cleanly when the consumer is actively running. Our
> >> concern
> >>>>>>>>>>> is the idle / late-rejoin case:
> >>>>>>>>>>>
> >>>>>>>>>>> T=0:         Group created.
> >>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> >>>>>>>>>>> T=50:        Partition added during the idle window.
> >>>>>>>>>>> T=100:       Consumer resumes.
> >>>>>>>>>>>
> >>>>>>>>>>> Under group creation time, the new partition is classified as
> >> new
> >>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from
> >> T=50.
> >>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> >> data that
> >>>>>>>>>>> the consumer accepts as lost — that is precisely the contract
> of
> >>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
> >> treat
> >>>>>>>>>>> the new partition differently; both contain backlog accumulated
> >> during
> >>>>>>>>>>> the same idle window.
> >>>>>>>>>>>
> >>>>>>>>>>> This aligns with the "backlog is backlog” principle you raised
> >> in
> >>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog
> >> on
> >>>>>>>>>>> every other partition during the same idle period; forcing
> >> 0-backlog
> >>>>>>>>>>> tolerance only on new partitions would be inconsistent with
> that
> >>>>>>>>>>> tolerance.
> >>>>>>>>>>>
> >>>>>>>>>>> 3. Partition age vs threshold
> >>>>>>>>>>>
> >>>>>>>>>>> Partition age corresponds to the actual silent data loss
> window,
> >>>>>>>>>>> the gap between partition creation and the consumer’s metadata
> >>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: the
> >>>>>>>>>>> consumer had no opportunity to know about the partition.
> >> Outside this
> >>>>>>>>>>> window, missing data reflects either:
> >>>>>>>>>>>
> >>>>>>>>>>> - (a) the user’s tolerated cost of running with idle consumers,
> >> or
> >>>>>>>>>>> - (b) an operational issue to surface via monitoring, not via
> >> reset
> >>>>>>>>>> policy.
> >>>>>>>>>>>
> >>>>>>>>>>> We did not choose partition age because it is more elegant than
> >> group
> >>>>>>>>>>> creation time — we chose it because its failure mode (requires
> a
> >>>>>>>>>>> threshold) is
> >>>>>>>>>>> less invasive than the failure mode of group creation time
> >> (overrides
> >>>>>>>>>>> user-stated
> >>>>>>>>>>> `latest` intent during idle periods).
> >>>>>>>>>>>
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>
> >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Jun,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Relying on both creation times will create an inconsistent
> >> scenario. A
> >>>>>>>>>>>> consumer that lost all offsets due to a long sleep will seek
> >> to the
> >>>>>>>>>>>> beginning for the partitions created later than the group.
> >>>>>>>>>>>>
> >>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> >> inconsistency
> >>>>>>>>>>> using a
> >>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
> >> KIP-1327
> >>>>>>>>>>> goes
> >>>>>>>>>>>> back to using flexible configurations to prevent users from
> >> falling
> >>>>>>>>>> into
> >>>>>>>>>>>> that pitfall.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> >> 上午6:49寫道:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >>>
> >>>>>>>>>> still feels weird to
> >>>>>>>>>>> me.
> >>>>>>>>>>>>> It
> >>>>>>>>>>>>> categorizes partitions as new or existing based on the
> >> partition
> >>>>>>>>>>> creation
> >>>>>>>>>>>>> time. Intuitively, the categorization should be based on the
> >> group
> >>>>>>>>>>> creation
> >>>>>>>>>>>>> time: all partitions existing when the group is created are
> >> existing
> >>>>>>>>>> and
> >>>>>>>>>>>>> all partitions created after the group creation are new
> >> partitions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> >> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Manually bumping this thread. If there is no further
> >>>>>>>>>>>>>> discussion, I will close the vote.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello Jian,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for your feedback,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Agreed, partition expansion is a common operational task,
> >> not an
> >>>>>>>>>> edge
> >>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Jiunn-Yang:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify
> >> that
> >>>>>>>>>> this
> >>>>>>>>>>>>> is a
> >>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
> >>>>>>>>>> demonstrates
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> need for this optimization. For example:
> >>>>>>>>>>>>>>>> A partition expansion is a common operational task in
> >> Kafka: To
> >>>>>>>>>>>>> balance
> >>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
> >> created with a
> >>>>>>>>>>>>>> moderate
> >>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
> >> time, it
> >>>>>>>>>> is
> >>>>>>>>>>>>>> often
> >>>>>>>>>>>>>>>> necessary to increase the number of partitions to
> >> accommodate the
> >>>>>>>>>>>>> higher
> >>>>>>>>>>>>>>>> workload.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Regards
> >>>>>>>>>>>>>>>> Jian
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29
> >> 寫道:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> >>>>>>>>>> describing
> >>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them
> to
> >>>>>>>>>> describe
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> >> applying
> >>>>>>>>>> these
> >>>>>>>>>>>>>> terms
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> >> whether a
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
> >> they won't
> >>>>>>>>>>>>>> silently
> >>>>>>>>>>>>>>>>>> lose any live records produced to the topic during their
> >> active
> >>>>>>>>>>>>>>>>> consumption.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> >> motivation
> >>>>>>>>>>> section.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
> >> 凌晨1:12
> >>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> >> motivation
> >>>>>>>>>> for
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> >> partitions
> >>>>>>>>>> without
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> >> partition
> >>>>>>>>>> (hot)
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer
> >> has
> >>>>>>>>>> never
> >>>>>>>>>>>>> seen
> >>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> >>>>>>>>>> partition
> >>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> >>>>>>>>>>> insufficient
> >>>>>>>>>>>>>>>>>>> because:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies
> >> across
> >>>>>>>>>> nodes
> >>>>>>>>>>>>>> due
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> >>>>>>>>>> duration,
> >>>>>>>>>>>>>>>>>>> causing
> >>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
> >>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek
> >> time on
> >>>>>>>>>>>>> retry,
> >>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data
> >> loss."
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If these
> >> issues
> >>>>>>>>>>>>> persist,
> >>>>>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> >> situations don't
> >>>>>>>>>>>>> need a
> >>>>>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>>>>> solution. If users care about those rare situations,
> >> they can
> >>>>>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>>>>> customized logic using
> >>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> >> s7133700@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> >> should
> >>>>>>>>>> always
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> >>  any
> >>>>>>>>>>>>>> field
> >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
> >>>>>>>>>> mismatch
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs
> >> inside
> >>>>>>>>>>>>> poll().
> >>>>>>>>>>>>>>>>>>>>> Therefore, the
> >>>>>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException
> >> from
> >>>>>>>>>> poll().
> >>>>>>>>>>>>>> I’ll
> >>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
> >> 下午4:50
> >>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> >>>>>>>>>> partition
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> >>>>>>>>>>>>>>>>> broker_current_time
> >>>>>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> >> does not
> >>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> >> should
> >>>>>>>>>> always
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
> >>  any
> >>>>>>>>>>>>>> field
> >>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
> >> calling
> >>>>>>>>>>>>>>>>> `poll()`?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on
> >> the
> >>>>>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>>>>>>>>> feedback.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition
> reset
> >>>>>>>>>> behavior
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >>>
> >>>>>>>>>> now applies to all
> >>>>>>>>>>>>>> auto.offset.reset
> >>>>>>>>>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> >>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets
> to
> >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>
> >>>>>>>>>> .partitions
> >>>>>>>>>>>>>>>>>>>>>>> config setting
> >>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> >>>>>>>>>> auto.offset.reset
> >>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>> continues
> >>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
> >>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented
> >> by a
> >>>>>>>>>>> separate
> >>>>>>>>>>>>>>>>>>>>> internal config
> >>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>> .partitions,
> >>>>>>>>>> currently fixed to
> >>>>>>>>>>>>> earliest).
> >>>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>>>>> decoupled design makes
> >>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> >> public
> >>>>>>>>>>>>>> user-facing
> >>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
> >> 清晨7:46
> >>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is
> >> listed
> >>>>>>>>>>> below:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>> .partitions
> >>>>>>>>>> with a default value
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> >> by_duration and
> >>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> does not change the logic of
> >> auto.offset.reset=earliest.
> >>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>> .partitions
> >>>>>>>>>> as an internal
> >>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>
> >>>>>>>>>> .partitions=earliest
> >>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases
> >> of
> >>>>>>>>>> other
> >>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>>> in a
> >>>>>>>>>>>>>>>>>>>>> separate KIP.
> >>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>> .partitions
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> >>> ,
> >>>>>>>>>> will be applied to all for
> >>>>>>>>>>>>>>>>>>>>> consistency.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When
> >> a Kafka
> >>>>>>>>>>>>> topic
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> expanded
> >>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest
> >> auto
> >>>>>>>>>> offset
> >>>>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> >>>>>>>>>> partitions
> >>>>>>>>>>>>>> before
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> >> record loss
> >>>>>>>>>>>>> issue
> >>>>>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>> .partitions
> >>>>>>>>>> will
> >>>>>>>>>>> take
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set
> >> it
> >>>>>>>>>>>>>> by_duration if
> >>>>>>>>>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> >>>>>>>>>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> >> something
> >>>>>>>>>>>>>> important
> >>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
> >> clarify
> >>>>>>>>>> with
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>> few
> >>>>>>>>>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
> >> logic
> >>>>>>>>>> to
> >>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest
> >> or
> >>>>>>>>>>>>> by_duration
> >>>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue
> >> when a
> >>>>>>>>>>>>>> partition
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>> expanded.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> >> configure for
> >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>> .partitions?
> >>>>>>>>>> If they set it to
> >>>>>>>>>>>>> earliest
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> >>>>>>>>>> example,
> >>>>>>>>>>>>> if a
> >>>>>>>>>>>>>>>>>>>>> consumer is
> >>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> >> created
> >>>>>>>>>> during
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> downtime,
> >>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest
> >> when
> >>>>>>>>>>>>> resuming,
> >>>>>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition
> >> is
> >>>>>>>>>>>>>> technically
> >>>>>>>>>>>>>>>>>>>>> "new" to
> >>>>>>>>>>>>>>>>>>>>>>>>>> the group.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
> >> max.age
> >>>>>>>>>>>>>> threshold.
> >>>>>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>>>>> gives
> >>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> >> genuinely
> >>>>>>>>>>> "hot/new"
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> what is
> >>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
> >> weird. It
> >>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>> applies when
> >>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems
> >> that the
> >>>>>>>>>>>>>>>>> motivation
> >>>>>>>>>>>>>>>>>>>>>>>>>> equally
> >>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
> >> values
> >>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>> by_duration.
> >>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> >> way to
> >>>>>>>>>>> control
> >>>>>>>>>>>>>>>>> newly
> >>>>>>>>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the
> group
> >>>>>>>>>> starts.
> >>>>>>>>>>>>>> Have we
> >>>>>>>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >>>
> >>>>>>>>>>> .partitions?
> >>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> >> defaults to
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> >> explicitly
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>> customize
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> >> s7133700@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37
> >> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> >>>>>>>>>>>>>> PartitionMetadata
> >>>>>>>>>>>>>>>>>>>>> struct
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> >> computed
> >>>>>>>>>>>>>> server-side
> >>>>>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time -
> partition_creation_time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> >>>>>>>>>>>>>> MembershipManager
> >>>>>>>>>>>>>>>>>>>>> detects
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> >> metadata for
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even
> >> if the
> >>>>>>>>>>>>> topic
> >>>>>>>>>>>>>> ID
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> >> supported
> >>>>>>>>>>>>>>>>>>>>> MetadataResponse
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time.
> >> If the
> >>>>>>>>>>>>>> negotiated
> >>>>>>>>>>>>>>>>>>>>>>>>>> version is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> >> PartitionAgeMs at
> >>>>>>>>>> all
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>>>>> throw an
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> >> back to
> >>>>>>>>>> latest
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> risking
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> >> 2026年4月29日
> >>>>>>>>>>> 下午4:04
> >>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> >> 'age'
> >>>>>>>>>>>>> resolution
> >>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
> >> control
> >>>>>>>>>>>>> plane
> >>>>>>>>>>>>>>>>>>> clean.
> >>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> >>>>>>>>>>> inter-broker
> >>>>>>>>>>>>>>>>> clock
> >>>>>>>>>>>>>>>>>>>>> skew.
> >>>>>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
> >> source of
> >>>>>>>>>>>>> truth
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> >> should be
> >>>>>>>>>>>>>> negligible.
> >>>>>>>>>>>>>>>>>>>>> Given
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> >> configured in
> >>>>>>>>>>> minutes
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> hours, a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> >> brokers
> >>>>>>>>>> won't
> >>>>>>>>>>>>>> impact
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org>
> >> 於
> >>>>>>>>>>>>> 2026年4月29日
> >>>>>>>>>>>>>>>>>>> 下午3:29
> >>>>>>>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> >> previous
> >>>>>>>>>>>>>> conversation
> >>>>>>>>>>>>>>>>>>> but I
> >>>>>>>>>>>>>>>>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow
> >> with the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
> >> boolean;
> >>>>>>>>>> the
> >>>>>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>>>>> states
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
> >> boolean
> >>>>>>>>>> is
> >>>>>>>>>>>>>> set.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> >>>>>>>>>> partition,
> >>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>>>>> issue a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive
> >> the
> >>>>>>>>>> ages.
> >>>>>>>>>>> Is
> >>>>>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow
> >> and
> >>>>>>>>>> also
> >>>>>>>>>>>>>> explain
> >>>>>>>>>>>>>>>>>>>>> how it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets,
> >> fetch
> >>>>>>>>>>>>> offsets,
> >>>>>>>>>>>>>>>>>>> etc.).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> >> wonder if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
> >> place
> >>>>>>>>>> for
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>> given
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively,
> >> it could
> >>>>>>>>>>>>> simply
> >>>>>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> >> cautious
> >>>>>>>>>>> about
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>> overloading
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> >> unrelated
> >>>>>>>>>>> concepts.
> >>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>> API
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> >>>>>>>>>>> partitions.
> >>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>> fact
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
> >> Streams
> >>>>>>>>>> API
> >>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>> suggests
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we
> >> do if
> >>>>>>>>>> we
> >>>>>>>>>>>>>> want to
> >>>>>>>>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> >> Muralidhar Basani
> >>>>>>>>>>> via
> >>>>>>>>>>>>>> dev
> >>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
> >> about
> >>>>>>>>>> the
> >>>>>>>>>>>>>> gap.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> >>>>>>>>>>>>>> RequestPartitionAges
> >>>>>>>>>>>>>>>>>>>>> field.
> >>>>>>>>>>>>>>>>>>>>>>>>>> Can a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> >> PartitionAges
> >>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> >> recommended
> >>>>>>>>>>>>> value
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> >>>>>>>>>>>>>>>>>>> metadata.max.age.ms
> >>>>>>>>>> <
> >>
> https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> >>>
> >>>>>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> >> guess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to
> >> Kafka
> >>>>>>>>>>>>> Streams,
> >>>>>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> >>>>>>>>>>>>> auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it?
> >> This is
> >>>>>>>>>> the
> >>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> >> mistakenly
> >>>>>>>>>>>>>> configure
> >>>>>>>>>>>>>>>>>>>>> it. Or
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer
> >> falls
> >>>>>>>>>> back
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> earliest"
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> >> per-partition
> >>>>>>>>>> which
> >>>>>>>>>>>>>> isn't
> >>>>>>>>>>>>>>>>>>>>>>>>>> supported.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> May
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
> >> resolves
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> initial
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> >> earliest
> >>>>>>>>>> was
> >>>>>>>>>>>>>> applied
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> >> config is
> >>>>>>>>>>>>>> unchanged.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> >>>>>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> >> change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> >>>>>>>>>> 2026年4月28日
> >>>>>>>>>>>>>> 晚上8:03
> >>>>>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
> >> creation
> >>>>>>>>>>> time
> >>>>>>>>>>>>>> via
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> Admin
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users
> to
> >>>>>>>>>> diagnose
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> >> auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> >> KIP-1327
> >>>>>>>>>>>>>> Prevent
> >>>>>>>>>>>>>>>>> Hot
> >>>>>>>>>>>>>>>>>>>>> Data
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
> >> expanded
> >>>>>>>>>>>>> (hot)
> >>>>>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> >>>>>>>>>>> automatically
> >>>>>>>>>>>>>> fall
> >>>>>>>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing
> >> a full
> >>>>>>>>>>>>>> historical
> >>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello Andrew,

Thanks for your feedback,
>> I've just been re-reading this KIP to ensure that share group support is
>> not needed. I've discovered that we made a mistake implementing KIP-932 in
>> this area, so we'll rectify that and check whether it's really tight enough
>> not to require KIP-1327.

Could you walk us through the specific area where the mistake was found? That would 
help us assess whether it naturally fits within the scope of KIP-1327 or should be addressed 
separately.

AS1 I have updated the KIP

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月24日 晚上8:49 寫道:
> 
> hi Andrew
> 
>> I've discovered that we made a mistake implementing KIP-932 in this area,
> so we'll rectify that and check whether it's really tight enough not to
> require KIP-1327.
> 
> Would you mind sharing more details about that? If it's not anything close
> to NP-hard, we could definitely look into covering it within KIP-1327
> 
> Best,
> Chia-Ping
> 
> 
> Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:
> 
>> Hi Jiunn-Yang,
>> I've just been re-reading this KIP to ensure that share group support is
>> not needed. I've discovered that we made a mistake implementing KIP-932 in
>> this area, so we'll rectify that and check whether it's really tight enough
>> not to require KIP-1327.
>> 
>> One trivial comments for consistency.
>> 
>> AS1: In kafka-consumer-groups.sh, unknown column data is represented by
>> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
>> 
>> Thanks,
>> Andrew
>> 
>> On 2026/06/23 23:23:38 黃竣陽 wrote:
>>> Hi all,
>>> 
>>> Manually bumping this thread.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
>>>> 
>>>> Hello chia,
>>>> 
>>>> Thanks for the feedback, I have updated the KIP.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
>>>>> 
>>>>> hi Jiunn-Yang
>>>>> 
>>>>>> When the config is set on a cluster that has not yet been
>> upgraded... classification cannot occur... the consumer falls back to the
>> base auto.offset.reset for the affected partitions. No exception is thrown,
>> and no operational disruption results.
>>>>> 
>>>>> Existing group can't take advantage of this excellent new
>> configuration. Allowing users to modify the group creation time might be
>> overkill. Instead, we could print a useful warning message to guide users.
>> For example, we can suggest that they re-create the group with their
>> existing committed offsets
>>>>> 
>>>>>> Protocol changes
>>>>> 
>>>>> Would you mind listing those RPC changes in a table format?
>>>>> 
>>>>>> The full interaction matrix between the base policy and the
>> new-partition policy is:
>>>>> 
>>>>> Please add a filed to describe the target scenario when using these
>> policies
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> 
>>>>> On 2026/06/16 16:14:49 黃竣陽 wrote:
>>>>>> Hello Jun, chia,
>>>>>> 
>>>>>> Thanks for the feedback, I have updated the KIP for the new
>>>>>> approach, PTAL
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
>>>>>>> 
>>>>>>> hi Jun
>>>>>>> 
>>>>>>> Yes, your approach is great. I think the combination of latest (for
>> existing partitions) and by_duration (for new partitions) can address 99%
>> of the complaints I have heard regarding this issue.
>>>>>>> 
>>>>>>> Also, leveraging the group creation time here opens the door to
>> implementing a new policy based on timestamp seek in the future, should the
>> community want to pursue that.
>>>>>>> 
>>>>>>> Thanks for your patience and constructive feedback. We will update
>> the KIP accordingly.
>>>>>>> 
>>>>>>> Best, Chia-Ping
>>>>>>> 
>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
>>>>>>>> 
>>>>>>>> Hi, Chia-Ping,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> I agree that it's probably useful to allow a user to configure a
>> different
>>>>>>>> offset policy for existing partitions vs new partitions. However,
>> using
>>>>>>>> group creation time to capture that seems more intuitive. Here is
>> another
>>>>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new
>> partitions
>>>>>>>> based on group creation time. Introduce
>>>>>>>> a new config auto.offset.reset.new.partitions whose values can be
>> earliest,
>>>>>>>> latest and by_duration, the same as auto.offset.reset. Users can
>> set
>>>>>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to
>> guarantee
>>>>>>>> no data loss on new partitions. They can also use by_duration to
>> set an
>>>>>>>> upper bound on the backlog replayed, which can be different from
>> that of
>>>>>>>> the existing partitions. This will address your concern about too
>> much
>>>>>>>> backlog being replayed when the offsets are lost. What do you
>> think?
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
>> chia7712@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> hi Jun
>>>>>>>>> 
>>>>>>>>> The most important part of this story is how users should expect
>> the data
>>>>>>>>> they can see when using the latest or by_duration policy with
>> expanded
>>>>>>>>> partitions.
>>>>>>>>> 
>>>>>>>>> Yes, the by_duration policy can minimize data loss, but it is
>>>>>>>>> non-deterministic, which means users will either read too many
>> historical
>>>>>>>>> records from existing partitions or lose some records from
>> expanded
>>>>>>>>> partitions.
>>>>>>>>> 
>>>>>>>>> Also, I agree that auto.offset.reset.max.age.ms
>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
>>> 
>>>>>>>>> is a bit hard to understand, and that is why I preferred having a
>> whole new
>>>>>>>>> policy based entirely on group creation time (KIP-1282)
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>>>>>>>>> 
>>>>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply. I am still trying to understand the value
>> of the new
>>>>>>>>>> configs with the KIP.
>>>>>>>>>> 
>>>>>>>>>> The motivation of the KIP is that a user doesn't want to miss
>> the data if
>>>>>>>>>> the backlog is small. The backlog of the existing partition is
>> easy to
>>>>>>>>>> understand because it relates to retention time. The backlog for
>> the new
>>>>>>>>>> partition is a bit subtle to understand since it depends on the
>> metadata
>>>>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
>>> ,
>>>>>>>>>> the user needs to
>>>>>>>>>> understand the metadata refresh delay on the consumer side and
>> use it to
>>>>>>>>>> set the config.
>>>>>>>>>> 
>>>>>>>>>> Now, let's consider the alternative: setting the same value for
>> the
>>>>>>>>>> existing by_duration policy. The KIP lists three issues with this
>>>>>>>>>> approach.
>>>>>>>>>> 1. It computes the seek target client-side as now() - duration,
>> which
>>>>>>>>>> introduces clock skew across consumers and forces operators to
>> choose
>>>>>>>>>> overly large durations, causing unnecessary reprocessing.
>>>>>>>>>> 2. The target timestamp is recomputed on each retry, so failed
>>>>>>>>>> ListOffsetsRequest retries can shift the target forward and
>> potentially
>>>>>>>>>> miss records produced between attempts.
>>>>>>>>>> 3. It applies uniformly to all partitions without committed
>> offsets, and
>>>>>>>>>> cannot distinguish newly expanded partitions from long-existing
>> partitions
>>>>>>>>>> newly assigned to the group, leading to unnecessary replay.
>>>>>>>>>> 
>>>>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
>> buffer to
>>>>>>>>>> the metadata refresh delay. We could also consider improving the
>>>>>>>>>> implementation. For issue 3, the metadata refresh delay is
>> typically low
>>>>>>>>>> (in the order of minutes with the classic consumer and tens of
>> seconds
>>>>>>>>>> with
>>>>>>>>>> the new consumer). If a user is ok with reading that much
>> backlog for new
>>>>>>>>>> partitions, it seems they will be ok doing the same for existing
>>>>>>>>>> partitions.
>>>>>>>>>> 
>>>>>>>>>> So, instead of introducing a new config, could we just reuse the
>> existing
>>>>>>>>>> config with better documentation and/or implementation?
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello Jun,
>>>>>>>>>>> 
>>>>>>>>>>> You're right that group creation time is the more intuitive
>> answer at
>>>>>>>>>>> first glance,
>>>>>>>>>>> the KIP's own motivation talks about partitions that "predate
>> the group"
>>>>>>>>>>> vs partitions
>>>>>>>>>>> "created during group runtime," which directly points to a
>>>>>>>>>> group-lifecycle
>>>>>>>>>>> classifier.
>>>>>>>>>>> I'd like to walk through why we landed on partition age, and the
>>>>>>>>>>> trade-offs we considered.
>>>>>>>>>>> 
>>>>>>>>>>> We evaluated three candidate signals:
>>>>>>>>>>> 
>>>>>>>>>>> 1. `by_duration:5secs`
>>>>>>>>>>> 
>>>>>>>>>>> This covers the metadata blindness window, but has issues the
>> KIP
>>>>>>>>>>> currently documents
>>>>>>>>>>> under "Why not use `by_duration`?":
>>>>>>>>>>> 
>>>>>>>>>>> - Client-side `now() - duration` introduces clock skew across
>> consumers.
>>>>>>>>>>> - `ListOffsets` retries shift the target forward, potentially
>> missing
>>>>>>>>>>> records produced between
>>>>>>>>>>> attempts.
>>>>>>>>>>> - It applies uniformly to all partitions without committed
>> offsets,
>>>>>>>>>>> including pre-existing partitions
>>>>>>>>>>> newly assigned to the group, causing unnecessary replay.
>>>>>>>>>>> 
>>>>>>>>>>> 2. Group creation time as classifier
>>>>>>>>>>> 
>>>>>>>>>>> This works cleanly when the consumer is actively running. Our
>> concern
>>>>>>>>>>> is the idle / late-rejoin case:
>>>>>>>>>>> 
>>>>>>>>>>> T=0:         Group created.
>>>>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>>>>>>>>> T=50:        Partition added during the idle window.
>>>>>>>>>>> T=100:       Consumer resumes.
>>>>>>>>>>> 
>>>>>>>>>>> Under group creation time, the new partition is classified as
>> new
>>>>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from
>> T=50.
>>>>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
>> data that
>>>>>>>>>>> the consumer accepts as lost — that is precisely the contract of
>>>>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
>> treat
>>>>>>>>>>> the new partition differently; both contain backlog accumulated
>> during
>>>>>>>>>>> the same idle window.
>>>>>>>>>>> 
>>>>>>>>>>> This aligns with the "backlog is backlog” principle you raised
>> in
>>>>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog
>> on
>>>>>>>>>>> every other partition during the same idle period; forcing
>> 0-backlog
>>>>>>>>>>> tolerance only on new partitions would be inconsistent with that
>>>>>>>>>>> tolerance.
>>>>>>>>>>> 
>>>>>>>>>>> 3. Partition age vs threshold
>>>>>>>>>>> 
>>>>>>>>>>> Partition age corresponds to the actual silent data loss window,
>>>>>>>>>>> the gap between partition creation and the consumer’s metadata
>>>>>>>>>>> refresh. Within this window, data loss is genuinely silent: the
>>>>>>>>>>> consumer had no opportunity to know about the partition.
>> Outside this
>>>>>>>>>>> window, missing data reflects either:
>>>>>>>>>>> 
>>>>>>>>>>> - (a) the user’s tolerated cost of running with idle consumers,
>> or
>>>>>>>>>>> - (b) an operational issue to surface via monitoring, not via
>> reset
>>>>>>>>>> policy.
>>>>>>>>>>> 
>>>>>>>>>>> We did not choose partition age because it is more elegant than
>> group
>>>>>>>>>>> creation time — we chose it because its failure mode (requires a
>>>>>>>>>>> threshold) is
>>>>>>>>>>> less invasive than the failure mode of group creation time
>> (overrides
>>>>>>>>>>> user-stated
>>>>>>>>>>> `latest` intent during idle periods).
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Jun,
>>>>>>>>>>>> 
>>>>>>>>>>>> Relying on both creation times will create an inconsistent
>> scenario. A
>>>>>>>>>>>> consumer that lost all offsets due to a long sleep will seek
>> to the
>>>>>>>>>>>> beginning for the partitions created later than the group.
>>>>>>>>>>>> 
>>>>>>>>>>>> That is why we initially proposed KIP-1282 to fix the
>> inconsistency
>>>>>>>>>>> using a
>>>>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
>> KIP-1327
>>>>>>>>>>> goes
>>>>>>>>>>>> back to using flexible configurations to prevent users from
>> falling
>>>>>>>>>> into
>>>>>>>>>>>> that pitfall.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
>> 上午6:49寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
>>> 
>>>>>>>>>> still feels weird to
>>>>>>>>>>> me.
>>>>>>>>>>>>> It
>>>>>>>>>>>>> categorizes partitions as new or existing based on the
>> partition
>>>>>>>>>>> creation
>>>>>>>>>>>>> time. Intuitively, the categorization should be based on the
>> group
>>>>>>>>>>> creation
>>>>>>>>>>>>> time: all partitions existing when the group is created are
>> existing
>>>>>>>>>> and
>>>>>>>>>>>>> all partitions created after the group creation are new
>> partitions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Manually bumping this thread. If there is no further
>>>>>>>>>>>>>> discussion, I will close the vote.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Jian,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for your feedback,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Agreed, partition expansion is a common operational task,
>> not an
>>>>>>>>>> edge
>>>>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify
>> that
>>>>>>>>>> this
>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>> common scenario rather than an edge case, which further
>>>>>>>>>> demonstrates
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> need for this optimization. For example:
>>>>>>>>>>>>>>>> A partition expansion is a common operational task in
>> Kafka: To
>>>>>>>>>>>>> balance
>>>>>>>>>>>>>>>> resource utilization and cost, topics are typically
>> created with a
>>>>>>>>>>>>>> moderate
>>>>>>>>>>>>>>>> default partition count. However, as traffic grows over
>> time, it
>>>>>>>>>> is
>>>>>>>>>>>>>> often
>>>>>>>>>>>>>>>> necessary to increase the number of partitions to
>> accommodate the
>>>>>>>>>>>>> higher
>>>>>>>>>>>>>>>> workload.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>> Jian
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29
>> 寫道:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
>>>>>>>>>> describing
>>>>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to
>>>>>>>>>> describe
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
>> applying
>>>>>>>>>> these
>>>>>>>>>>>>>> terms
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> After all, in this scenario, users don't really care
>> whether a
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
>> they won't
>>>>>>>>>>>>>> silently
>>>>>>>>>>>>>>>>>> lose any live records produced to the topic during their
>> active
>>>>>>>>>>>>>>>>> consumption.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
>> motivation
>>>>>>>>>>> section.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
>> 凌晨1:12
>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
>> motivation
>>>>>>>>>> for
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
>> partitions
>>>>>>>>>> without
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
>> partition
>>>>>>>>>> (hot)
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer
>> has
>>>>>>>>>> never
>>>>>>>>>>>>> seen
>>>>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
>>>>>>>>>> partition
>>>>>>>>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
>>>>>>>>>>> insufficient
>>>>>>>>>>>>>>>>>>> because:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies
>> across
>>>>>>>>>> nodes
>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
>>>>>>>>>> duration,
>>>>>>>>>>>>>>>>>>> causing
>>>>>>>>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek
>> time on
>>>>>>>>>>>>> retry,
>>>>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data
>> loss."
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> However, both of these situations are rare. If these
>> issues
>>>>>>>>>>>>> persist,
>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
>> situations don't
>>>>>>>>>>>>> need a
>>>>>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>>>>> solution. If users care about those rare situations,
>> they can
>>>>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>>>>> customized logic using
>>>>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
>> s7133700@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
>> should
>>>>>>>>>> always
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>>  any
>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
>>>>>>>>>> mismatch
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs
>> inside
>>>>>>>>>>>>> poll().
>>>>>>>>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException
>> from
>>>>>>>>>> poll().
>>>>>>>>>>>>>> I’ll
>>>>>>>>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
>> 下午4:50
>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>>>>>>>>>> partition
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>>>>>>>>>>>>> broker_current_time
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
>> does not
>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
>> should
>>>>>>>>>> always
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>>  any
>>>>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
>> calling
>>>>>>>>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on
>> the
>>>>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
>>>>>>>>>> behavior
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
>>> 
>>>>>>>>>> now applies to all
>>>>>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> 
>>>>>>>>>> .partitions
>>>>>>>>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented
>> by a
>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> .partitions,
>>>>>>>>>> currently fixed to
>>>>>>>>>>>>> earliest).
>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
>> public
>>>>>>>>>>>>>> user-facing
>>>>>>>>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
>> 清晨7:46
>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is
>> listed
>>>>>>>>>>> below:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> .partitions
>>>>>>>>>> with a default value
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
>> by_duration and
>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> does not change the logic of
>> auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> .partitions
>>>>>>>>>> as an internal
>>>>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> 
>>>>>>>>>> .partitions=earliest
>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases
>> of
>>>>>>>>>> other
>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> .partitions
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
>>> ,
>>>>>>>>>> will be applied to all for
>>>>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When
>> a Kafka
>>>>>>>>>>>>> topic
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest
>> auto
>>>>>>>>>> offset
>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
>>>>>>>>>> partitions
>>>>>>>>>>>>>> before
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
>> record loss
>>>>>>>>>>>>> issue
>>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> .partitions
>>>>>>>>>> will
>>>>>>>>>>> take
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set
>> it
>>>>>>>>>>>>>> by_duration if
>>>>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
>> something
>>>>>>>>>>>>>> important
>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
>> clarify
>>>>>>>>>> with
>>>>>>>>>>>>> a
>>>>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
>> logic
>>>>>>>>>> to
>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest
>> or
>>>>>>>>>>>>> by_duration
>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue
>> when a
>>>>>>>>>>>>>> partition
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
>> configure for
>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> .partitions?
>>>>>>>>>> If they set it to
>>>>>>>>>>>>> earliest
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
>>>>>>>>>> example,
>>>>>>>>>>>>> if a
>>>>>>>>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
>> created
>>>>>>>>>> during
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest
>> when
>>>>>>>>>>>>> resuming,
>>>>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition
>> is
>>>>>>>>>>>>>> technically
>>>>>>>>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
>> max.age
>>>>>>>>>>>>>> threshold.
>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
>> genuinely
>>>>>>>>>>> "hot/new"
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
>> weird. It
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems
>> that the
>>>>>>>>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
>> values
>>>>>>>>>> like
>>>>>>>>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
>> way to
>>>>>>>>>>> control
>>>>>>>>>>>>>>>>> newly
>>>>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
>>>>>>>>>> starts.
>>>>>>>>>>>>>> Have we
>>>>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
>>> 
>>>>>>>>>>> .partitions?
>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
>> defaults to
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
>> explicitly
>>>>>>>>>> to
>>>>>>>>>>>>>>>>> customize
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
>> s7133700@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37
>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>>>>>>>>>>>>> PartitionMetadata
>>>>>>>>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
>> computed
>>>>>>>>>>>>>> server-side
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>>>>>>>>> MembershipManager
>>>>>>>>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
>> metadata for
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even
>> if the
>>>>>>>>>>>>> topic
>>>>>>>>>>>>>> ID
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
>> supported
>>>>>>>>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time.
>> If the
>>>>>>>>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
>> PartitionAgeMs at
>>>>>>>>>> all
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
>> back to
>>>>>>>>>> latest
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
>> 2026年4月29日
>>>>>>>>>>> 下午4:04
>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
>> 'age'
>>>>>>>>>>>>> resolution
>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
>> control
>>>>>>>>>>>>> plane
>>>>>>>>>>>>>>>>>>> clean.
>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>>>>>>>>>>> inter-broker
>>>>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
>> source of
>>>>>>>>>>>>> truth
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
>> should be
>>>>>>>>>>>>>> negligible.
>>>>>>>>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
>> configured in
>>>>>>>>>>> minutes
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
>> brokers
>>>>>>>>>> won't
>>>>>>>>>>>>>> impact
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org>
>> 於
>>>>>>>>>>>>> 2026年4月29日
>>>>>>>>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
>> previous
>>>>>>>>>>>>>> conversation
>>>>>>>>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow
>> with the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
>> boolean;
>>>>>>>>>> the
>>>>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
>> boolean
>>>>>>>>>> is
>>>>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>>>>>>>>>> partition,
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive
>> the
>>>>>>>>>> ages.
>>>>>>>>>>> Is
>>>>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow
>> and
>>>>>>>>>> also
>>>>>>>>>>>>>> explain
>>>>>>>>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets,
>> fetch
>>>>>>>>>>>>> offsets,
>>>>>>>>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
>> wonder if
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
>> place
>>>>>>>>>> for
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively,
>> it could
>>>>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
>> cautious
>>>>>>>>>>> about
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
>> unrelated
>>>>>>>>>>> concepts.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>>>>>>>>>>> partitions.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
>> Streams
>>>>>>>>>> API
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we
>> do if
>>>>>>>>>> we
>>>>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
>> Muralidhar Basani
>>>>>>>>>>> via
>>>>>>>>>>>>>> dev
>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
>> about
>>>>>>>>>> the
>>>>>>>>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>>>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
>> PartitionAges
>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
>> recommended
>>>>>>>>>>>>> value
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>>>>>>>>>>>>> metadata.max.age.ms
>>>>>>>>>> <
>> https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
>>> 
>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
>> guess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to
>> Kafka
>>>>>>>>>>>>> Streams,
>>>>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>>>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it?
>> This is
>>>>>>>>>> the
>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
>> mistakenly
>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer
>> falls
>>>>>>>>>> back
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
>> per-partition
>>>>>>>>>> which
>>>>>>>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
>> resolves
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
>> earliest
>>>>>>>>>> was
>>>>>>>>>>>>>> applied
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
>> config is
>>>>>>>>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
>> change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>>>>>>>>> 2026年4月28日
>>>>>>>>>>>>>> 晚上8:03
>>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
>> creation
>>>>>>>>>>> time
>>>>>>>>>>>>>> via
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
>>>>>>>>>> diagnose
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
>> auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
>> KIP-1327
>>>>>>>>>>>>>> Prevent
>>>>>>>>>>>>>>>>> Hot
>>>>>>>>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
>> expanded
>>>>>>>>>>>>> (hot)
>>>>>>>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>>>>>>>>>>> automatically
>>>>>>>>>>>>>> fall
>>>>>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing
>> a full
>>>>>>>>>>>>>> historical
>>>>>>>>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>> 
>>> 
>>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Andrew

> I've discovered that we made a mistake implementing KIP-932 in this area,
so we'll rectify that and check whether it's really tight enough not to
require KIP-1327.

Would you mind sharing more details about that? If it's not anything close
to NP-hard, we could definitely look into covering it within KIP-1327

Best,
Chia-Ping


Andrew Schofield <sc...@apache.org> 於 2026年6月24日週三 下午7:49寫道:

> Hi Jiunn-Yang,
> I've just been re-reading this KIP to ensure that share group support is
> not needed. I've discovered that we made a mistake implementing KIP-932 in
> this area, so we'll rectify that and check whether it's really tight enough
> not to require KIP-1327.
>
> One trivial comments for consistency.
>
> AS1: In kafka-consumer-groups.sh, unknown column data is represented by
> "-", not "N/A". In kafka-topics.sh, we do use "N/A".
>
> Thanks,
> Andrew
>
> On 2026/06/23 23:23:38 黃竣陽 wrote:
> > Hi all,
> >
> > Manually bumping this thread.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > >
> > > Hello chia,
> > >
> > > Thanks for the feedback, I have updated the KIP.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > >> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> > >>
> > >> hi Jiunn-Yang
> > >>
> > >>> When the config is set on a cluster that has not yet been
> upgraded... classification cannot occur... the consumer falls back to the
> base auto.offset.reset for the affected partitions. No exception is thrown,
> and no operational disruption results.
> > >>
> > >> Existing group can't take advantage of this excellent new
> configuration. Allowing users to modify the group creation time might be
> overkill. Instead, we could print a useful warning message to guide users.
> For example, we can suggest that they re-create the group with their
> existing committed offsets
> > >>
> > >>> Protocol changes
> > >>
> > >> Would you mind listing those RPC changes in a table format?
> > >>
> > >>> The full interaction matrix between the base policy and the
> new-partition policy is:
> > >>
> > >> Please add a filed to describe the target scenario when using these
> policies
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >>
> > >> On 2026/06/16 16:14:49 黃竣陽 wrote:
> > >>> Hello Jun, chia,
> > >>>
> > >>> Thanks for the feedback, I have updated the KIP for the new
> > >>> approach, PTAL
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> > >>>>
> > >>>> hi Jun
> > >>>>
> > >>>> Yes, your approach is great. I think the combination of latest (for
> existing partitions) and by_duration (for new partitions) can address 99%
> of the complaints I have heard regarding this issue.
> > >>>>
> > >>>> Also, leveraging the group creation time here opens the door to
> implementing a new policy based on timestamp seek in the future, should the
> community want to pursue that.
> > >>>>
> > >>>> Thanks for your patience and constructive feedback. We will update
> the KIP accordingly.
> > >>>>
> > >>>> Best, Chia-Ping
> > >>>>
> > >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
> > >>>>>
> > >>>>> Hi, Chia-Ping,
> > >>>>>
> > >>>>> Thanks for the reply.
> > >>>>>
> > >>>>> I agree that it's probably useful to allow a user to configure a
> different
> > >>>>> offset policy for existing partitions vs new partitions. However,
> using
> > >>>>> group creation time to capture that seems more intuitive. Here is
> another
> > >>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new
> partitions
> > >>>>> based on group creation time. Introduce
> > >>>>> a new config auto.offset.reset.new.partitions whose values can be
> earliest,
> > >>>>> latest and by_duration, the same as auto.offset.reset. Users can
> set
> > >>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to
> guarantee
> > >>>>> no data loss on new partitions. They can also use by_duration to
> set an
> > >>>>> upper bound on the backlog replayed, which can be different from
> that of
> > >>>>> the existing partitions. This will address your concern about too
> much
> > >>>>> backlog being replayed when the offsets are lost. What do you
> think?
> > >>>>>
> > >>>>> Jun
> > >>>>>
> > >>>>>
> > >>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <
> chia7712@gmail.com> wrote:
> > >>>>>>
> > >>>>>> hi Jun
> > >>>>>>
> > >>>>>> The most important part of this story is how users should expect
> the data
> > >>>>>> they can see when using the latest or by_duration policy with
> expanded
> > >>>>>> partitions.
> > >>>>>>
> > >>>>>> Yes, the by_duration policy can minimize data loss, but it is
> > >>>>>> non-deterministic, which means users will either read too many
> historical
> > >>>>>> records from existing partitions or lose some records from
> expanded
> > >>>>>> partitions.
> > >>>>>>
> > >>>>>> Also, I agree that auto.offset.reset.max.age.ms
> > >>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >
> > >>>>>> is a bit hard to understand, and that is why I preferred having a
> whole new
> > >>>>>> policy based entirely on group creation time (KIP-1282)
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Chia-Ping
> > >>>>>>
> > >>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> > >>>>>>
> > >>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> > >>>>>>>
> > >>>>>>> Thanks for the reply. I am still trying to understand the value
> of the new
> > >>>>>>> configs with the KIP.
> > >>>>>>>
> > >>>>>>> The motivation of the KIP is that a user doesn't want to miss
> the data if
> > >>>>>>> the backlog is small. The backlog of the existing partition is
> easy to
> > >>>>>>> understand because it relates to retention time. The backlog for
> the new
> > >>>>>>> partition is a bit subtle to understand since it depends on the
> metadata
> > >>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >,
> > >>>>>>> the user needs to
> > >>>>>>> understand the metadata refresh delay on the consumer side and
> use it to
> > >>>>>>> set the config.
> > >>>>>>>
> > >>>>>>> Now, let's consider the alternative: setting the same value for
> the
> > >>>>>>> existing by_duration policy. The KIP lists three issues with this
> > >>>>>>> approach.
> > >>>>>>> 1. It computes the seek target client-side as now() - duration,
> which
> > >>>>>>> introduces clock skew across consumers and forces operators to
> choose
> > >>>>>>> overly large durations, causing unnecessary reprocessing.
> > >>>>>>> 2. The target timestamp is recomputed on each retry, so failed
> > >>>>>>> ListOffsetsRequest retries can shift the target forward and
> potentially
> > >>>>>>> miss records produced between attempts.
> > >>>>>>> 3. It applies uniformly to all partitions without committed
> offsets, and
> > >>>>>>> cannot distinguish newly expanded partitions from long-existing
> partitions
> > >>>>>>> newly assigned to the group, leading to unnecessary replay.
> > >>>>>>>
> > >>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit
> buffer to
> > >>>>>>> the metadata refresh delay. We could also consider improving the
> > >>>>>>> implementation. For issue 3, the metadata refresh delay is
> typically low
> > >>>>>>> (in the order of minutes with the classic consumer and tens of
> seconds
> > >>>>>>> with
> > >>>>>>> the new consumer). If a user is ok with reading that much
> backlog for new
> > >>>>>>> partitions, it seems they will be ok doing the same for existing
> > >>>>>>> partitions.
> > >>>>>>>
> > >>>>>>> So, instead of introducing a new config, could we just reuse the
> existing
> > >>>>>>> config with better documentation and/or implementation?
> > >>>>>>>
> > >>>>>>> Jun
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > >>>>>>>
> > >>>>>>>> Hello Jun,
> > >>>>>>>>
> > >>>>>>>> You're right that group creation time is the more intuitive
> answer at
> > >>>>>>>> first glance,
> > >>>>>>>> the KIP's own motivation talks about partitions that "predate
> the group"
> > >>>>>>>> vs partitions
> > >>>>>>>> "created during group runtime," which directly points to a
> > >>>>>>> group-lifecycle
> > >>>>>>>> classifier.
> > >>>>>>>> I'd like to walk through why we landed on partition age, and the
> > >>>>>>>> trade-offs we considered.
> > >>>>>>>>
> > >>>>>>>> We evaluated three candidate signals:
> > >>>>>>>>
> > >>>>>>>> 1. `by_duration:5secs`
> > >>>>>>>>
> > >>>>>>>> This covers the metadata blindness window, but has issues the
> KIP
> > >>>>>>>> currently documents
> > >>>>>>>> under "Why not use `by_duration`?":
> > >>>>>>>>
> > >>>>>>>> - Client-side `now() - duration` introduces clock skew across
> consumers.
> > >>>>>>>> - `ListOffsets` retries shift the target forward, potentially
> missing
> > >>>>>>>> records produced between
> > >>>>>>>> attempts.
> > >>>>>>>> - It applies uniformly to all partitions without committed
> offsets,
> > >>>>>>>> including pre-existing partitions
> > >>>>>>>> newly assigned to the group, causing unnecessary replay.
> > >>>>>>>>
> > >>>>>>>> 2. Group creation time as classifier
> > >>>>>>>>
> > >>>>>>>> This works cleanly when the consumer is actively running. Our
> concern
> > >>>>>>>> is the idle / late-rejoin case:
> > >>>>>>>>
> > >>>>>>>> T=0:         Group created.
> > >>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > >>>>>>>> T=50:        Partition added during the idle window.
> > >>>>>>>> T=100:       Consumer resumes.
> > >>>>>>>>
> > >>>>>>>> Under group creation time, the new partition is classified as
> new
> > >>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from
> T=50.
> > >>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated
> data that
> > >>>>>>>> the consumer accepts as lost — that is precisely the contract of
> > >>>>>>>> `auto.offset.reset=latest`. There is no principled reason to
> treat
> > >>>>>>>> the new partition differently; both contain backlog accumulated
> during
> > >>>>>>>> the same idle window.
> > >>>>>>>>
> > >>>>>>>> This aligns with the "backlog is backlog” principle you raised
> in
> > >>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog
> on
> > >>>>>>>> every other partition during the same idle period; forcing
> 0-backlog
> > >>>>>>>> tolerance only on new partitions would be inconsistent with that
> > >>>>>>>> tolerance.
> > >>>>>>>>
> > >>>>>>>> 3. Partition age vs threshold
> > >>>>>>>>
> > >>>>>>>> Partition age corresponds to the actual silent data loss window,
> > >>>>>>>> the gap between partition creation and the consumer’s metadata
> > >>>>>>>> refresh. Within this window, data loss is genuinely silent: the
> > >>>>>>>> consumer had no opportunity to know about the partition.
> Outside this
> > >>>>>>>> window, missing data reflects either:
> > >>>>>>>>
> > >>>>>>>> - (a) the user’s tolerated cost of running with idle consumers,
> or
> > >>>>>>>> - (b) an operational issue to surface via monitoring, not via
> reset
> > >>>>>>> policy.
> > >>>>>>>>
> > >>>>>>>> We did not choose partition age because it is more elegant than
> group
> > >>>>>>>> creation time — we chose it because its failure mode (requires a
> > >>>>>>>> threshold) is
> > >>>>>>>> less invasive than the failure mode of group creation time
> (overrides
> > >>>>>>>> user-stated
> > >>>>>>>> `latest` intent during idle periods).
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Jiunn-Yang
> > >>>>>>>>
> > >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > >>>>>>>>>
> > >>>>>>>>> Hi Jun,
> > >>>>>>>>>
> > >>>>>>>>> Relying on both creation times will create an inconsistent
> scenario. A
> > >>>>>>>>> consumer that lost all offsets due to a long sleep will seek
> to the
> > >>>>>>>>> beginning for the partitions created later than the group.
> > >>>>>>>>>
> > >>>>>>>>> That is why we initially proposed KIP-1282 to fix the
> inconsistency
> > >>>>>>>> using a
> > >>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus,
> KIP-1327
> > >>>>>>>> goes
> > >>>>>>>>> back to using flexible configurations to prevent users from
> falling
> > >>>>>>> into
> > >>>>>>>>> that pitfall.
> > >>>>>>>>>
> > >>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>
> > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六
> 上午6:49寫道:
> > >>>>>>>>>
> > >>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the reply and sorry for the late reply.
> > >>>>>>>>>>
> > >>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >
> > >>>>>>> still feels weird to
> > >>>>>>>> me.
> > >>>>>>>>>> It
> > >>>>>>>>>> categorizes partitions as new or existing based on the
> partition
> > >>>>>>>> creation
> > >>>>>>>>>> time. Intuitively, the categorization should be based on the
> group
> > >>>>>>>> creation
> > >>>>>>>>>> time: all partitions existing when the group is created are
> existing
> > >>>>>>> and
> > >>>>>>>>>> all partitions created after the group creation are new
> partitions.
> > >>>>>>>>>>
> > >>>>>>>>>> Jun
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi all,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Manually bumping this thread. If there is no further
> > >>>>>>>>>>> discussion, I will close the vote.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best Regards,
> > >>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>
> > >>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hello Jian,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for your feedback,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Agreed, partition expansion is a common operational task,
> not an
> > >>>>>>> edge
> > >>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Hi Jiunn-Yang:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify
> that
> > >>>>>>> this
> > >>>>>>>>>> is a
> > >>>>>>>>>>>>> common scenario rather than an edge case, which further
> > >>>>>>> demonstrates
> > >>>>>>>>>> the
> > >>>>>>>>>>>>> need for this optimization. For example:
> > >>>>>>>>>>>>> A partition expansion is a common operational task in
> Kafka: To
> > >>>>>>>>>> balance
> > >>>>>>>>>>>>> resource utilization and cost, topics are typically
> created with a
> > >>>>>>>>>>> moderate
> > >>>>>>>>>>>>> default partition count. However, as traffic grows over
> time, it
> > >>>>>>> is
> > >>>>>>>>>>> often
> > >>>>>>>>>>>>> necessary to increase the number of partitions to
> accommodate the
> > >>>>>>>>>> higher
> > >>>>>>>>>>>>> workload.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Regards
> > >>>>>>>>>>>>> Jian
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29
> 寫道:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi Jiunn-Yang,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> > >>>>>>> describing
> > >>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to
> > >>>>>>> describe
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>> "freshness" or the users' need for the records, but
> applying
> > >>>>>>> these
> > >>>>>>>>>>> terms
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> After all, in this scenario, users don't really care
> whether a
> > >>>>>>>>>>> partition
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that
> they won't
> > >>>>>>>>>>> silently
> > >>>>>>>>>>>>>>> lose any live records produced to the topic during their
> active
> > >>>>>>>>>>>>>> consumption.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best, Chia-Ping
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Hello Jun,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP
> motivation
> > >>>>>>>> section.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日
> 凌晨1:12
> > >>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger
> motivation
> > >>>>>>> for
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>> KIP.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all
> partitions
> > >>>>>>> without
> > >>>>>>>>>> a
> > >>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded
> partition
> > >>>>>>> (hot)
> > >>>>>>>> is
> > >>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer
> has
> > >>>>>>> never
> > >>>>>>>>>> seen
> > >>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> > >>>>>>> partition
> > >>>>>>>>>>>>>>>>> fundamentally different from the cold?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> > >>>>>>>> insufficient
> > >>>>>>>>>>>>>>>> because:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies
> across
> > >>>>>>> nodes
> > >>>>>>>>>>> due
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> > >>>>>>> duration,
> > >>>>>>>>>>>>>>>> causing
> > >>>>>>>>>>>>>>>>> unnecessary reprocessing.
> > >>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek
> time on
> > >>>>>>>>>> retry,
> > >>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data
> loss."
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> However, both of these situations are rare. If these
> issues
> > >>>>>>>>>> persist,
> > >>>>>>>>>>>>>> more
> > >>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare
> situations don't
> > >>>>>>>>>> need a
> > >>>>>>>>>>>>>>>> common
> > >>>>>>>>>>>>>>>>> solution. If users care about those rare situations,
> they can
> > >>>>>>>>>>> implement
> > >>>>>>>>>>>>>>>>> customized logic using
> > >>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <
> s7133700@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hello chia,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for the feedback,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> should
> > >>>>>>> always
> > >>>>>>>> be
> > >>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>   any
> > >>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
> > >>>>>>> mismatch
> > >>>>>>>> is
> > >>>>>>>>>>>>>> during
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs
> inside
> > >>>>>>>>>> poll().
> > >>>>>>>>>>>>>>>>>> Therefore, the
> > >>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException
> from
> > >>>>>>> poll().
> > >>>>>>>>>>> I’ll
> > >>>>>>>>>>>>>>>>>> clarify this in the KIP.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日
> 下午4:50
> > >>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> hi Jiunn
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> > >>>>>>> partition
> > >>>>>>>>>> in
> > >>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> > >>>>>>>>>>>>>> broker_current_time
> > >>>>>>>>>>>>>>>> -
> > >>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker
> does not
> > >>>>>>>>>> support
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value
> should
> > >>>>>>> always
> > >>>>>>>> be
> > >>>>>>>>>>>>>>>> greater
> > >>>>>>>>>>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive
>   any
> > >>>>>>>>>>> field
> > >>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when
> calling
> > >>>>>>>>>>>>>> `poll()`?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on
> the
> > >>>>>>>>>>> discussion
> > >>>>>>>>>>>>>>>>>>>> feedback.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
> > >>>>>>> behavior
> > >>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$
> >
> > >>>>>>> now applies to all
> > >>>>>>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>>>>>> values
> > >>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> > >>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> > >>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >
> > >>>>>>> .partitions
> > >>>>>>>>>>>>>>>>>>>> config setting
> > >>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> > >>>>>>> auto.offset.reset
> > >>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>> continues
> > >>>>>>>>>>>>>>>>>>>> to apply unchanged.
> > >>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented
> by a
> > >>>>>>>> separate
> > >>>>>>>>>>>>>>>>>> internal config
> > >>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >.partitions,
> > >>>>>>> currently fixed to
> > >>>>>>>>>> earliest).
> > >>>>>>>>>>>>>> This
> > >>>>>>>>>>>>>>>>>> decoupled design makes
> > >>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a
> public
> > >>>>>>>>>>> user-facing
> > >>>>>>>>>>>>>>>>>> configuration in a future KIP.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日
> 清晨7:46
> > >>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is
> listed
> > >>>>>>>> below:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >.partitions
> > >>>>>>> with a default value
> > >>>>>>>>>> of
> > >>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both
> by_duration and
> > >>>>>>>>>> latest,
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>> does not change the logic of
> auto.offset.reset=earliest.
> > >>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >.partitions
> > >>>>>>> as an internal
> > >>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >
> > >>>>>>> .partitions=earliest
> > >>>>>>>> already
> > >>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases
> of
> > >>>>>>> other
> > >>>>>>>>>>> values
> > >>>>>>>>>>>>>>>> in a
> > >>>>>>>>>>>>>>>>>> separate KIP.
> > >>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >.partitions
> > >>>>>>> and
> > >>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$
> >,
> > >>>>>>> will be applied to all for
> > >>>>>>>>>>>>>>>>>> consistency.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When
> a Kafka
> > >>>>>>>>>> topic
> > >>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> expanded
> > >>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest
> auto
> > >>>>>>> offset
> > >>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> > >>>>>>> partitions
> > >>>>>>>>>>> before
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > >>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same
> record loss
> > >>>>>>>>>> issue
> > >>>>>>>>>>>>>> could
> > >>>>>>>>>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>>>>>> happen, right?
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >.partitions
> > >>>>>>> will
> > >>>>>>>> take
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>> same
> > >>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set
> it
> > >>>>>>>>>>> by_duration if
> > >>>>>>>>>>>>>>>>>> needed.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > >>>>>>>>>>>>>> chia7712@apache.org
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing
> something
> > >>>>>>>>>>> important
> > >>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to
> clarify
> > >>>>>>> with
> > >>>>>>>>>> a
> > >>>>>>>>>>> few
> > >>>>>>>>>>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this
> logic
> > >>>>>>> to
> > >>>>>>>>>>> other
> > >>>>>>>>>>>>>>>> reset
> > >>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest
> or
> > >>>>>>>>>> by_duration
> > >>>>>>>>>>>>>>>> don't
> > >>>>>>>>>>>>>>>>>> seem
> > >>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue
> when a
> > >>>>>>>>>>> partition
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> expanded.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to
> configure for
> > >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >.partitions?
> > >>>>>>> If they set it to
> > >>>>>>>>>> earliest
> > >>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>> latest,
> > >>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> > >>>>>>> example,
> > >>>>>>>>>> if a
> > >>>>>>>>>>>>>>>>>> consumer is
> > >>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is
> created
> > >>>>>>> during
> > >>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>> downtime,
> > >>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest
> when
> > >>>>>>>>>> resuming,
> > >>>>>>>>>>>>>>>> rather
> > >>>>>>>>>>>>>>>>>> than
> > >>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition
> is
> > >>>>>>>>>>> technically
> > >>>>>>>>>>>>>>>>>> "new" to
> > >>>>>>>>>>>>>>>>>>>>>>> the group.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a
> max.age
> > >>>>>>>>>>> threshold.
> > >>>>>>>>>>>>>>>> It
> > >>>>>>>>>>>>>>>>>> gives
> > >>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is
> genuinely
> > >>>>>>>> "hot/new"
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>> what is
> > >>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit
> weird. It
> > >>>>>>>>>> only
> > >>>>>>>>>>>>>>>>>> applies when
> > >>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems
> that the
> > >>>>>>>>>>>>>> motivation
> > >>>>>>>>>>>>>>>>>>>>>>> equally
> > >>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other
> values
> > >>>>>>> like
> > >>>>>>>>>>>>>>>>>> by_duration.
> > >>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate
> way to
> > >>>>>>>> control
> > >>>>>>>>>>>>>> newly
> > >>>>>>>>>>>>>>>>>> created
> > >>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
> > >>>>>>> starts.
> > >>>>>>>>>>> Have we
> > >>>>>>>>>>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > >>>>>>> <
> https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$
> >
> > >>>>>>>> .partitions?
> > >>>>>>>>>>> If
> > >>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy
> defaults to
> > >>>>>>> the
> > >>>>>>>>>>>>>> policy
> > >>>>>>>>>>>>>>>>>> used
> > >>>>>>>>>>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it
> explicitly
> > >>>>>>> to
> > >>>>>>>>>>>>>> customize
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <
> s7133700@gmail.com
> > >>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37
> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > >>>>>>>>>>> PartitionMetadata
> > >>>>>>>>>>>>>>>>>> struct
> > >>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1),
> computed
> > >>>>>>>>>>> server-side
> > >>>>>>>>>>>>>>>> by
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> broker as
> > >>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > >>>>>>>>>>> MembershipManager
> > >>>>>>>>>>>>>>>>>> detects
> > >>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> > >>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the
> metadata for
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>> affected
> > >>>>>>>>>>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > >>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even
> if the
> > >>>>>>>>>> topic
> > >>>>>>>>>>> ID
> > >>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum
> supported
> > >>>>>>>>>>>>>>>>>> MetadataResponse
> > >>>>>>>>>>>>>>>>>>>>>>>>> version via the
> > >>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time.
> If the
> > >>>>>>>>>>> negotiated
> > >>>>>>>>>>>>>>>>>>>>>>> version is
> > >>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> > >>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support
> PartitionAgeMs at
> > >>>>>>> all
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>> can
> > >>>>>>>>>>>>>>>>>>>>>>> throw an
> > >>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling
> back to
> > >>>>>>> latest
> > >>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>> risking
> > >>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於
> 2026年4月29日
> > >>>>>>>> 下午4:04
> > >>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the
> 'age'
> > >>>>>>>>>> resolution
> > >>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the
> control
> > >>>>>>>>>> plane
> > >>>>>>>>>>>>>>>> clean.
> > >>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>> main
> > >>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > >>>>>>>> inter-broker
> > >>>>>>>>>>>>>> clock
> > >>>>>>>>>>>>>>>>>> skew.
> > >>>>>>>>>>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single
> source of
> > >>>>>>>>>> truth
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>> time.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew
> should be
> > >>>>>>>>>>> negligible.
> > >>>>>>>>>>>>>>>>>> Given
> > >>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be
> configured in
> > >>>>>>>> minutes
> > >>>>>>>>>>> or
> > >>>>>>>>>>>>>>>>>> hours, a
> > >>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between
> brokers
> > >>>>>>> won't
> > >>>>>>>>>>> impact
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org>
> 於
> > >>>>>>>>>> 2026年4月29日
> > >>>>>>>>>>>>>>>> 下午3:29
> > >>>>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the
> previous
> > >>>>>>>>>>> conversation
> > >>>>>>>>>>>>>>>> but I
> > >>>>>>>>>>>>>>>>>>>>>>> took a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow
> with the
> > >>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new
> boolean;
> > >>>>>>> the
> > >>>>>>>>>>> KIP
> > >>>>>>>>>>>>>>>>>> states
> > >>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this
> boolean
> > >>>>>>> is
> > >>>>>>>>>>> set.
> > >>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> > >>>>>>>>>>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> > >>>>>>> partition,
> > >>>>>>>>>> it
> > >>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>>>>>>> issue a
> > >>>>>>>>>>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive
> the
> > >>>>>>> ages.
> > >>>>>>>> Is
> > >>>>>>>>>>> my
> > >>>>>>>>>>>>>>>>>>>>>>>>> understanding
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow
> and
> > >>>>>>> also
> > >>>>>>>>>>> explain
> > >>>>>>>>>>>>>>>>>> how it
> > >>>>>>>>>>>>>>>>>>>>>>>>> fits
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets,
> fetch
> > >>>>>>>>>> offsets,
> > >>>>>>>>>>>>>>>> etc.).
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I
> wonder if
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right
> place
> > >>>>>>> for
> > >>>>>>>>>>> this
> > >>>>>>>>>>>>>>>> given
> > >>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>> a new
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively,
> it could
> > >>>>>>>>>> simply
> > >>>>>>>>>>>>>>>>>> include
> > >>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather
> cautious
> > >>>>>>>> about
> > >>>>>>>>>>> not
> > >>>>>>>>>>>>>>>>>>>>>>> overloading
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with
> unrelated
> > >>>>>>>> concepts.
> > >>>>>>>>>>> The
> > >>>>>>>>>>>>>>>> API
> > >>>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > >>>>>>>> partitions.
> > >>>>>>>>>>> The
> > >>>>>>>>>>>>>>>> fact
> > >>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding
> Streams
> > >>>>>>> API
> > >>>>>>>>>>> also
> > >>>>>>>>>>>>>>>>>> suggests
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we
> do if
> > >>>>>>> we
> > >>>>>>>>>>> want to
> > >>>>>>>>>>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM
> Muralidhar Basani
> > >>>>>>>> via
> > >>>>>>>>>>> dev
> > >>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know
> about
> > >>>>>>> the
> > >>>>>>>>>>> gap.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > >>>>>>>>>>> RequestPartitionAges
> > >>>>>>>>>>>>>>>>>> field.
> > >>>>>>>>>>>>>>>>>>>>>>> Can a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response,
> PartitionAges
> > >>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> > >>>>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a
> recommended
> > >>>>>>>>>> value
> > >>>>>>>>>>> or
> > >>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > >>>>>>>>>>>>>>>> metadata.max.age.ms
> > >>>>>>> <
> https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$
> >
> > >>>>>>>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I
> guess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to
> Kafka
> > >>>>>>>>>> Streams,
> > >>>>>>>>>>>>>>>> would
> > >>>>>>>>>>>>>>>>>> it
> > >>>>>>>>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> > >>>>>>>>>> auto.offset.reset.latest.max.age
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > >>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > >>>>>>>>>>>>>>>>>>>>>>>>> for a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it?
> This is
> > >>>>>>> the
> > >>>>>>>>>>> most
> > >>>>>>>>>>>>>>>>>>>>>>> familiar
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily
> mistakenly
> > >>>>>>>>>>> configure
> > >>>>>>>>>>>>>>>>>> it. Or
> > >>>>>>>>>>>>>>>>>>>>>>>>> may be
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer
> falls
> > >>>>>>> back
> > >>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>> earliest"
> > >>>>>>>>>>>>>>>>>>>>>>>>> reads as
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed
> per-partition
> > >>>>>>> which
> > >>>>>>>>>>> isn't
> > >>>>>>>>>>>>>>>>>>>>>>> supported.
> > >>>>>>>>>>>>>>>>>>>>>>>>> May
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer
> resolves
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>> initial
> > >>>>>>>>>>>>>>>>>>>>>>>>> position to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if
> earliest
> > >>>>>>> was
> > >>>>>>>>>>> applied
> > >>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset
> config is
> > >>>>>>>>>>> unchanged.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > >>>>>>>>>>> s7133700@gmail.com>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this
> change.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> > >>>>>>> 2026年4月28日
> > >>>>>>>>>>> 晚上8:03
> > >>>>>>>>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition
> creation
> > >>>>>>>> time
> > >>>>>>>>>>> via
> > >>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>> Admin
> > >>>>>>>>>>>>>>>>>>>>>>>>> API?
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> > >>>>>>> diagnose
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> > >>>>>>>>>>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of
> auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on
> KIP-1327
> > >>>>>>>>>>> Prevent
> > >>>>>>>>>>>>>> Hot
> > >>>>>>>>>>>>>>>>>> Data
> > >>>>>>>>>>>>>>>>>>>>>>>>> Loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > >>>>>>>>>>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly
> expanded
> > >>>>>>>>>> (hot)
> > >>>>>>>>>>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > >>>>>>>> automatically
> > >>>>>>>>>>> fall
> > >>>>>>>>>>>>>>>>>> back
> > >>>>>>>>>>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing
> a full
> > >>>>>>>>>>> historical
> > >>>>>>>>>>>>>>>>>>>>>>> reprocess.
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>
> > >>>
> > >>>
> > >
> >
> >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Andrew Schofield <sc...@apache.org>.
Hi Jiunn-Yang,
I've just been re-reading this KIP to ensure that share group support is not needed. I've discovered that we made a mistake implementing KIP-932 in this area, so we'll rectify that and check whether it's really tight enough not to require KIP-1327.

One trivial comments for consistency.

AS1: In kafka-consumer-groups.sh, unknown column data is represented by "-", not "N/A". In kafka-topics.sh, we do use "N/A".

Thanks,
Andrew

On 2026/06/23 23:23:38 黃竣陽 wrote:
> Hi all,
> 
> Manually bumping this thread. 
> 
> Best Regards,
> Jiunn-Yang
> 
> > 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> > 
> > Hello chia,
> > 
> > Thanks for the feedback, I have updated the KIP.
> > 
> > Best Regards,
> > Jiunn-Yang
> > 
> >> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> >> 
> >> hi Jiunn-Yang
> >> 
> >>> When the config is set on a cluster that has not yet been upgraded... classification cannot occur... the consumer falls back to the base auto.offset.reset for the affected partitions. No exception is thrown, and no operational disruption results.
> >> 
> >> Existing group can't take advantage of this excellent new configuration. Allowing users to modify the group creation time might be overkill. Instead, we could print a useful warning message to guide users. For example, we can suggest that they re-create the group with their existing committed offsets
> >> 
> >>> Protocol changes
> >> 
> >> Would you mind listing those RPC changes in a table format?
> >> 
> >>> The full interaction matrix between the base policy and the new-partition policy is:
> >> 
> >> Please add a filed to describe the target scenario when using these policies
> >> 
> >> Best,
> >> Chia-Ping
> >> 
> >> 
> >> On 2026/06/16 16:14:49 黃竣陽 wrote:
> >>> Hello Jun, chia,
> >>> 
> >>> Thanks for the feedback, I have updated the KIP for the new 
> >>> approach, PTAL
> >>> 
> >>> Best Regards,
> >>> Jiunn-Yang
> >>> 
> >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> >>>> 
> >>>> hi Jun
> >>>> 
> >>>> Yes, your approach is great. I think the combination of latest (for existing partitions) and by_duration (for new partitions) can address 99% of the complaints I have heard regarding this issue.
> >>>> 
> >>>> Also, leveraging the group creation time here opens the door to implementing a new policy based on timestamp seek in the future, should the community want to pursue that.
> >>>> 
> >>>> Thanks for your patience and constructive feedback. We will update the KIP accordingly.
> >>>> 
> >>>> Best, Chia-Ping
> >>>> 
> >>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
> >>>>> 
> >>>>> Hi, Chia-Ping,
> >>>>> 
> >>>>> Thanks for the reply.
> >>>>> 
> >>>>> I agree that it's probably useful to allow a user to configure a different
> >>>>> offset policy for existing partitions vs new partitions. However, using
> >>>>> group creation time to capture that seems more intuitive. Here is another
> >>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new partitions
> >>>>> based on group creation time. Introduce
> >>>>> a new config auto.offset.reset.new.partitions whose values can be earliest,
> >>>>> latest and by_duration, the same as auto.offset.reset. Users can set
> >>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to guarantee
> >>>>> no data loss on new partitions. They can also use by_duration to set an
> >>>>> upper bound on the backlog replayed, which can be different from that of
> >>>>> the existing partitions. This will address your concern about too much
> >>>>> backlog being replayed when the offsets are lost. What do you think?
> >>>>> 
> >>>>> Jun
> >>>>> 
> >>>>> 
> >>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
> >>>>>> 
> >>>>>> hi Jun
> >>>>>> 
> >>>>>> The most important part of this story is how users should expect the data
> >>>>>> they can see when using the latest or by_duration policy with expanded
> >>>>>> partitions.
> >>>>>> 
> >>>>>> Yes, the by_duration policy can minimize data loss, but it is
> >>>>>> non-deterministic, which means users will either read too many historical
> >>>>>> records from existing partitions or lose some records from expanded
> >>>>>> partitions.
> >>>>>> 
> >>>>>> Also, I agree that auto.offset.reset.max.age.ms
> >>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
> >>>>>> is a bit hard to understand, and that is why I preferred having a whole new
> >>>>>> policy based entirely on group creation time (KIP-1282)
> >>>>>> 
> >>>>>> Best,
> >>>>>> Chia-Ping
> >>>>>> 
> >>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> >>>>>> 
> >>>>>>> Hi, Chia-Ping and Jiunn-Yang,
> >>>>>>> 
> >>>>>>> Thanks for the reply. I am still trying to understand the value of the new
> >>>>>>> configs with the KIP.
> >>>>>>> 
> >>>>>>> The motivation of the KIP is that a user doesn't want to miss the data if
> >>>>>>> the backlog is small. The backlog of the existing partition is easy to
> >>>>>>> understand because it relates to retention time. The backlog for the new
> >>>>>>> partition is a bit subtle to understand since it depends on the metadata
> >>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>,
> >>>>>>> the user needs to
> >>>>>>> understand the metadata refresh delay on the consumer side and use it to
> >>>>>>> set the config.
> >>>>>>> 
> >>>>>>> Now, let's consider the alternative: setting the same value for the
> >>>>>>> existing by_duration policy. The KIP lists three issues with this
> >>>>>>> approach.
> >>>>>>> 1. It computes the seek target client-side as now() - duration, which
> >>>>>>> introduces clock skew across consumers and forces operators to choose
> >>>>>>> overly large durations, causing unnecessary reprocessing.
> >>>>>>> 2. The target timestamp is recomputed on each retry, so failed
> >>>>>>> ListOffsetsRequest retries can shift the target forward and potentially
> >>>>>>> miss records produced between attempts.
> >>>>>>> 3. It applies uniformly to all partitions without committed offsets, and
> >>>>>>> cannot distinguish newly expanded partitions from long-existing partitions
> >>>>>>> newly assigned to the group, leading to unnecessary replay.
> >>>>>>> 
> >>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
> >>>>>>> the metadata refresh delay. We could also consider improving the
> >>>>>>> implementation. For issue 3, the metadata refresh delay is typically low
> >>>>>>> (in the order of minutes with the classic consumer and tens of seconds
> >>>>>>> with
> >>>>>>> the new consumer). If a user is ok with reading that much backlog for new
> >>>>>>> partitions, it seems they will be ok doing the same for existing
> >>>>>>> partitions.
> >>>>>>> 
> >>>>>>> So, instead of introducing a new config, could we just reuse the existing
> >>>>>>> config with better documentation and/or implementation?
> >>>>>>> 
> >>>>>>> Jun
> >>>>>>> 
> >>>>>>> 
> >>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>> 
> >>>>>>>> Hello Jun,
> >>>>>>>> 
> >>>>>>>> You're right that group creation time is the more intuitive answer at
> >>>>>>>> first glance,
> >>>>>>>> the KIP's own motivation talks about partitions that "predate the group"
> >>>>>>>> vs partitions
> >>>>>>>> "created during group runtime," which directly points to a
> >>>>>>> group-lifecycle
> >>>>>>>> classifier.
> >>>>>>>> I'd like to walk through why we landed on partition age, and the
> >>>>>>>> trade-offs we considered.
> >>>>>>>> 
> >>>>>>>> We evaluated three candidate signals:
> >>>>>>>> 
> >>>>>>>> 1. `by_duration:5secs`
> >>>>>>>> 
> >>>>>>>> This covers the metadata blindness window, but has issues the KIP
> >>>>>>>> currently documents
> >>>>>>>> under "Why not use `by_duration`?":
> >>>>>>>> 
> >>>>>>>> - Client-side `now() - duration` introduces clock skew across consumers.
> >>>>>>>> - `ListOffsets` retries shift the target forward, potentially missing
> >>>>>>>> records produced between
> >>>>>>>> attempts.
> >>>>>>>> - It applies uniformly to all partitions without committed offsets,
> >>>>>>>> including pre-existing partitions
> >>>>>>>> newly assigned to the group, causing unnecessary replay.
> >>>>>>>> 
> >>>>>>>> 2. Group creation time as classifier
> >>>>>>>> 
> >>>>>>>> This works cleanly when the consumer is actively running. Our concern
> >>>>>>>> is the idle / late-rejoin case:
> >>>>>>>> 
> >>>>>>>> T=0:         Group created.
> >>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> >>>>>>>> T=50:        Partition added during the idle window.
> >>>>>>>> T=100:       Consumer resumes.
> >>>>>>>> 
> >>>>>>>> Under group creation time, the new partition is classified as new
> >>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
> >>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated data that
> >>>>>>>> the consumer accepts as lost — that is precisely the contract of
> >>>>>>>> `auto.offset.reset=latest`. There is no principled reason to treat
> >>>>>>>> the new partition differently; both contain backlog accumulated during
> >>>>>>>> the same idle window.
> >>>>>>>> 
> >>>>>>>> This aligns with the "backlog is backlog” principle you raised in
> >>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
> >>>>>>>> every other partition during the same idle period; forcing 0-backlog
> >>>>>>>> tolerance only on new partitions would be inconsistent with that
> >>>>>>>> tolerance.
> >>>>>>>> 
> >>>>>>>> 3. Partition age vs threshold
> >>>>>>>> 
> >>>>>>>> Partition age corresponds to the actual silent data loss window,
> >>>>>>>> the gap between partition creation and the consumer’s metadata
> >>>>>>>> refresh. Within this window, data loss is genuinely silent: the
> >>>>>>>> consumer had no opportunity to know about the partition. Outside this
> >>>>>>>> window, missing data reflects either:
> >>>>>>>> 
> >>>>>>>> - (a) the user’s tolerated cost of running with idle consumers, or
> >>>>>>>> - (b) an operational issue to surface via monitoring, not via reset
> >>>>>>> policy.
> >>>>>>>> 
> >>>>>>>> We did not choose partition age because it is more elegant than group
> >>>>>>>> creation time — we chose it because its failure mode (requires a
> >>>>>>>> threshold) is
> >>>>>>>> less invasive than the failure mode of group creation time (overrides
> >>>>>>>> user-stated
> >>>>>>>> `latest` intent during idle periods).
> >>>>>>>> 
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>> 
> >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> >>>>>>>>> 
> >>>>>>>>> Hi Jun,
> >>>>>>>>> 
> >>>>>>>>> Relying on both creation times will create an inconsistent scenario. A
> >>>>>>>>> consumer that lost all offsets due to a long sleep will seek to the
> >>>>>>>>> beginning for the partitions created later than the group.
> >>>>>>>>> 
> >>>>>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
> >>>>>>>> using a
> >>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
> >>>>>>>> goes
> >>>>>>>>> back to using flexible configurations to prevent users from falling
> >>>>>>> into
> >>>>>>>>> that pitfall.
> >>>>>>>>> 
> >>>>>>>>> Best, Chia-Ping
> >>>>>>>>> 
> >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> >>>>>>>>> 
> >>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>> 
> >>>>>>>>>> Thanks for the reply and sorry for the late reply.
> >>>>>>>>>> 
> >>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
> >>>>>>> still feels weird to
> >>>>>>>> me.
> >>>>>>>>>> It
> >>>>>>>>>> categorizes partitions as new or existing based on the partition
> >>>>>>>> creation
> >>>>>>>>>> time. Intuitively, the categorization should be based on the group
> >>>>>>>> creation
> >>>>>>>>>> time: all partitions existing when the group is created are existing
> >>>>>>> and
> >>>>>>>>>> all partitions created after the group creation are new partitions.
> >>>>>>>>>> 
> >>>>>>>>>> Jun
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>> 
> >>>>>>>>>>> Hi all,
> >>>>>>>>>>> 
> >>>>>>>>>>> Manually bumping this thread. If there is no further
> >>>>>>>>>>> discussion, I will close the vote.
> >>>>>>>>>>> 
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>> 
> >>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Hello Jian,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Thanks for your feedback,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Agreed, partition expansion is a common operational task, not an
> >>>>>>> edge
> >>>>>>>>>>>> case. I've updated the Motivation section accordingly.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Hi Jiunn-Yang:
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
> >>>>>>> this
> >>>>>>>>>> is a
> >>>>>>>>>>>>> common scenario rather than an edge case, which further
> >>>>>>> demonstrates
> >>>>>>>>>> the
> >>>>>>>>>>>>> need for this optimization. For example:
> >>>>>>>>>>>>> A partition expansion is a common operational task in Kafka: To
> >>>>>>>>>> balance
> >>>>>>>>>>>>> resource utilization and cost, topics are typically created with a
> >>>>>>>>>>> moderate
> >>>>>>>>>>>>> default partition count. However, as traffic grows over time, it
> >>>>>>> is
> >>>>>>>>>>> often
> >>>>>>>>>>>>> necessary to increase the number of partitions to accommodate the
> >>>>>>>>>> higher
> >>>>>>>>>>>>> workload.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Regards
> >>>>>>>>>>>>> Jian
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> >>>>>>> describing
> >>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to
> >>>>>>> describe
> >>>>>>>>>> the
> >>>>>>>>>>>>>>> "freshness" or the users' need for the records, but applying
> >>>>>>> these
> >>>>>>>>>>> terms
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> After all, in this scenario, users don't really care whether a
> >>>>>>>>>>> partition
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't
> >>>>>>>>>>> silently
> >>>>>>>>>>>>>>> lose any live records produced to the topic during their active
> >>>>>>>>>>>>>> consumption.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
> >>>>>>>> section.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12
> >>>>>>> 寫道:
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation
> >>>>>>> for
> >>>>>>>>>> the
> >>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
> >>>>>>> without
> >>>>>>>>>> a
> >>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded partition
> >>>>>>> (hot)
> >>>>>>>> is
> >>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer has
> >>>>>>> never
> >>>>>>>>>> seen
> >>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> >>>>>>> partition
> >>>>>>>>>>>>>>>>> fundamentally different from the cold?
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> >>>>>>>> insufficient
> >>>>>>>>>>>>>>>> because:
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
> >>>>>>> nodes
> >>>>>>>>>>> due
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> >>>>>>> duration,
> >>>>>>>>>>>>>>>> causing
> >>>>>>>>>>>>>>>>> unnecessary reprocessing.
> >>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on
> >>>>>>>>>> retry,
> >>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss."
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> However, both of these situations are rare. If these issues
> >>>>>>>>>> persist,
> >>>>>>>>>>>>>> more
> >>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
> >>>>>>>>>> need a
> >>>>>>>>>>>>>>>> common
> >>>>>>>>>>>>>>>>> solution. If users care about those rare situations, they can
> >>>>>>>>>>> implement
> >>>>>>>>>>>>>>>>> customized logic using
> >>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Thanks for the feedback,
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
> >>>>>>> always
> >>>>>>>> be
> >>>>>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >>>>>>>>>>> field
> >>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
> >>>>>>> mismatch
> >>>>>>>> is
> >>>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
> >>>>>>>>>> poll().
> >>>>>>>>>>>>>>>>>> Therefore, the
> >>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
> >>>>>>> poll().
> >>>>>>>>>>> I’ll
> >>>>>>>>>>>>>>>>>> clarify this in the KIP.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50
> >>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> >>>>>>> partition
> >>>>>>>>>> in
> >>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> >>>>>>>>>>>>>> broker_current_time
> >>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
> >>>>>>>>>> support
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
> >>>>>>> always
> >>>>>>>> be
> >>>>>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >>>>>>>>>>> field
> >>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
> >>>>>>>>>>>>>> `poll()`?
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
> >>>>>>>>>>> discussion
> >>>>>>>>>>>>>>>>>>>> feedback.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
> >>>>>>> behavior
> >>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
> >>>>>>> now applies to all
> >>>>>>>>>>> auto.offset.reset
> >>>>>>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> >>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> >>>>>>>>>>>>>>>>>> auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
> >>>>>>> .partitions
> >>>>>>>>>>>>>>>>>>>> config setting
> >>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> >>>>>>> auto.offset.reset
> >>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>> continues
> >>>>>>>>>>>>>>>>>>>> to apply unchanged.
> >>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
> >>>>>>>> separate
> >>>>>>>>>>>>>>>>>> internal config
> >>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions,
> >>>>>>> currently fixed to
> >>>>>>>>>> earliest).
> >>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>> decoupled design makes
> >>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
> >>>>>>>>>>> user-facing
> >>>>>>>>>>>>>>>>>> configuration in a future KIP.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
> >>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
> >>>>>>>> below:
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
> >>>>>>> with a default value
> >>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
> >>>>>>>>>> latest,
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
> >>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
> >>>>>>> as an internal
> >>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
> >>>>>>> .partitions=earliest
> >>>>>>>> already
> >>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
> >>>>>>> other
> >>>>>>>>>>> values
> >>>>>>>>>>>>>>>> in a
> >>>>>>>>>>>>>>>>>> separate KIP.
> >>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
> >>>>>>> and
> >>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>,
> >>>>>>> will be applied to all for
> >>>>>>>>>>>>>>>>>> consistency.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
> >>>>>>>>>> topic
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> expanded
> >>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
> >>>>>>> offset
> >>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> >>>>>>> partitions
> >>>>>>>>>>> before
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
> >>>>>>>>>> issue
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>>>>> happen, right?
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
> >>>>>>> will
> >>>>>>>> take
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
> >>>>>>>>>>> by_duration if
> >>>>>>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> >>>>>>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
> >>>>>>>>>>> important
> >>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
> >>>>>>> with
> >>>>>>>>>> a
> >>>>>>>>>>> few
> >>>>>>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic
> >>>>>>> to
> >>>>>>>>>>> other
> >>>>>>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> >>>>>>>>>> by_duration
> >>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
> >>>>>>>>>>> partition
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> expanded.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
> >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions?
> >>>>>>> If they set it to
> >>>>>>>>>> earliest
> >>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> >>>>>>> example,
> >>>>>>>>>> if a
> >>>>>>>>>>>>>>>>>> consumer is
> >>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
> >>>>>>> during
> >>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> downtime,
> >>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
> >>>>>>>>>> resuming,
> >>>>>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
> >>>>>>>>>>> technically
> >>>>>>>>>>>>>>>>>> "new" to
> >>>>>>>>>>>>>>>>>>>>>>> the group.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
> >>>>>>>>>>> threshold.
> >>>>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>>>>> gives
> >>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
> >>>>>>>> "hot/new"
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> what is
> >>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
> >>>>>>>>>> only
> >>>>>>>>>>>>>>>>>> applies when
> >>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> >>>>>>>>>>>>>> motivation
> >>>>>>>>>>>>>>>>>>>>>>> equally
> >>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
> >>>>>>> like
> >>>>>>>>>>>>>>>>>> by_duration.
> >>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
> >>>>>>>> control
> >>>>>>>>>>>>>> newly
> >>>>>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
> >>>>>>> starts.
> >>>>>>>>>>> Have we
> >>>>>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> >>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
> >>>>>>>> .partitions?
> >>>>>>>>>>> If
> >>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
> >>>>>>> the
> >>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly
> >>>>>>> to
> >>>>>>>>>>>>>> customize
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7133700@gmail.com
> >>>>>>>> 
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> >>>>>>>>>>> PartitionMetadata
> >>>>>>>>>>>>>>>>>> struct
> >>>>>>>>>>>>>>>>>>>>>>>>> gains a new
> >>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> >>>>>>>>>>> server-side
> >>>>>>>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> broker as
> >>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> >>>>>>>>>>> MembershipManager
> >>>>>>>>>>>>>>>>>> detects
> >>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
> >>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
> >>>>>>>> the
> >>>>>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
> >>>>>>>>>> topic
> >>>>>>>>>>> ID
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> >>>>>>>>>>>>>>>>>> MetadataResponse
> >>>>>>>>>>>>>>>>>>>>>>>>> version via the
> >>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> >>>>>>>>>>> negotiated
> >>>>>>>>>>>>>>>>>>>>>>> version is
> >>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
> >>>>>>> all
> >>>>>>>>>> and
> >>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>> throw an
> >>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
> >>>>>>> latest
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> risking
> >>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
> >>>>>>>> 下午4:04
> >>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> >>>>>>>>>> resolution
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
> >>>>>>>>>> plane
> >>>>>>>>>>>>>>>> clean.
> >>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> >>>>>>>> inter-broker
> >>>>>>>>>>>>>> clock
> >>>>>>>>>>>>>>>>>> skew.
> >>>>>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
> >>>>>>>>>> truth
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
> >>>>>>>>>>> negligible.
> >>>>>>>>>>>>>>>>>> Given
> >>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
> >>>>>>>> minutes
> >>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>> hours, a
> >>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
> >>>>>>> won't
> >>>>>>>>>>> impact
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> >>>>>>>>>> 2026年4月29日
> >>>>>>>>>>>>>>>> 下午3:29
> >>>>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> >>>>>>>>>>> conversation
> >>>>>>>>>>>>>>>> but I
> >>>>>>>>>>>>>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
> >>>>>>> the
> >>>>>>>>>>> KIP
> >>>>>>>>>>>>>>>>>> states
> >>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
> >>>>>>> is
> >>>>>>>>>>> set.
> >>>>>>>>>>>>>>>>>>>>>>> Implicitly,
> >>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> >>>>>>> partition,
> >>>>>>>>>> it
> >>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>>> issue a
> >>>>>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
> >>>>>>> ages.
> >>>>>>>> Is
> >>>>>>>>>>> my
> >>>>>>>>>>>>>>>>>>>>>>>>> understanding
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
> >>>>>>> also
> >>>>>>>>>>> explain
> >>>>>>>>>>>>>>>>>> how it
> >>>>>>>>>>>>>>>>>>>>>>>>> fits
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> >>>>>>>>>> offsets,
> >>>>>>>>>>>>>>>> etc.).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
> >>>>>>> for
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>>> given
> >>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> a new
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
> >>>>>>>>>> simply
> >>>>>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
> >>>>>>>> about
> >>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>> overloading
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
> >>>>>>>> concepts.
> >>>>>>>>>>> The
> >>>>>>>>>>>>>>>> API
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> >>>>>>>> partitions.
> >>>>>>>>>>> The
> >>>>>>>>>>>>>>>> fact
> >>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
> >>>>>>> API
> >>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>> suggests
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if
> >>>>>>> we
> >>>>>>>>>>> want to
> >>>>>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
> >>>>>>>> via
> >>>>>>>>>>> dev
> >>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
> >>>>>>> the
> >>>>>>>>>>> gap.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> >>>>>>>>>>> RequestPartitionAges
> >>>>>>>>>>>>>>>>>> field.
> >>>>>>>>>>>>>>>>>>>>>>> Can a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges
> >>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
> >>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
> >>>>>>>>>> value
> >>>>>>>>>>> or
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> >>>>>>>>>>>>>>>> metadata.max.age.ms
> >>>>>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$>
> >>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> >>>>>>>>>> Streams,
> >>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> >>>>>>>>>> auto.offset.reset.latest.max.age
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
> >>>>>>> the
> >>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
> >>>>>>>>>>> configure
> >>>>>>>>>>>>>>>>>> it. Or
> >>>>>>>>>>>>>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
> >>>>>>> back
> >>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> earliest"
> >>>>>>>>>>>>>>>>>>>>>>>>> reads as
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
> >>>>>>> which
> >>>>>>>>>>> isn't
> >>>>>>>>>>>>>>>>>>>>>>> supported.
> >>>>>>>>>>>>>>>>>>>>>>>>> May
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
> >>>>>>>> the
> >>>>>>>>>>>>>>>> initial
> >>>>>>>>>>>>>>>>>>>>>>>>> position to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest
> >>>>>>> was
> >>>>>>>>>>> applied
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> >>>>>>>>>>> unchanged.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> >>>>>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> >>>>>>> 2026年4月28日
> >>>>>>>>>>> 晚上8:03
> >>>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
> >>>>>>>> time
> >>>>>>>>>>> via
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> Admin
> >>>>>>>>>>>>>>>>>>>>>>>>> API?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> >>>>>>> diagnose
> >>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>> troubleshoot
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
> >>>>>>>>>>> Prevent
> >>>>>>>>>>>>>> Hot
> >>>>>>>>>>>>>>>>>> Data
> >>>>>>>>>>>>>>>>>>>>>>>>> Loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>> 
> >>>>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
> >>>>>>>>>> (hot)
> >>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> >>>>>>>> automatically
> >>>>>>>>>>> fall
> >>>>>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
> >>>>>>>>>>> historical
> >>>>>>>>>>>>>>>>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>> 
> >>> 
> >>> 
> > 
> 
> 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hi all,

Manually bumping this thread. 

Best Regards,
Jiunn-Yang

> 黃竣陽 <s7...@gmail.com> 於 2026年6月17日 晚上9:17 寫道:
> 
> Hello chia,
> 
> Thanks for the feedback, I have updated the KIP.
> 
> Best Regards,
> Jiunn-Yang
> 
>> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
>> 
>> hi Jiunn-Yang
>> 
>>> When the config is set on a cluster that has not yet been upgraded... classification cannot occur... the consumer falls back to the base auto.offset.reset for the affected partitions. No exception is thrown, and no operational disruption results.
>> 
>> Existing group can't take advantage of this excellent new configuration. Allowing users to modify the group creation time might be overkill. Instead, we could print a useful warning message to guide users. For example, we can suggest that they re-create the group with their existing committed offsets
>> 
>>> Protocol changes
>> 
>> Would you mind listing those RPC changes in a table format?
>> 
>>> The full interaction matrix between the base policy and the new-partition policy is:
>> 
>> Please add a filed to describe the target scenario when using these policies
>> 
>> Best,
>> Chia-Ping
>> 
>> 
>> On 2026/06/16 16:14:49 黃竣陽 wrote:
>>> Hello Jun, chia,
>>> 
>>> Thanks for the feedback, I have updated the KIP for the new 
>>> approach, PTAL
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
>>>> 
>>>> hi Jun
>>>> 
>>>> Yes, your approach is great. I think the combination of latest (for existing partitions) and by_duration (for new partitions) can address 99% of the complaints I have heard regarding this issue.
>>>> 
>>>> Also, leveraging the group creation time here opens the door to implementing a new policy based on timestamp seek in the future, should the community want to pursue that.
>>>> 
>>>> Thanks for your patience and constructive feedback. We will update the KIP accordingly.
>>>> 
>>>> Best, Chia-Ping
>>>> 
>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
>>>>> 
>>>>> Hi, Chia-Ping,
>>>>> 
>>>>> Thanks for the reply.
>>>>> 
>>>>> I agree that it's probably useful to allow a user to configure a different
>>>>> offset policy for existing partitions vs new partitions. However, using
>>>>> group creation time to capture that seems more intuitive. Here is another
>>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new partitions
>>>>> based on group creation time. Introduce
>>>>> a new config auto.offset.reset.new.partitions whose values can be earliest,
>>>>> latest and by_duration, the same as auto.offset.reset. Users can set
>>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to guarantee
>>>>> no data loss on new partitions. They can also use by_duration to set an
>>>>> upper bound on the backlog replayed, which can be different from that of
>>>>> the existing partitions. This will address your concern about too much
>>>>> backlog being replayed when the offsets are lost. What do you think?
>>>>> 
>>>>> Jun
>>>>> 
>>>>> 
>>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>>>>>> 
>>>>>> hi Jun
>>>>>> 
>>>>>> The most important part of this story is how users should expect the data
>>>>>> they can see when using the latest or by_duration policy with expanded
>>>>>> partitions.
>>>>>> 
>>>>>> Yes, the by_duration policy can minimize data loss, but it is
>>>>>> non-deterministic, which means users will either read too many historical
>>>>>> records from existing partitions or lose some records from expanded
>>>>>> partitions.
>>>>>> 
>>>>>> Also, I agree that auto.offset.reset.max.age.ms
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>>>>> is a bit hard to understand, and that is why I preferred having a whole new
>>>>>> policy based entirely on group creation time (KIP-1282)
>>>>>> 
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>> 
>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>>>>>> 
>>>>>>> Hi, Chia-Ping and Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the reply. I am still trying to understand the value of the new
>>>>>>> configs with the KIP.
>>>>>>> 
>>>>>>> The motivation of the KIP is that a user doesn't want to miss the data if
>>>>>>> the backlog is small. The backlog of the existing partition is easy to
>>>>>>> understand because it relates to retention time. The backlog for the new
>>>>>>> partition is a bit subtle to understand since it depends on the metadata
>>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>,
>>>>>>> the user needs to
>>>>>>> understand the metadata refresh delay on the consumer side and use it to
>>>>>>> set the config.
>>>>>>> 
>>>>>>> Now, let's consider the alternative: setting the same value for the
>>>>>>> existing by_duration policy. The KIP lists three issues with this
>>>>>>> approach.
>>>>>>> 1. It computes the seek target client-side as now() - duration, which
>>>>>>> introduces clock skew across consumers and forces operators to choose
>>>>>>> overly large durations, causing unnecessary reprocessing.
>>>>>>> 2. The target timestamp is recomputed on each retry, so failed
>>>>>>> ListOffsetsRequest retries can shift the target forward and potentially
>>>>>>> miss records produced between attempts.
>>>>>>> 3. It applies uniformly to all partitions without committed offsets, and
>>>>>>> cannot distinguish newly expanded partitions from long-existing partitions
>>>>>>> newly assigned to the group, leading to unnecessary replay.
>>>>>>> 
>>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
>>>>>>> the metadata refresh delay. We could also consider improving the
>>>>>>> implementation. For issue 3, the metadata refresh delay is typically low
>>>>>>> (in the order of minutes with the classic consumer and tens of seconds
>>>>>>> with
>>>>>>> the new consumer). If a user is ok with reading that much backlog for new
>>>>>>> partitions, it seems they will be ok doing the same for existing
>>>>>>> partitions.
>>>>>>> 
>>>>>>> So, instead of introducing a new config, could we just reuse the existing
>>>>>>> config with better documentation and/or implementation?
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> 
>>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hello Jun,
>>>>>>>> 
>>>>>>>> You're right that group creation time is the more intuitive answer at
>>>>>>>> first glance,
>>>>>>>> the KIP's own motivation talks about partitions that "predate the group"
>>>>>>>> vs partitions
>>>>>>>> "created during group runtime," which directly points to a
>>>>>>> group-lifecycle
>>>>>>>> classifier.
>>>>>>>> I'd like to walk through why we landed on partition age, and the
>>>>>>>> trade-offs we considered.
>>>>>>>> 
>>>>>>>> We evaluated three candidate signals:
>>>>>>>> 
>>>>>>>> 1. `by_duration:5secs`
>>>>>>>> 
>>>>>>>> This covers the metadata blindness window, but has issues the KIP
>>>>>>>> currently documents
>>>>>>>> under "Why not use `by_duration`?":
>>>>>>>> 
>>>>>>>> - Client-side `now() - duration` introduces clock skew across consumers.
>>>>>>>> - `ListOffsets` retries shift the target forward, potentially missing
>>>>>>>> records produced between
>>>>>>>> attempts.
>>>>>>>> - It applies uniformly to all partitions without committed offsets,
>>>>>>>> including pre-existing partitions
>>>>>>>> newly assigned to the group, causing unnecessary replay.
>>>>>>>> 
>>>>>>>> 2. Group creation time as classifier
>>>>>>>> 
>>>>>>>> This works cleanly when the consumer is actively running. Our concern
>>>>>>>> is the idle / late-rejoin case:
>>>>>>>> 
>>>>>>>> T=0:         Group created.
>>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>>>>>> T=50:        Partition added during the idle window.
>>>>>>>> T=100:       Consumer resumes.
>>>>>>>> 
>>>>>>>> Under group creation time, the new partition is classified as new
>>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
>>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated data that
>>>>>>>> the consumer accepts as lost — that is precisely the contract of
>>>>>>>> `auto.offset.reset=latest`. There is no principled reason to treat
>>>>>>>> the new partition differently; both contain backlog accumulated during
>>>>>>>> the same idle window.
>>>>>>>> 
>>>>>>>> This aligns with the "backlog is backlog” principle you raised in
>>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
>>>>>>>> every other partition during the same idle period; forcing 0-backlog
>>>>>>>> tolerance only on new partitions would be inconsistent with that
>>>>>>>> tolerance.
>>>>>>>> 
>>>>>>>> 3. Partition age vs threshold
>>>>>>>> 
>>>>>>>> Partition age corresponds to the actual silent data loss window,
>>>>>>>> the gap between partition creation and the consumer’s metadata
>>>>>>>> refresh. Within this window, data loss is genuinely silent: the
>>>>>>>> consumer had no opportunity to know about the partition. Outside this
>>>>>>>> window, missing data reflects either:
>>>>>>>> 
>>>>>>>> - (a) the user’s tolerated cost of running with idle consumers, or
>>>>>>>> - (b) an operational issue to surface via monitoring, not via reset
>>>>>>> policy.
>>>>>>>> 
>>>>>>>> We did not choose partition age because it is more elegant than group
>>>>>>>> creation time — we chose it because its failure mode (requires a
>>>>>>>> threshold) is
>>>>>>>> less invasive than the failure mode of group creation time (overrides
>>>>>>>> user-stated
>>>>>>>> `latest` intent during idle periods).
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>>>>>>>>> 
>>>>>>>>> Hi Jun,
>>>>>>>>> 
>>>>>>>>> Relying on both creation times will create an inconsistent scenario. A
>>>>>>>>> consumer that lost all offsets due to a long sleep will seek to the
>>>>>>>>> beginning for the partitions created later than the group.
>>>>>>>>> 
>>>>>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
>>>>>>>> using a
>>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
>>>>>>>> goes
>>>>>>>>> back to using flexible configurations to prevent users from falling
>>>>>>> into
>>>>>>>>> that pitfall.
>>>>>>>>> 
>>>>>>>>> Best, Chia-Ping
>>>>>>>>> 
>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
>>>>>>>>> 
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>>>>>> 
>>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>>>>>> still feels weird to
>>>>>>>> me.
>>>>>>>>>> It
>>>>>>>>>> categorizes partitions as new or existing based on the partition
>>>>>>>> creation
>>>>>>>>>> time. Intuitively, the categorization should be based on the group
>>>>>>>> creation
>>>>>>>>>> time: all partitions existing when the group is created are existing
>>>>>>> and
>>>>>>>>>> all partitions created after the group creation are new partitions.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi all,
>>>>>>>>>>> 
>>>>>>>>>>> Manually bumping this thread. If there is no further
>>>>>>>>>>> discussion, I will close the vote.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hello Jian,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for your feedback,
>>>>>>>>>>>> 
>>>>>>>>>>>> Agreed, partition expansion is a common operational task, not an
>>>>>>> edge
>>>>>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
>>>>>>> this
>>>>>>>>>> is a
>>>>>>>>>>>>> common scenario rather than an edge case, which further
>>>>>>> demonstrates
>>>>>>>>>> the
>>>>>>>>>>>>> need for this optimization. For example:
>>>>>>>>>>>>> A partition expansion is a common operational task in Kafka: To
>>>>>>>>>> balance
>>>>>>>>>>>>> resource utilization and cost, topics are typically created with a
>>>>>>>>>>> moderate
>>>>>>>>>>>>> default partition count. However, as traffic grows over time, it
>>>>>>> is
>>>>>>>>>>> often
>>>>>>>>>>>>> necessary to increase the number of partitions to accommodate the
>>>>>>>>>> higher
>>>>>>>>>>>>> workload.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Jian
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
>>>>>>> describing
>>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to
>>>>>>> describe
>>>>>>>>>> the
>>>>>>>>>>>>>>> "freshness" or the users' need for the records, but applying
>>>>>>> these
>>>>>>>>>>> terms
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> After all, in this scenario, users don't really care whether a
>>>>>>>>>>> partition
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't
>>>>>>>>>>> silently
>>>>>>>>>>>>>>> lose any live records produced to the topic during their active
>>>>>>>>>>>>>> consumption.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
>>>>>>>> section.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12
>>>>>>> 寫道:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation
>>>>>>> for
>>>>>>>>>> the
>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
>>>>>>> without
>>>>>>>>>> a
>>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded partition
>>>>>>> (hot)
>>>>>>>> is
>>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer has
>>>>>>> never
>>>>>>>>>> seen
>>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
>>>>>>> partition
>>>>>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
>>>>>>>> insufficient
>>>>>>>>>>>>>>>> because:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
>>>>>>> nodes
>>>>>>>>>>> due
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
>>>>>>> duration,
>>>>>>>>>>>>>>>> causing
>>>>>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on
>>>>>>>>>> retry,
>>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss."
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> However, both of these situations are rare. If these issues
>>>>>>>>>> persist,
>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
>>>>>>>>>> need a
>>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>>> solution. If users care about those rare situations, they can
>>>>>>>>>>> implement
>>>>>>>>>>>>>>>>> customized logic using
>>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>>>>>> always
>>>>>>>> be
>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
>>>>>>> mismatch
>>>>>>>> is
>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
>>>>>>>>>> poll().
>>>>>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
>>>>>>> poll().
>>>>>>>>>>> I’ll
>>>>>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50
>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>>>>>>> partition
>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>>>>>>>>>> broker_current_time
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
>>>>>>>>>> support
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>>>>>> always
>>>>>>>> be
>>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>>>>>> field
>>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
>>>>>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
>>>>>>>>>>> discussion
>>>>>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
>>>>>>> behavior
>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>>>>>> now applies to all
>>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>>>>> .partitions
>>>>>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>>>>>> auto.offset.reset
>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
>>>>>>>> separate
>>>>>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions,
>>>>>>> currently fixed to
>>>>>>>>>> earliest).
>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
>>>>>>>>>>> user-facing
>>>>>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
>>>>>>>> below:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>>>>> with a default value
>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
>>>>>>>>>> latest,
>>>>>>>>>>> and
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>>>>> as an internal
>>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>>>>> .partitions=earliest
>>>>>>>> already
>>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
>>>>>>> other
>>>>>>>>>>> values
>>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>>>>> and
>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>,
>>>>>>> will be applied to all for
>>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
>>>>>>>>>> topic
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
>>>>>>> offset
>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
>>>>>>> partitions
>>>>>>>>>>> before
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
>>>>>>>>>> issue
>>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>>>>> will
>>>>>>>> take
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
>>>>>>>>>>> by_duration if
>>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
>>>>>>>>>>> important
>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
>>>>>>> with
>>>>>>>>>> a
>>>>>>>>>>> few
>>>>>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic
>>>>>>> to
>>>>>>>>>>> other
>>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
>>>>>>>>>> by_duration
>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
>>>>>>>>>>> partition
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions?
>>>>>>> If they set it to
>>>>>>>>>> earliest
>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
>>>>>>> example,
>>>>>>>>>> if a
>>>>>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
>>>>>>> during
>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
>>>>>>>>>> resuming,
>>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
>>>>>>>>>>> technically
>>>>>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
>>>>>>>>>>> threshold.
>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
>>>>>>>> "hot/new"
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
>>>>>>>>>> only
>>>>>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>>>>>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
>>>>>>> like
>>>>>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
>>>>>>>> control
>>>>>>>>>>>>>> newly
>>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
>>>>>>> starts.
>>>>>>>>>>> Have we
>>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>>>>>> .partitions?
>>>>>>>>>>> If
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
>>>>>>> the
>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly
>>>>>>> to
>>>>>>>>>>>>>> customize
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7133700@gmail.com
>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>>>>>>>>>> PartitionMetadata
>>>>>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
>>>>>>>>>>> server-side
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>>>>>> MembershipManager
>>>>>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
>>>>>>>> the
>>>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
>>>>>>>>>> topic
>>>>>>>>>>> ID
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
>>>>>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
>>>>>>> all
>>>>>>>>>> and
>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
>>>>>>> latest
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
>>>>>>>> 下午4:04
>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
>>>>>>>>>> resolution
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
>>>>>>>>>> plane
>>>>>>>>>>>>>>>> clean.
>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>>>>>>>> inter-broker
>>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
>>>>>>>>>> truth
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
>>>>>>>>>>> negligible.
>>>>>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
>>>>>>>> minutes
>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
>>>>>>> won't
>>>>>>>>>>> impact
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
>>>>>>>>>> 2026年4月29日
>>>>>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
>>>>>>>>>>> conversation
>>>>>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
>>>>>>> the
>>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
>>>>>>> is
>>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>>>>>>> partition,
>>>>>>>>>> it
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
>>>>>>> ages.
>>>>>>>> Is
>>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
>>>>>>> also
>>>>>>>>>>> explain
>>>>>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
>>>>>>>>>> offsets,
>>>>>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
>>>>>>> for
>>>>>>>>>>> this
>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
>>>>>>>>>> simply
>>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
>>>>>>>> about
>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
>>>>>>>> concepts.
>>>>>>>>>>> The
>>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>>>>>>>> partitions.
>>>>>>>>>>> The
>>>>>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
>>>>>>> API
>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if
>>>>>>> we
>>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
>>>>>>>> via
>>>>>>>>>>> dev
>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
>>>>>>> the
>>>>>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges
>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
>>>>>>>>>> value
>>>>>>>>>>> or
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>>>>>>>>>> metadata.max.age.ms
>>>>>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$>
>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
>>>>>>>>>> Streams,
>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
>>>>>>> the
>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
>>>>>>>>>>> configure
>>>>>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
>>>>>>> back
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
>>>>>>> which
>>>>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
>>>>>>>> the
>>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest
>>>>>>> was
>>>>>>>>>>> applied
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
>>>>>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>>>>>> 2026年4月28日
>>>>>>>>>>> 晚上8:03
>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
>>>>>>>> time
>>>>>>>>>>> via
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
>>>>>>> diagnose
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
>>>>>>>>>>> Prevent
>>>>>>>>>>>>>> Hot
>>>>>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
>>>>>>>>>> (hot)
>>>>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>>>>>>>> automatically
>>>>>>>>>>> fall
>>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
>>>>>>>>>>> historical
>>>>>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>> 
>>> 
>>> 
> 



Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello chia,

Thanks for the feedback, I have updated the KIP.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@apache.org> 於 2026年6月17日 凌晨12:47 寫道:
> 
> hi Jiunn-Yang
> 
>> When the config is set on a cluster that has not yet been upgraded... classification cannot occur... the consumer falls back to the base auto.offset.reset for the affected partitions. No exception is thrown, and no operational disruption results.
> 
> Existing group can't take advantage of this excellent new configuration. Allowing users to modify the group creation time might be overkill. Instead, we could print a useful warning message to guide users. For example, we can suggest that they re-create the group with their existing committed offsets
> 
>> Protocol changes
> 
> Would you mind listing those RPC changes in a table format?
> 
>> The full interaction matrix between the base policy and the new-partition policy is:
> 
> Please add a filed to describe the target scenario when using these policies
> 
> Best,
> Chia-Ping
> 
> 
> On 2026/06/16 16:14:49 黃竣陽 wrote:
>> Hello Jun, chia,
>> 
>> Thanks for the feedback, I have updated the KIP for the new 
>> approach, PTAL
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
>>> 
>>> hi Jun
>>> 
>>> Yes, your approach is great. I think the combination of latest (for existing partitions) and by_duration (for new partitions) can address 99% of the complaints I have heard regarding this issue.
>>> 
>>> Also, leveraging the group creation time here opens the door to implementing a new policy based on timestamp seek in the future, should the community want to pursue that.
>>> 
>>> Thanks for your patience and constructive feedback. We will update the KIP accordingly.
>>> 
>>> Best, Chia-Ping
>>> 
>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
>>>> 
>>>> Hi, Chia-Ping,
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> I agree that it's probably useful to allow a user to configure a different
>>>> offset policy for existing partitions vs new partitions. However, using
>>>> group creation time to capture that seems more intuitive. Here is another
>>>> proposal: remove auto.offset.reset.max.age.ms and categorize new partitions
>>>> based on group creation time. Introduce
>>>> a new config auto.offset.reset.new.partitions whose values can be earliest,
>>>> latest and by_duration, the same as auto.offset.reset. Users can set
>>>> `auto.offset.reset.new.partitions` to `earliest` if they want to guarantee
>>>> no data loss on new partitions. They can also use by_duration to set an
>>>> upper bound on the backlog replayed, which can be different from that of
>>>> the existing partitions. This will address your concern about too much
>>>> backlog being replayed when the offsets are lost. What do you think?
>>>> 
>>>> Jun
>>>> 
>>>> 
>>>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>>>>> 
>>>>> hi Jun
>>>>> 
>>>>> The most important part of this story is how users should expect the data
>>>>> they can see when using the latest or by_duration policy with expanded
>>>>> partitions.
>>>>> 
>>>>> Yes, the by_duration policy can minimize data loss, but it is
>>>>> non-deterministic, which means users will either read too many historical
>>>>> records from existing partitions or lose some records from expanded
>>>>> partitions.
>>>>> 
>>>>> Also, I agree that auto.offset.reset.max.age.ms
>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>>>> is a bit hard to understand, and that is why I preferred having a whole new
>>>>> policy based entirely on group creation time (KIP-1282)
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>>>>> 
>>>>>> Hi, Chia-Ping and Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply. I am still trying to understand the value of the new
>>>>>> configs with the KIP.
>>>>>> 
>>>>>> The motivation of the KIP is that a user doesn't want to miss the data if
>>>>>> the backlog is small. The backlog of the existing partition is easy to
>>>>>> understand because it relates to retention time. The backlog for the new
>>>>>> partition is a bit subtle to understand since it depends on the metadata
>>>>>> refresh delay. To set auto.offset.reset.max.age.ms
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>,
>>>>>> the user needs to
>>>>>> understand the metadata refresh delay on the consumer side and use it to
>>>>>> set the config.
>>>>>> 
>>>>>> Now, let's consider the alternative: setting the same value for the
>>>>>> existing by_duration policy. The KIP lists three issues with this
>>>>>> approach.
>>>>>> 1. It computes the seek target client-side as now() - duration, which
>>>>>> introduces clock skew across consumers and forces operators to choose
>>>>>> overly large durations, causing unnecessary reprocessing.
>>>>>> 2. The target timestamp is recomputed on each retry, so failed
>>>>>> ListOffsetsRequest retries can shift the target forward and potentially
>>>>>> miss records produced between attempts.
>>>>>> 3. It applies uniformly to all partitions without committed offsets, and
>>>>>> cannot distinguish newly expanded partitions from long-existing partitions
>>>>>> newly assigned to the group, leading to unnecessary replay.
>>>>>> 
>>>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
>>>>>> the metadata refresh delay. We could also consider improving the
>>>>>> implementation. For issue 3, the metadata refresh delay is typically low
>>>>>> (in the order of minutes with the classic consumer and tens of seconds
>>>>>> with
>>>>>> the new consumer). If a user is ok with reading that much backlog for new
>>>>>> partitions, it seems they will be ok doing the same for existing
>>>>>> partitions.
>>>>>> 
>>>>>> So, instead of introducing a new config, could we just reuse the existing
>>>>>> config with better documentation and/or implementation?
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> 
>>>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hello Jun,
>>>>>>> 
>>>>>>> You're right that group creation time is the more intuitive answer at
>>>>>>> first glance,
>>>>>>> the KIP's own motivation talks about partitions that "predate the group"
>>>>>>> vs partitions
>>>>>>> "created during group runtime," which directly points to a
>>>>>> group-lifecycle
>>>>>>> classifier.
>>>>>>> I'd like to walk through why we landed on partition age, and the
>>>>>>> trade-offs we considered.
>>>>>>> 
>>>>>>> We evaluated three candidate signals:
>>>>>>> 
>>>>>>> 1. `by_duration:5secs`
>>>>>>> 
>>>>>>> This covers the metadata blindness window, but has issues the KIP
>>>>>>> currently documents
>>>>>>> under "Why not use `by_duration`?":
>>>>>>> 
>>>>>>> - Client-side `now() - duration` introduces clock skew across consumers.
>>>>>>> - `ListOffsets` retries shift the target forward, potentially missing
>>>>>>> records produced between
>>>>>>> attempts.
>>>>>>> - It applies uniformly to all partitions without committed offsets,
>>>>>>> including pre-existing partitions
>>>>>>> newly assigned to the group, causing unnecessary replay.
>>>>>>> 
>>>>>>> 2. Group creation time as classifier
>>>>>>> 
>>>>>>> This works cleanly when the consumer is actively running. Our concern
>>>>>>> is the idle / late-rejoin case:
>>>>>>> 
>>>>>>> T=0:         Group created.
>>>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>>>>> T=50:        Partition added during the idle window.
>>>>>>> T=100:       Consumer resumes.
>>>>>>> 
>>>>>>> Under group creation time, the new partition is classified as new
>>>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
>>>>>>> But during `[T=1, T=100]`, base partitions also accumulated data that
>>>>>>> the consumer accepts as lost — that is precisely the contract of
>>>>>>> `auto.offset.reset=latest`. There is no principled reason to treat
>>>>>>> the new partition differently; both contain backlog accumulated during
>>>>>>> the same idle window.
>>>>>>> 
>>>>>>> This aligns with the "backlog is backlog” principle you raised in
>>>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
>>>>>>> every other partition during the same idle period; forcing 0-backlog
>>>>>>> tolerance only on new partitions would be inconsistent with that
>>>>>>> tolerance.
>>>>>>> 
>>>>>>> 3. Partition age vs threshold
>>>>>>> 
>>>>>>> Partition age corresponds to the actual silent data loss window,
>>>>>>> the gap between partition creation and the consumer’s metadata
>>>>>>> refresh. Within this window, data loss is genuinely silent: the
>>>>>>> consumer had no opportunity to know about the partition. Outside this
>>>>>>> window, missing data reflects either:
>>>>>>> 
>>>>>>> - (a) the user’s tolerated cost of running with idle consumers, or
>>>>>>> - (b) an operational issue to surface via monitoring, not via reset
>>>>>> policy.
>>>>>>> 
>>>>>>> We did not choose partition age because it is more elegant than group
>>>>>>> creation time — we chose it because its failure mode (requires a
>>>>>>> threshold) is
>>>>>>> less invasive than the failure mode of group creation time (overrides
>>>>>>> user-stated
>>>>>>> `latest` intent during idle periods).
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>>>>>>>> 
>>>>>>>> Hi Jun,
>>>>>>>> 
>>>>>>>> Relying on both creation times will create an inconsistent scenario. A
>>>>>>>> consumer that lost all offsets due to a long sleep will seek to the
>>>>>>>> beginning for the partitions created later than the group.
>>>>>>>> 
>>>>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
>>>>>>> using a
>>>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
>>>>>>> goes
>>>>>>>> back to using flexible configurations to prevent users from falling
>>>>>> into
>>>>>>>> that pitfall.
>>>>>>>> 
>>>>>>>> Best, Chia-Ping
>>>>>>>> 
>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>>>>> 
>>>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>>>>> still feels weird to
>>>>>>> me.
>>>>>>>>> It
>>>>>>>>> categorizes partitions as new or existing based on the partition
>>>>>>> creation
>>>>>>>>> time. Intuitively, the categorization should be based on the group
>>>>>>> creation
>>>>>>>>> time: all partitions existing when the group is created are existing
>>>>>> and
>>>>>>>>> all partitions created after the group creation are new partitions.
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> Manually bumping this thread. If there is no further
>>>>>>>>>> discussion, I will close the vote.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>>>>> 
>>>>>>>>>>> Hello Jian,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for your feedback,
>>>>>>>>>>> 
>>>>>>>>>>> Agreed, partition expansion is a common operational task, not an
>>>>>> edge
>>>>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
>>>>>> this
>>>>>>>>> is a
>>>>>>>>>>>> common scenario rather than an edge case, which further
>>>>>> demonstrates
>>>>>>>>> the
>>>>>>>>>>>> need for this optimization. For example:
>>>>>>>>>>>> A partition expansion is a common operational task in Kafka: To
>>>>>>>>> balance
>>>>>>>>>>>> resource utilization and cost, topics are typically created with a
>>>>>>>>>> moderate
>>>>>>>>>>>> default partition count. However, as traffic grows over time, it
>>>>>> is
>>>>>>>>>> often
>>>>>>>>>>>> necessary to increase the number of partitions to accommodate the
>>>>>>>>> higher
>>>>>>>>>>>> workload.
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Jian
>>>>>>>>>>>> 
>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
>>>>>> describing
>>>>>>>>>>>>>> partitions in the KIP? I understand you are using them to
>>>>>> describe
>>>>>>>>> the
>>>>>>>>>>>>>> "freshness" or the users' need for the records, but applying
>>>>>> these
>>>>>>>>>> terms
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> After all, in this scenario, users don't really care whether a
>>>>>>>>>> partition
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't
>>>>>>>>>> silently
>>>>>>>>>>>>>> lose any live records produced to the topic during their active
>>>>>>>>>>>>> consumption.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
>>>>>>> section.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12
>>>>>> 寫道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation
>>>>>> for
>>>>>>>>> the
>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
>>>>>> without
>>>>>>>>> a
>>>>>>>>>>>>>>>> committed offset are the same. A newly expanded partition
>>>>>> (hot)
>>>>>>> is
>>>>>>>>>>>>>>>> fundamentally different from a partition the consumer has
>>>>>> never
>>>>>>>>> seen
>>>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
>>>>>> partition
>>>>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
>>>>>>> insufficient
>>>>>>>>>>>>>>> because:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
>>>>>> nodes
>>>>>>>>>> due
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
>>>>>> duration,
>>>>>>>>>>>>>>> causing
>>>>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on
>>>>>>>>> retry,
>>>>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss."
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> However, both of these situations are rare. If these issues
>>>>>>>>> persist,
>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
>>>>>>>>> need a
>>>>>>>>>>>>>>> common
>>>>>>>>>>>>>>>> solution. If users care about those rare situations, they can
>>>>>>>>>> implement
>>>>>>>>>>>>>>>> customized logic using
>>>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>>>>> always
>>>>>>> be
>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>>>>> field
>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The earliest point at which we can detect the version
>>>>>> mismatch
>>>>>>> is
>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
>>>>>>>>> poll().
>>>>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
>>>>>> poll().
>>>>>>>>>> I’ll
>>>>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50
>>>>>> 寫道:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>>>>>> partition
>>>>>>>>> in
>>>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>>>>>>>>> broker_current_time
>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
>>>>>>>>> support
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>>>>> always
>>>>>>> be
>>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>>>>> field
>>>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
>>>>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
>>>>>>>>>> discussion
>>>>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
>>>>>> behavior
>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>>>>> now applies to all
>>>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>>>> .partitions
>>>>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>>>>> auto.offset.reset
>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
>>>>>>> separate
>>>>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>>>>> (auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions,
>>>>>> currently fixed to
>>>>>>>>> earliest).
>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
>>>>>>>>>> user-facing
>>>>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
>>>>>>> below:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>>>> with a default value
>>>>>>>>> of
>>>>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
>>>>>>>>> latest,
>>>>>>>>>> and
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>>>> as an internal
>>>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>>>> .partitions=earliest
>>>>>>> already
>>>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
>>>>>> other
>>>>>>>>>> values
>>>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>>>> and
>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>,
>>>>>> will be applied to all for
>>>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
>>>>>>>>> topic
>>>>>>>>>> is
>>>>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
>>>>>> offset
>>>>>>>>>> reset
>>>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
>>>>>> partitions
>>>>>>>>>> before
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
>>>>>>>>> issue
>>>>>>>>>>>>> could
>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>>>> will
>>>>>>> take
>>>>>>>>>> the
>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
>>>>>>>>>> by_duration if
>>>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
>>>>>>>>>> important
>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
>>>>>> with
>>>>>>>>> a
>>>>>>>>>> few
>>>>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic
>>>>>> to
>>>>>>>>>> other
>>>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
>>>>>>>>> by_duration
>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
>>>>>>>>>> partition
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions?
>>>>>> If they set it to
>>>>>>>>> earliest
>>>>>>>>>> or
>>>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
>>>>>> example,
>>>>>>>>> if a
>>>>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
>>>>>> during
>>>>>>>>> that
>>>>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
>>>>>>>>> resuming,
>>>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
>>>>>>>>>> technically
>>>>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
>>>>>>>>>> threshold.
>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
>>>>>>> "hot/new"
>>>>>>>>>> and
>>>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
>>>>>>>>> only
>>>>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>>>>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
>>>>>> like
>>>>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
>>>>>>> control
>>>>>>>>>>>>> newly
>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
>>>>>> starts.
>>>>>>>>>> Have we
>>>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>>>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>>>>> .partitions?
>>>>>>>>>> If
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
>>>>>> the
>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly
>>>>>> to
>>>>>>>>>>>>> customize
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7133700@gmail.com
>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>>>>>>>>> PartitionMetadata
>>>>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
>>>>>>>>>> server-side
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>>>>> MembershipManager
>>>>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
>>>>>>> the
>>>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
>>>>>>>>> topic
>>>>>>>>>> ID
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
>>>>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
>>>>>> all
>>>>>>>>> and
>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
>>>>>> latest
>>>>>>>>>> and
>>>>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
>>>>>>> 下午4:04
>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
>>>>>>>>> resolution
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
>>>>>>>>> plane
>>>>>>>>>>>>>>> clean.
>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>>>>>>> inter-broker
>>>>>>>>>>>>> clock
>>>>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
>>>>>>>>> truth
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
>>>>>>>>>> negligible.
>>>>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
>>>>>>> minutes
>>>>>>>>>> or
>>>>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
>>>>>> won't
>>>>>>>>>> impact
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
>>>>>>>>> 2026年4月29日
>>>>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
>>>>>>>>>> conversation
>>>>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
>>>>>> the
>>>>>>>>>> KIP
>>>>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
>>>>>> is
>>>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>>>>>> partition,
>>>>>>>>> it
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
>>>>>> ages.
>>>>>>> Is
>>>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
>>>>>> also
>>>>>>>>>> explain
>>>>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
>>>>>>>>> offsets,
>>>>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
>>>>>> for
>>>>>>>>>> this
>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
>>>>>>>>> simply
>>>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
>>>>>>> about
>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
>>>>>>> concepts.
>>>>>>>>>> The
>>>>>>>>>>>>>>> API
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>>>>>>> partitions.
>>>>>>>>>> The
>>>>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
>>>>>> API
>>>>>>>>>> also
>>>>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if
>>>>>> we
>>>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
>>>>>>> via
>>>>>>>>>> dev
>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
>>>>>> the
>>>>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges
>>>>>> on
>>>>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
>>>>>>>>> value
>>>>>>>>>> or
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>>>>>>>>> metadata.max.age.ms
>>>>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$>
>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
>>>>>>>>> Streams,
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
>>>>>> the
>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
>>>>>>>>>> configure
>>>>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
>>>>>> back
>>>>>>>>> to
>>>>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
>>>>>> which
>>>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
>>>>>>> the
>>>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest
>>>>>> was
>>>>>>>>>> applied
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
>>>>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>>>>> 2026年4月28日
>>>>>>>>>> 晚上8:03
>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
>>>>>>> time
>>>>>>>>>> via
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
>>>>>> diagnose
>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
>>>>>>>>>> Prevent
>>>>>>>>>>>>> Hot
>>>>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
>>>>>>>>> (hot)
>>>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>>>>>>> automatically
>>>>>>>>>> fall
>>>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
>>>>>>>>>> historical
>>>>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>> 
>> 
>> 



Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Jiunn-Yang

> When the config is set on a cluster that has not yet been upgraded... classification cannot occur... the consumer falls back to the base auto.offset.reset for the affected partitions. No exception is thrown, and no operational disruption results.

Existing group can't take advantage of this excellent new configuration. Allowing users to modify the group creation time might be overkill. Instead, we could print a useful warning message to guide users. For example, we can suggest that they re-create the group with their existing committed offsets

> Protocol changes

Would you mind listing those RPC changes in a table format?

> The full interaction matrix between the base policy and the new-partition policy is:

Please add a filed to describe the target scenario when using these policies

Best,
Chia-Ping


On 2026/06/16 16:14:49 黃竣陽 wrote:
> Hello Jun, chia,
> 
> Thanks for the feedback, I have updated the KIP for the new 
> approach, PTAL
> 
> Best Regards,
> Jiunn-Yang
> 
> > Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> > 
> > hi Jun
> > 
> > Yes, your approach is great. I think the combination of latest (for existing partitions) and by_duration (for new partitions) can address 99% of the complaints I have heard regarding this issue.
> > 
> > Also, leveraging the group creation time here opens the door to implementing a new policy based on timestamp seek in the future, should the community want to pursue that.
> > 
> > Thanks for your patience and constructive feedback. We will update the KIP accordingly.
> > 
> > Best, Chia-Ping
> > 
> >> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
> >> 
> >> Hi, Chia-Ping,
> >> 
> >> Thanks for the reply.
> >> 
> >> I agree that it's probably useful to allow a user to configure a different
> >> offset policy for existing partitions vs new partitions. However, using
> >> group creation time to capture that seems more intuitive. Here is another
> >> proposal: remove auto.offset.reset.max.age.ms and categorize new partitions
> >> based on group creation time. Introduce
> >> a new config auto.offset.reset.new.partitions whose values can be earliest,
> >> latest and by_duration, the same as auto.offset.reset. Users can set
> >> `auto.offset.reset.new.partitions` to `earliest` if they want to guarantee
> >> no data loss on new partitions. They can also use by_duration to set an
> >> upper bound on the backlog replayed, which can be different from that of
> >> the existing partitions. This will address your concern about too much
> >> backlog being replayed when the offsets are lost. What do you think?
> >> 
> >> Jun
> >> 
> >> 
> >>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
> >>> 
> >>> hi Jun
> >>> 
> >>> The most important part of this story is how users should expect the data
> >>> they can see when using the latest or by_duration policy with expanded
> >>> partitions.
> >>> 
> >>> Yes, the by_duration policy can minimize data loss, but it is
> >>> non-deterministic, which means users will either read too many historical
> >>> records from existing partitions or lose some records from expanded
> >>> partitions.
> >>> 
> >>> Also, I agree that auto.offset.reset.max.age.ms
> >>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
> >>> is a bit hard to understand, and that is why I preferred having a whole new
> >>> policy based entirely on group creation time (KIP-1282)
> >>> 
> >>> Best,
> >>> Chia-Ping
> >>> 
> >>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
> >>> 
> >>>> Hi, Chia-Ping and Jiunn-Yang,
> >>>> 
> >>>> Thanks for the reply. I am still trying to understand the value of the new
> >>>> configs with the KIP.
> >>>> 
> >>>> The motivation of the KIP is that a user doesn't want to miss the data if
> >>>> the backlog is small. The backlog of the existing partition is easy to
> >>>> understand because it relates to retention time. The backlog for the new
> >>>> partition is a bit subtle to understand since it depends on the metadata
> >>>> refresh delay. To set auto.offset.reset.max.age.ms
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>,
> >>>> the user needs to
> >>>> understand the metadata refresh delay on the consumer side and use it to
> >>>> set the config.
> >>>> 
> >>>> Now, let's consider the alternative: setting the same value for the
> >>>> existing by_duration policy. The KIP lists three issues with this
> >>>> approach.
> >>>> 1. It computes the seek target client-side as now() - duration, which
> >>>> introduces clock skew across consumers and forces operators to choose
> >>>> overly large durations, causing unnecessary reprocessing.
> >>>> 2. The target timestamp is recomputed on each retry, so failed
> >>>> ListOffsetsRequest retries can shift the target forward and potentially
> >>>> miss records produced between attempts.
> >>>> 3. It applies uniformly to all partitions without committed offsets, and
> >>>> cannot distinguish newly expanded partitions from long-existing partitions
> >>>> newly assigned to the group, leading to unnecessary replay.
> >>>> 
> >>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
> >>>> the metadata refresh delay. We could also consider improving the
> >>>> implementation. For issue 3, the metadata refresh delay is typically low
> >>>> (in the order of minutes with the classic consumer and tens of seconds
> >>>> with
> >>>> the new consumer). If a user is ok with reading that much backlog for new
> >>>> partitions, it seems they will be ok doing the same for existing
> >>>> partitions.
> >>>> 
> >>>> So, instead of introducing a new config, could we just reuse the existing
> >>>> config with better documentation and/or implementation?
> >>>> 
> >>>> Jun
> >>>> 
> >>>> 
> >>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>> 
> >>>>> Hello Jun,
> >>>>> 
> >>>>> You're right that group creation time is the more intuitive answer at
> >>>>> first glance,
> >>>>> the KIP's own motivation talks about partitions that "predate the group"
> >>>>> vs partitions
> >>>>> "created during group runtime," which directly points to a
> >>>> group-lifecycle
> >>>>> classifier.
> >>>>> I'd like to walk through why we landed on partition age, and the
> >>>>> trade-offs we considered.
> >>>>> 
> >>>>> We evaluated three candidate signals:
> >>>>> 
> >>>>> 1. `by_duration:5secs`
> >>>>> 
> >>>>> This covers the metadata blindness window, but has issues the KIP
> >>>>> currently documents
> >>>>> under "Why not use `by_duration`?":
> >>>>> 
> >>>>> - Client-side `now() - duration` introduces clock skew across consumers.
> >>>>> - `ListOffsets` retries shift the target forward, potentially missing
> >>>>> records produced between
> >>>>> attempts.
> >>>>> - It applies uniformly to all partitions without committed offsets,
> >>>>> including pre-existing partitions
> >>>>> newly assigned to the group, causing unnecessary replay.
> >>>>> 
> >>>>> 2. Group creation time as classifier
> >>>>> 
> >>>>> This works cleanly when the consumer is actively running. Our concern
> >>>>> is the idle / late-rejoin case:
> >>>>> 
> >>>>> T=0:         Group created.
> >>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> >>>>> T=50:        Partition added during the idle window.
> >>>>> T=100:       Consumer resumes.
> >>>>> 
> >>>>> Under group creation time, the new partition is classified as new
> >>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
> >>>>> But during `[T=1, T=100]`, base partitions also accumulated data that
> >>>>> the consumer accepts as lost — that is precisely the contract of
> >>>>> `auto.offset.reset=latest`. There is no principled reason to treat
> >>>>> the new partition differently; both contain backlog accumulated during
> >>>>> the same idle window.
> >>>>> 
> >>>>> This aligns with the "backlog is backlog” principle you raised in
> >>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
> >>>>> every other partition during the same idle period; forcing 0-backlog
> >>>>> tolerance only on new partitions would be inconsistent with that
> >>>>> tolerance.
> >>>>> 
> >>>>> 3. Partition age vs threshold
> >>>>> 
> >>>>> Partition age corresponds to the actual silent data loss window,
> >>>>> the gap between partition creation and the consumer’s metadata
> >>>>> refresh. Within this window, data loss is genuinely silent: the
> >>>>> consumer had no opportunity to know about the partition. Outside this
> >>>>> window, missing data reflects either:
> >>>>> 
> >>>>> - (a) the user’s tolerated cost of running with idle consumers, or
> >>>>> - (b) an operational issue to surface via monitoring, not via reset
> >>>> policy.
> >>>>> 
> >>>>> We did not choose partition age because it is more elegant than group
> >>>>> creation time — we chose it because its failure mode (requires a
> >>>>> threshold) is
> >>>>> less invasive than the failure mode of group creation time (overrides
> >>>>> user-stated
> >>>>> `latest` intent during idle periods).
> >>>>> 
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>> 
> >>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> >>>>>> 
> >>>>>> Hi Jun,
> >>>>>> 
> >>>>>> Relying on both creation times will create an inconsistent scenario. A
> >>>>>> consumer that lost all offsets due to a long sleep will seek to the
> >>>>>> beginning for the partitions created later than the group.
> >>>>>> 
> >>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
> >>>>> using a
> >>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
> >>>>> goes
> >>>>>> back to using flexible configurations to prevent users from falling
> >>>> into
> >>>>>> that pitfall.
> >>>>>> 
> >>>>>> Best, Chia-Ping
> >>>>>> 
> >>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> >>>>>> 
> >>>>>>> Hi, Jiunn-Yang,
> >>>>>>> 
> >>>>>>> Thanks for the reply and sorry for the late reply.
> >>>>>>> 
> >>>>>>> JR1. The design of auto.offset.reset.max.age.ms
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
> >>>> still feels weird to
> >>>>> me.
> >>>>>>> It
> >>>>>>> categorizes partitions as new or existing based on the partition
> >>>>> creation
> >>>>>>> time. Intuitively, the categorization should be based on the group
> >>>>> creation
> >>>>>>> time: all partitions existing when the group is created are existing
> >>>> and
> >>>>>>> all partitions created after the group creation are new partitions.
> >>>>>>> 
> >>>>>>> Jun
> >>>>>>> 
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>> 
> >>>>>>>> Hi all,
> >>>>>>>> 
> >>>>>>>> Manually bumping this thread. If there is no further
> >>>>>>>> discussion, I will close the vote.
> >>>>>>>> 
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>> 
> >>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> >>>>>>>>> 
> >>>>>>>>> Hello Jian,
> >>>>>>>>> 
> >>>>>>>>> Thanks for your feedback,
> >>>>>>>>> 
> >>>>>>>>> Agreed, partition expansion is a common operational task, not an
> >>>> edge
> >>>>>>>>> case. I've updated the Motivation section accordingly.
> >>>>>>>>> 
> >>>>>>>>> Best Regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>> 
> >>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> >>>>>>>>>> 
> >>>>>>>>>> Hi Jiunn-Yang:
> >>>>>>>>>> 
> >>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
> >>>> this
> >>>>>>> is a
> >>>>>>>>>> common scenario rather than an edge case, which further
> >>>> demonstrates
> >>>>>>> the
> >>>>>>>>>> need for this optimization. For example:
> >>>>>>>>>> A partition expansion is a common operational task in Kafka: To
> >>>>>>> balance
> >>>>>>>>>> resource utilization and cost, topics are typically created with a
> >>>>>>>> moderate
> >>>>>>>>>> default partition count. However, as traffic grows over time, it
> >>>> is
> >>>>>>>> often
> >>>>>>>>>> necessary to increase the number of partitions to accommodate the
> >>>>>>> higher
> >>>>>>>>>> workload.
> >>>>>>>>>> 
> >>>>>>>>>> Regards
> >>>>>>>>>> Jian
> >>>>>>>>>> 
> >>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> >>>>>>>>>> 
> >>>>>>>>>>> Hello chia,
> >>>>>>>>>>> 
> >>>>>>>>>>> Thanks for the comments, I have updated the KIP!
> >>>>>>>>>>> 
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>> 
> >>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Hi Jiunn-Yang,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
> >>>> describing
> >>>>>>>>>>>> partitions in the KIP? I understand you are using them to
> >>>> describe
> >>>>>>> the
> >>>>>>>>>>>> "freshness" or the users' need for the records, but applying
> >>>> these
> >>>>>>>> terms
> >>>>>>>>>>> to
> >>>>>>>>>>>> the partition itself feels a bit unnatural.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> After all, in this scenario, users don't really care whether a
> >>>>>>>> partition
> >>>>>>>>>>> is
> >>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't
> >>>>>>>> silently
> >>>>>>>>>>>> lose any live records produced to the topic during their active
> >>>>>>>>>>> consumption.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Best, Chia-Ping
> >>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> Hello Jun,
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
> >>>>> section.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12
> >>>> 寫道:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation
> >>>> for
> >>>>>>> the
> >>>>>>>>>>> KIP.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
> >>>> without
> >>>>>>> a
> >>>>>>>>>>>>>> committed offset are the same. A newly expanded partition
> >>>> (hot)
> >>>>> is
> >>>>>>>>>>>>>> fundamentally different from a partition the consumer has
> >>>> never
> >>>>>>> seen
> >>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
> >>>> partition
> >>>>>>>>>>>>>> fundamentally different from the cold?
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
> >>>>> insufficient
> >>>>>>>>>>>>> because:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
> >>>> nodes
> >>>>>>>> due
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
> >>>> duration,
> >>>>>>>>>>>>> causing
> >>>>>>>>>>>>>> unnecessary reprocessing.
> >>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on
> >>>>>>> retry,
> >>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss."
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> However, both of these situations are rare. If these issues
> >>>>>>> persist,
> >>>>>>>>>>> more
> >>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
> >>>>>>> need a
> >>>>>>>>>>>>> common
> >>>>>>>>>>>>>> solution. If users care about those rare situations, they can
> >>>>>>>> implement
> >>>>>>>>>>>>>> customized logic using
> >>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Hello chia,
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Thanks for the feedback,
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> If the creation time exists, the returned value should
> >>>> always
> >>>>> be
> >>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >>>>>>>> field
> >>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> The earliest point at which we can detect the version
> >>>> mismatch
> >>>>> is
> >>>>>>>>>>> during
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
> >>>>>>> poll().
> >>>>>>>>>>>>>>> Therefore, the
> >>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
> >>>> poll().
> >>>>>>>> I’ll
> >>>>>>>>>>>>>>> clarify this in the KIP.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50
> >>>> 寫道:
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> >>>> partition
> >>>>>>> in
> >>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
> >>>>>>>>>>> broker_current_time
> >>>>>>>>>>>>> -
> >>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
> >>>>>>> support
> >>>>>>>>>>> this
> >>>>>>>>>>>>>>> feature or the partition creation time is unknown.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> If the creation time exists, the returned value should
> >>>> always
> >>>>> be
> >>>>>>>>>>>>> greater
> >>>>>>>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >>>>>>>> field
> >>>>>>>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
> >>>>>>>>>>> `poll()`?
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>> Hello Jun, chia,
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
> >>>>>>>> discussion
> >>>>>>>>>>>>>>>>> feedback.
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
> >>>> behavior
> >>>>>>>> from
> >>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
> >>>> now applies to all
> >>>>>>>> auto.offset.reset
> >>>>>>>>>>>>>>> values
> >>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
> >>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> >>>>>>>>>>>>>>> auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
> >>>> .partitions
> >>>>>>>>>>>>>>>>> config setting
> >>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
> >>>> auto.offset.reset
> >>>>>>>>>>> policy
> >>>>>>>>>>>>>>> continues
> >>>>>>>>>>>>>>>>> to apply unchanged.
> >>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
> >>>>> separate
> >>>>>>>>>>>>>>> internal config
> >>>>>>>>>>>>>>>>> (auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions,
> >>>> currently fixed to
> >>>>>>> earliest).
> >>>>>>>>>>> This
> >>>>>>>>>>>>>>> decoupled design makes
> >>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
> >>>>>>>> user-facing
> >>>>>>>>>>>>>>> configuration in a future KIP.
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
> >>>> 寫道:
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
> >>>>> below:
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
> >>>> with a default value
> >>>>>>> of
> >>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
> >>>>>>> latest,
> >>>>>>>> and
> >>>>>>>>>>>>> it
> >>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
> >>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
> >>>> as an internal
> >>>>>>>>>>>>>>> configuration. auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
> >>>> .partitions=earliest
> >>>>> already
> >>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
> >>>> other
> >>>>>>>> values
> >>>>>>>>>>>>> in a
> >>>>>>>>>>>>>>> separate KIP.
> >>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
> >>>> and
> >>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>,
> >>>> will be applied to all for
> >>>>>>>>>>>>>>> consistency.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
> >>>>>>> topic
> >>>>>>>> is
> >>>>>>>>>>>>>>> expanded
> >>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
> >>>> offset
> >>>>>>>> reset
> >>>>>>>>>>>>>>> policy
> >>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
> >>>> partitions
> >>>>>>>> before
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
> >>>>>>> issue
> >>>>>>>>>>> could
> >>>>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>>>>> happen, right?
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
> >>>> will
> >>>>> take
> >>>>>>>> the
> >>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
> >>>>>>>> by_duration if
> >>>>>>>>>>>>>>> needed.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> >>>>>>>>>>> chia7712@apache.org
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
> >>>>>>>> important
> >>>>>>>>>>>>> from
> >>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
> >>>> with
> >>>>>>> a
> >>>>>>>> few
> >>>>>>>>>>>>>>> questions:
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic
> >>>> to
> >>>>>>>> other
> >>>>>>>>>>>>> reset
> >>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> >>>>>>> by_duration
> >>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>> seem
> >>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
> >>>>>>>> partition
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>> expanded.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
> >>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions?
> >>>> If they set it to
> >>>>>>> earliest
> >>>>>>>> or
> >>>>>>>>>>>>>>> latest,
> >>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
> >>>> example,
> >>>>>>> if a
> >>>>>>>>>>>>>>> consumer is
> >>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
> >>>> during
> >>>>>>> that
> >>>>>>>>>>>>>>> downtime,
> >>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
> >>>>>>> resuming,
> >>>>>>>>>>>>> rather
> >>>>>>>>>>>>>>> than
> >>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
> >>>>>>>> technically
> >>>>>>>>>>>>>>> "new" to
> >>>>>>>>>>>>>>>>>>>> the group.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
> >>>>>>>> threshold.
> >>>>>>>>>>>>> It
> >>>>>>>>>>>>>>> gives
> >>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
> >>>>> "hot/new"
> >>>>>>>> and
> >>>>>>>>>>>>>>> what is
> >>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
> >>>>>>> only
> >>>>>>>>>>>>>>> applies when
> >>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> >>>>>>>>>>> motivation
> >>>>>>>>>>>>>>>>>>>> equally
> >>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
> >>>> like
> >>>>>>>>>>>>>>> by_duration.
> >>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
> >>>>> control
> >>>>>>>>>>> newly
> >>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
> >>>> starts.
> >>>>>>>> Have we
> >>>>>>>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> >>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
> >>>>> .partitions?
> >>>>>>>> If
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
> >>>> the
> >>>>>>>>>>> policy
> >>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly
> >>>> to
> >>>>>>>>>>> customize
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7133700@gmail.com
> >>>>> 
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> >>>>>>>> PartitionMetadata
> >>>>>>>>>>>>>>> struct
> >>>>>>>>>>>>>>>>>>>>>> gains a new
> >>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> >>>>>>>> server-side
> >>>>>>>>>>>>> by
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> broker as
> >>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> >>>>>>>> MembershipManager
> >>>>>>>>>>>>>>> detects
> >>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>> newly assigned
> >>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
> >>>>> the
> >>>>>>>>>>>>> affected
> >>>>>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
> >>>>>>> topic
> >>>>>>>> ID
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>>>>>>> in the cache.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> MB0:
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> >>>>>>>>>>>>>>> MetadataResponse
> >>>>>>>>>>>>>>>>>>>>>> version via the
> >>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> >>>>>>>> negotiated
> >>>>>>>>>>>>>>>>>>>> version is
> >>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
> >>>> all
> >>>>>>> and
> >>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>> throw an
> >>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
> >>>> latest
> >>>>>>>> and
> >>>>>>>>>>>>>>> risking
> >>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
> >>>>> 下午4:04
> >>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> >>>>>>> resolution
> >>>>>>>>>>> from
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
> >>>>>>> plane
> >>>>>>>>>>>>> clean.
> >>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> >>>>> inter-broker
> >>>>>>>>>>> clock
> >>>>>>>>>>>>>>> skew.
> >>>>>>>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
> >>>>>>> truth
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
> >>>>>>>> negligible.
> >>>>>>>>>>>>>>> Given
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
> >>>>> minutes
> >>>>>>>> or
> >>>>>>>>>>>>>>> hours, a
> >>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
> >>>> won't
> >>>>>>>> impact
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> >>>>>>> 2026年4月29日
> >>>>>>>>>>>>> 下午3:29
> >>>>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> >>>>>>>> conversation
> >>>>>>>>>>>>> but I
> >>>>>>>>>>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
> >>>> the
> >>>>>>>> KIP
> >>>>>>>>>>>>>>> states
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
> >>>> is
> >>>>>>>> set.
> >>>>>>>>>>>>>>>>>>>> Implicitly,
> >>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> >>>> partition,
> >>>>>>> it
> >>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>> issue a
> >>>>>>>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
> >>>> ages.
> >>>>> Is
> >>>>>>>> my
> >>>>>>>>>>>>>>>>>>>>>> understanding
> >>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
> >>>> also
> >>>>>>>> explain
> >>>>>>>>>>>>>>> how it
> >>>>>>>>>>>>>>>>>>>>>> fits
> >>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> >>>>>>> offsets,
> >>>>>>>>>>>>> etc.).
> >>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
> >>>> for
> >>>>>>>> this
> >>>>>>>>>>>>> given
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>> a new
> >>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
> >>>>>>> simply
> >>>>>>>>>>>>>>> include
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
> >>>>> about
> >>>>>>>> not
> >>>>>>>>>>>>>>>>>>>> overloading
> >>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
> >>>>> concepts.
> >>>>>>>> The
> >>>>>>>>>>>>> API
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> >>>>> partitions.
> >>>>>>>> The
> >>>>>>>>>>>>> fact
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
> >>>> API
> >>>>>>>> also
> >>>>>>>>>>>>>>> suggests
> >>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if
> >>>> we
> >>>>>>>> want to
> >>>>>>>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
> >>>>> via
> >>>>>>>> dev
> >>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
> >>>> the
> >>>>>>>> gap.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> >>>>>>>> RequestPartitionAges
> >>>>>>>>>>>>>>> field.
> >>>>>>>>>>>>>>>>>>>> Can a
> >>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges
> >>>> on
> >>>>>>>>>>>>>>>>>>>> TopicPartitions)
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
> >>>>>>> value
> >>>>>>>> or
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> >>>>>>>>>>>>> metadata.max.age.ms
> >>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$>
> >>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> >>>>>>> Streams,
> >>>>>>>>>>>>> would
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
> >>>>>>> auto.offset.reset.latest.max.age
> >>>>>>>> to
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
> >>>> the
> >>>>>>>> most
> >>>>>>>>>>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
> >>>>>>>> configure
> >>>>>>>>>>>>>>> it. Or
> >>>>>>>>>>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
> >>>> back
> >>>>>>> to
> >>>>>>>>>>>>>>> earliest"
> >>>>>>>>>>>>>>>>>>>>>> reads as
> >>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
> >>>> which
> >>>>>>>> isn't
> >>>>>>>>>>>>>>>>>>>> supported.
> >>>>>>>>>>>>>>>>>>>>>> May
> >>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
> >>>>> the
> >>>>>>>>>>>>> initial
> >>>>>>>>>>>>>>>>>>>>>> position to
> >>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest
> >>>> was
> >>>>>>>> applied
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> >>>>>>>> unchanged.
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> >>>>>>>> s7133700@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> >>>> 2026年4月28日
> >>>>>>>> 晚上8:03
> >>>>>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
> >>>>> time
> >>>>>>>> via
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> Admin
> >>>>>>>>>>>>>>>>>>>>>> API?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> >>>> diagnose
> >>>>>>> and
> >>>>>>>>>>>>>>>>>>>> troubleshoot
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
> >>>>>>>> Prevent
> >>>>>>>>>>> Hot
> >>>>>>>>>>>>>>> Data
> >>>>>>>>>>>>>>>>>>>>>> Loss
> >>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>> 
> >>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
> >>>>>>> (hot)
> >>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> >>>>> automatically
> >>>>>>>> fall
> >>>>>>>>>>>>>>> back
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
> >>>>>>>> historical
> >>>>>>>>>>>>>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>> 
> 
> 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello Jun, chia,

Thanks for the feedback, I have updated the KIP for the new 
approach, PTAL

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月16日 上午8:23 寫道:
> 
> hi Jun
> 
> Yes, your approach is great. I think the combination of latest (for existing partitions) and by_duration (for new partitions) can address 99% of the complaints I have heard regarding this issue.
> 
> Also, leveraging the group creation time here opens the door to implementing a new policy based on timestamp seek in the future, should the community want to pursue that.
> 
> Thanks for your patience and constructive feedback. We will update the KIP accordingly.
> 
> Best, Chia-Ping
> 
>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
>> 
>> Hi, Chia-Ping,
>> 
>> Thanks for the reply.
>> 
>> I agree that it's probably useful to allow a user to configure a different
>> offset policy for existing partitions vs new partitions. However, using
>> group creation time to capture that seems more intuitive. Here is another
>> proposal: remove auto.offset.reset.max.age.ms and categorize new partitions
>> based on group creation time. Introduce
>> a new config auto.offset.reset.new.partitions whose values can be earliest,
>> latest and by_duration, the same as auto.offset.reset. Users can set
>> `auto.offset.reset.new.partitions` to `earliest` if they want to guarantee
>> no data loss on new partitions. They can also use by_duration to set an
>> upper bound on the backlog replayed, which can be different from that of
>> the existing partitions. This will address your concern about too much
>> backlog being replayed when the offsets are lost. What do you think?
>> 
>> Jun
>> 
>> 
>>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>>> 
>>> hi Jun
>>> 
>>> The most important part of this story is how users should expect the data
>>> they can see when using the latest or by_duration policy with expanded
>>> partitions.
>>> 
>>> Yes, the by_duration policy can minimize data loss, but it is
>>> non-deterministic, which means users will either read too many historical
>>> records from existing partitions or lose some records from expanded
>>> partitions.
>>> 
>>> Also, I agree that auto.offset.reset.max.age.ms
>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>> is a bit hard to understand, and that is why I preferred having a whole new
>>> policy based entirely on group creation time (KIP-1282)
>>> 
>>> Best,
>>> Chia-Ping
>>> 
>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>>> 
>>>> Hi, Chia-Ping and Jiunn-Yang,
>>>> 
>>>> Thanks for the reply. I am still trying to understand the value of the new
>>>> configs with the KIP.
>>>> 
>>>> The motivation of the KIP is that a user doesn't want to miss the data if
>>>> the backlog is small. The backlog of the existing partition is easy to
>>>> understand because it relates to retention time. The backlog for the new
>>>> partition is a bit subtle to understand since it depends on the metadata
>>>> refresh delay. To set auto.offset.reset.max.age.ms
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>,
>>>> the user needs to
>>>> understand the metadata refresh delay on the consumer side and use it to
>>>> set the config.
>>>> 
>>>> Now, let's consider the alternative: setting the same value for the
>>>> existing by_duration policy. The KIP lists three issues with this
>>>> approach.
>>>> 1. It computes the seek target client-side as now() - duration, which
>>>> introduces clock skew across consumers and forces operators to choose
>>>> overly large durations, causing unnecessary reprocessing.
>>>> 2. The target timestamp is recomputed on each retry, so failed
>>>> ListOffsetsRequest retries can shift the target forward and potentially
>>>> miss records produced between attempts.
>>>> 3. It applies uniformly to all partitions without committed offsets, and
>>>> cannot distinguish newly expanded partitions from long-existing partitions
>>>> newly assigned to the group, leading to unnecessary replay.
>>>> 
>>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
>>>> the metadata refresh delay. We could also consider improving the
>>>> implementation. For issue 3, the metadata refresh delay is typically low
>>>> (in the order of minutes with the classic consumer and tens of seconds
>>>> with
>>>> the new consumer). If a user is ok with reading that much backlog for new
>>>> partitions, it seems they will be ok doing the same for existing
>>>> partitions.
>>>> 
>>>> So, instead of introducing a new config, could we just reuse the existing
>>>> config with better documentation and/or implementation?
>>>> 
>>>> Jun
>>>> 
>>>> 
>>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hello Jun,
>>>>> 
>>>>> You're right that group creation time is the more intuitive answer at
>>>>> first glance,
>>>>> the KIP's own motivation talks about partitions that "predate the group"
>>>>> vs partitions
>>>>> "created during group runtime," which directly points to a
>>>> group-lifecycle
>>>>> classifier.
>>>>> I'd like to walk through why we landed on partition age, and the
>>>>> trade-offs we considered.
>>>>> 
>>>>> We evaluated three candidate signals:
>>>>> 
>>>>> 1. `by_duration:5secs`
>>>>> 
>>>>> This covers the metadata blindness window, but has issues the KIP
>>>>> currently documents
>>>>> under "Why not use `by_duration`?":
>>>>> 
>>>>> - Client-side `now() - duration` introduces clock skew across consumers.
>>>>> - `ListOffsets` retries shift the target forward, potentially missing
>>>>> records produced between
>>>>> attempts.
>>>>> - It applies uniformly to all partitions without committed offsets,
>>>>> including pre-existing partitions
>>>>> newly assigned to the group, causing unnecessary replay.
>>>>> 
>>>>> 2. Group creation time as classifier
>>>>> 
>>>>> This works cleanly when the consumer is actively running. Our concern
>>>>> is the idle / late-rejoin case:
>>>>> 
>>>>> T=0:         Group created.
>>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>>> T=50:        Partition added during the idle window.
>>>>> T=100:       Consumer resumes.
>>>>> 
>>>>> Under group creation time, the new partition is classified as new
>>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
>>>>> But during `[T=1, T=100]`, base partitions also accumulated data that
>>>>> the consumer accepts as lost — that is precisely the contract of
>>>>> `auto.offset.reset=latest`. There is no principled reason to treat
>>>>> the new partition differently; both contain backlog accumulated during
>>>>> the same idle window.
>>>>> 
>>>>> This aligns with the "backlog is backlog” principle you raised in
>>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
>>>>> every other partition during the same idle period; forcing 0-backlog
>>>>> tolerance only on new partitions would be inconsistent with that
>>>>> tolerance.
>>>>> 
>>>>> 3. Partition age vs threshold
>>>>> 
>>>>> Partition age corresponds to the actual silent data loss window,
>>>>> the gap between partition creation and the consumer’s metadata
>>>>> refresh. Within this window, data loss is genuinely silent: the
>>>>> consumer had no opportunity to know about the partition. Outside this
>>>>> window, missing data reflects either:
>>>>> 
>>>>> - (a) the user’s tolerated cost of running with idle consumers, or
>>>>> - (b) an operational issue to surface via monitoring, not via reset
>>>> policy.
>>>>> 
>>>>> We did not choose partition age because it is more elegant than group
>>>>> creation time — we chose it because its failure mode (requires a
>>>>> threshold) is
>>>>> less invasive than the failure mode of group creation time (overrides
>>>>> user-stated
>>>>> `latest` intent during idle periods).
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>>>>>> 
>>>>>> Hi Jun,
>>>>>> 
>>>>>> Relying on both creation times will create an inconsistent scenario. A
>>>>>> consumer that lost all offsets due to a long sleep will seek to the
>>>>>> beginning for the partitions created later than the group.
>>>>>> 
>>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
>>>>> using a
>>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
>>>>> goes
>>>>>> back to using flexible configurations to prevent users from falling
>>>> into
>>>>>> that pitfall.
>>>>>> 
>>>>>> Best, Chia-Ping
>>>>>> 
>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
>>>>>> 
>>>>>>> Hi, Jiunn-Yang,
>>>>>>> 
>>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>>> 
>>>>>>> JR1. The design of auto.offset.reset.max.age.ms
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>>> still feels weird to
>>>>> me.
>>>>>>> It
>>>>>>> categorizes partitions as new or existing based on the partition
>>>>> creation
>>>>>>> time. Intuitively, the categorization should be based on the group
>>>>> creation
>>>>>>> time: all partitions existing when the group is created are existing
>>>> and
>>>>>>> all partitions created after the group creation are new partitions.
>>>>>>> 
>>>>>>> Jun
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Manually bumping this thread. If there is no further
>>>>>>>> discussion, I will close the vote.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>>> 
>>>>>>>>> Hello Jian,
>>>>>>>>> 
>>>>>>>>> Thanks for your feedback,
>>>>>>>>> 
>>>>>>>>> Agreed, partition expansion is a common operational task, not an
>>>> edge
>>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>>> 
>>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>>> 
>>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
>>>> this
>>>>>>> is a
>>>>>>>>>> common scenario rather than an edge case, which further
>>>> demonstrates
>>>>>>> the
>>>>>>>>>> need for this optimization. For example:
>>>>>>>>>> A partition expansion is a common operational task in Kafka: To
>>>>>>> balance
>>>>>>>>>> resource utilization and cost, topics are typically created with a
>>>>>>>> moderate
>>>>>>>>>> default partition count. However, as traffic grows over time, it
>>>> is
>>>>>>>> often
>>>>>>>>>> necessary to increase the number of partitions to accommodate the
>>>>>>> higher
>>>>>>>>>> workload.
>>>>>>>>>> 
>>>>>>>>>> Regards
>>>>>>>>>> Jian
>>>>>>>>>> 
>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>>> 
>>>>>>>>>>> Hello chia,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
>>>> describing
>>>>>>>>>>>> partitions in the KIP? I understand you are using them to
>>>> describe
>>>>>>> the
>>>>>>>>>>>> "freshness" or the users' need for the records, but applying
>>>> these
>>>>>>>> terms
>>>>>>>>>>> to
>>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>>> 
>>>>>>>>>>>> After all, in this scenario, users don't really care whether a
>>>>>>>> partition
>>>>>>>>>>> is
>>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't
>>>>>>>> silently
>>>>>>>>>>>> lose any live records produced to the topic during their active
>>>>>>>>>>> consumption.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
>>>>> section.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12
>>>> 寫道:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation
>>>> for
>>>>>>> the
>>>>>>>>>>> KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
>>>> without
>>>>>>> a
>>>>>>>>>>>>>> committed offset are the same. A newly expanded partition
>>>> (hot)
>>>>> is
>>>>>>>>>>>>>> fundamentally different from a partition the consumer has
>>>> never
>>>>>>> seen
>>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
>>>> partition
>>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
>>>>> insufficient
>>>>>>>>>>>>> because:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
>>>> nodes
>>>>>>>> due
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
>>>> duration,
>>>>>>>>>>>>> causing
>>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on
>>>>>>> retry,
>>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss."
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> However, both of these situations are rare. If these issues
>>>>>>> persist,
>>>>>>>>>>> more
>>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
>>>>>>> need a
>>>>>>>>>>>>> common
>>>>>>>>>>>>>> solution. If users care about those rare situations, they can
>>>>>>>> implement
>>>>>>>>>>>>>> customized logic using
>>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>>> always
>>>>> be
>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>>> field
>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The earliest point at which we can detect the version
>>>> mismatch
>>>>> is
>>>>>>>>>>> during
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
>>>>>>> poll().
>>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
>>>> poll().
>>>>>>>> I’ll
>>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50
>>>> 寫道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>>>> partition
>>>>>>> in
>>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>>>>>>> broker_current_time
>>>>>>>>>>>>> -
>>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
>>>>>>> support
>>>>>>>>>>> this
>>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>>> always
>>>>> be
>>>>>>>>>>>>> greater
>>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>>> field
>>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
>>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
>>>>>>>> discussion
>>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
>>>> behavior
>>>>>>>> from
>>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>>> now applies to all
>>>>>>>> auto.offset.reset
>>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>>>>>>>>>> auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>> .partitions
>>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>>> auto.offset.reset
>>>>>>>>>>> policy
>>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
>>>>> separate
>>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>>> (auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions,
>>>> currently fixed to
>>>>>>> earliest).
>>>>>>>>>>> This
>>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
>>>>>>>> user-facing
>>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
>>>> 寫道:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
>>>>> below:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>> with a default value
>>>>>>> of
>>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
>>>>>>> latest,
>>>>>>>> and
>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>> as an internal
>>>>>>>>>>>>>>> configuration. auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>> .partitions=earliest
>>>>> already
>>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
>>>> other
>>>>>>>> values
>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>> and
>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>,
>>>> will be applied to all for
>>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
>>>>>>> topic
>>>>>>>> is
>>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
>>>> offset
>>>>>>>> reset
>>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
>>>> partitions
>>>>>>>> before
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
>>>>>>> issue
>>>>>>>>>>> could
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>>> will
>>>>> take
>>>>>>>> the
>>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
>>>>>>>> by_duration if
>>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
>>>>>>>> important
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
>>>> with
>>>>>>> a
>>>>>>>> few
>>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic
>>>> to
>>>>>>>> other
>>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
>>>>>>> by_duration
>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
>>>>>>>> partition
>>>>>>>>>>> is
>>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions?
>>>> If they set it to
>>>>>>> earliest
>>>>>>>> or
>>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
>>>> example,
>>>>>>> if a
>>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
>>>> during
>>>>>>> that
>>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
>>>>>>> resuming,
>>>>>>>>>>>>> rather
>>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
>>>>>>>> technically
>>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
>>>>>>>> threshold.
>>>>>>>>>>>>> It
>>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
>>>>> "hot/new"
>>>>>>>> and
>>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
>>>>>>> only
>>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
>>>> like
>>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
>>>>> control
>>>>>>>>>>> newly
>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
>>>> starts.
>>>>>>>> Have we
>>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>>> .partitions?
>>>>>>>> If
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
>>>> the
>>>>>>>>>>> policy
>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly
>>>> to
>>>>>>>>>>> customize
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7133700@gmail.com
>>>>> 
>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>>>>>>> PartitionMetadata
>>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
>>>>>>>> server-side
>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>>> MembershipManager
>>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
>>>>> the
>>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
>>>>>>> topic
>>>>>>>> ID
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
>>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
>>>> all
>>>>>>> and
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
>>>> latest
>>>>>>>> and
>>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
>>>>> 下午4:04
>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
>>>>>>> resolution
>>>>>>>>>>> from
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
>>>>>>> plane
>>>>>>>>>>>>> clean.
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>>>>> inter-broker
>>>>>>>>>>> clock
>>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
>>>>>>> truth
>>>>>>>>>>> for
>>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
>>>>>>>> negligible.
>>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
>>>>> minutes
>>>>>>>> or
>>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
>>>> won't
>>>>>>>> impact
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
>>>>>>> 2026年4月29日
>>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
>>>>>>>> conversation
>>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
>>>> the
>>>>>>>> KIP
>>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
>>>> is
>>>>>>>> set.
>>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>>>> partition,
>>>>>>> it
>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
>>>> ages.
>>>>> Is
>>>>>>>> my
>>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
>>>> also
>>>>>>>> explain
>>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
>>>>>>> offsets,
>>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
>>>> for
>>>>>>>> this
>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
>>>>>>> simply
>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
>>>>> about
>>>>>>>> not
>>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
>>>>> concepts.
>>>>>>>> The
>>>>>>>>>>>>> API
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>>>>> partitions.
>>>>>>>> The
>>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
>>>> API
>>>>>>>> also
>>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if
>>>> we
>>>>>>>> want to
>>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
>>>>> via
>>>>>>>> dev
>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
>>>> the
>>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges
>>>> on
>>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
>>>>>>> value
>>>>>>>> or
>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>>>>>>> metadata.max.age.ms
>>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$>
>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
>>>>>>> Streams,
>>>>>>>>>>>>> would
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>>> auto.offset.reset.latest.max.age
>>>>>>>> to
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
>>>> the
>>>>>>>> most
>>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
>>>>>>>> configure
>>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
>>>> back
>>>>>>> to
>>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
>>>> which
>>>>>>>> isn't
>>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
>>>>> the
>>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest
>>>> was
>>>>>>>> applied
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
>>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>>> 2026年4月28日
>>>>>>>> 晚上8:03
>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
>>>>> time
>>>>>>>> via
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
>>>> diagnose
>>>>>>> and
>>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
>>>>>>>> Prevent
>>>>>>>>>>> Hot
>>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
>>>>>>> (hot)
>>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>>>>> automatically
>>>>>>>> fall
>>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
>>>>>>>> historical
>>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 



Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Jun

Yes, your approach is great. I think the combination of latest (for existing partitions) and by_duration (for new partitions) can address 99% of the complaints I have heard regarding this issue.

Also, leveraging the group creation time here opens the door to implementing a new policy based on timestamp seek in the future, should the community want to pursue that.

Thanks for your patience and constructive feedback. We will update the KIP accordingly.

Best, Chia-Ping

> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日 清晨5:11 寫道:
> 
> Hi, Chia-Ping,
> 
> Thanks for the reply.
> 
> I agree that it's probably useful to allow a user to configure a different
> offset policy for existing partitions vs new partitions. However, using
> group creation time to capture that seems more intuitive. Here is another
> proposal: remove auto.offset.reset.max.age.ms and categorize new partitions
> based on group creation time. Introduce
> a new config auto.offset.reset.new.partitions whose values can be earliest,
> latest and by_duration, the same as auto.offset.reset. Users can set
> `auto.offset.reset.new.partitions` to `earliest` if they want to guarantee
> no data loss on new partitions. They can also use by_duration to set an
> upper bound on the backlog replayed, which can be different from that of
> the existing partitions. This will address your concern about too much
> backlog being replayed when the offsets are lost. What do you think?
> 
> Jun
> 
> 
>> On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <ch...@gmail.com> wrote:
>> 
>> hi Jun
>> 
>> The most important part of this story is how users should expect the data
>> they can see when using the latest or by_duration policy with expanded
>> partitions.
>> 
>> Yes, the by_duration policy can minimize data loss, but it is
>> non-deterministic, which means users will either read too many historical
>> records from existing partitions or lose some records from expanded
>> partitions.
>> 
>> Also, I agree that auto.offset.reset.max.age.ms
>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>> is a bit hard to understand, and that is why I preferred having a whole new
>> policy based entirely on group creation time (KIP-1282)
>> 
>> Best,
>> Chia-Ping
>> 
>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>> 
>>> Hi, Chia-Ping and Jiunn-Yang,
>>> 
>>> Thanks for the reply. I am still trying to understand the value of the new
>>> configs with the KIP.
>>> 
>>> The motivation of the KIP is that a user doesn't want to miss the data if
>>> the backlog is small. The backlog of the existing partition is easy to
>>> understand because it relates to retention time. The backlog for the new
>>> partition is a bit subtle to understand since it depends on the metadata
>>> refresh delay. To set auto.offset.reset.max.age.ms
>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>,
>>> the user needs to
>>> understand the metadata refresh delay on the consumer side and use it to
>>> set the config.
>>> 
>>> Now, let's consider the alternative: setting the same value for the
>>> existing by_duration policy. The KIP lists three issues with this
>>> approach.
>>> 1. It computes the seek target client-side as now() - duration, which
>>> introduces clock skew across consumers and forces operators to choose
>>> overly large durations, causing unnecessary reprocessing.
>>> 2. The target timestamp is recomputed on each retry, so failed
>>> ListOffsetsRequest retries can shift the target forward and potentially
>>> miss records produced between attempts.
>>> 3. It applies uniformly to all partitions without committed offsets, and
>>> cannot distinguish newly expanded partitions from long-existing partitions
>>> newly assigned to the group, leading to unnecessary replay.
>>> 
>>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
>>> the metadata refresh delay. We could also consider improving the
>>> implementation. For issue 3, the metadata refresh delay is typically low
>>> (in the order of minutes with the classic consumer and tens of seconds
>>> with
>>> the new consumer). If a user is ok with reading that much backlog for new
>>> partitions, it seems they will be ok doing the same for existing
>>> partitions.
>>> 
>>> So, instead of introducing a new config, could we just reuse the existing
>>> config with better documentation and/or implementation?
>>> 
>>> Jun
>>> 
>>> 
>>>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hello Jun,
>>>> 
>>>> You're right that group creation time is the more intuitive answer at
>>>> first glance,
>>>> the KIP's own motivation talks about partitions that "predate the group"
>>>> vs partitions
>>>> "created during group runtime," which directly points to a
>>> group-lifecycle
>>>> classifier.
>>>> I'd like to walk through why we landed on partition age, and the
>>>> trade-offs we considered.
>>>> 
>>>> We evaluated three candidate signals:
>>>> 
>>>> 1. `by_duration:5secs`
>>>> 
>>>> This covers the metadata blindness window, but has issues the KIP
>>>> currently documents
>>>> under "Why not use `by_duration`?":
>>>> 
>>>> - Client-side `now() - duration` introduces clock skew across consumers.
>>>> - `ListOffsets` retries shift the target forward, potentially missing
>>>> records produced between
>>>> attempts.
>>>> - It applies uniformly to all partitions without committed offsets,
>>>> including pre-existing partitions
>>>> newly assigned to the group, causing unnecessary replay.
>>>> 
>>>> 2. Group creation time as classifier
>>>> 
>>>> This works cleanly when the consumer is actively running. Our concern
>>>> is the idle / late-rejoin case:
>>>> 
>>>> T=0:         Group created.
>>>> T=1..T=100:  Consumer idle (down, disconnected, etc.).
>>>> T=50:        Partition added during the idle window.
>>>> T=100:       Consumer resumes.
>>>> 
>>>> Under group creation time, the new partition is classified as new
>>>> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
>>>> But during `[T=1, T=100]`, base partitions also accumulated data that
>>>> the consumer accepts as lost — that is precisely the contract of
>>>> `auto.offset.reset=latest`. There is no principled reason to treat
>>>> the new partition differently; both contain backlog accumulated during
>>>> the same idle window.
>>>> 
>>>> This aligns with the "backlog is backlog” principle you raised in
>>>> the KIP-1282 thread: a `latest` user has tolerated some backlog on
>>>> every other partition during the same idle period; forcing 0-backlog
>>>> tolerance only on new partitions would be inconsistent with that
>>>> tolerance.
>>>> 
>>>> 3. Partition age vs threshold
>>>> 
>>>> Partition age corresponds to the actual silent data loss window,
>>>> the gap between partition creation and the consumer’s metadata
>>>> refresh. Within this window, data loss is genuinely silent: the
>>>> consumer had no opportunity to know about the partition. Outside this
>>>> window, missing data reflects either:
>>>> 
>>>> - (a) the user’s tolerated cost of running with idle consumers, or
>>>> - (b) an operational issue to surface via monitoring, not via reset
>>> policy.
>>>> 
>>>> We did not choose partition age because it is more elegant than group
>>>> creation time — we chose it because its failure mode (requires a
>>>> threshold) is
>>>> less invasive than the failure mode of group creation time (overrides
>>>> user-stated
>>>> `latest` intent during idle periods).
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>>>>> 
>>>>> Hi Jun,
>>>>> 
>>>>> Relying on both creation times will create an inconsistent scenario. A
>>>>> consumer that lost all offsets due to a long sleep will seek to the
>>>>> beginning for the partitions created later than the group.
>>>>> 
>>>>> That is why we initially proposed KIP-1282 to fix the inconsistency
>>>> using a
>>>>> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
>>>> goes
>>>>> back to using flexible configurations to prevent users from falling
>>> into
>>>>> that pitfall.
>>>>> 
>>>>> Best, Chia-Ping
>>>>> 
>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply and sorry for the late reply.
>>>>>> 
>>>>>> JR1. The design of auto.offset.reset.max.age.ms
>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>> still feels weird to
>>>> me.
>>>>>> It
>>>>>> categorizes partitions as new or existing based on the partition
>>>> creation
>>>>>> time. Intuitively, the categorization should be based on the group
>>>> creation
>>>>>> time: all partitions existing when the group is created are existing
>>> and
>>>>>> all partitions created after the group creation are new partitions.
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> Manually bumping this thread. If there is no further
>>>>>>> discussion, I will close the vote.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>>>>>> 
>>>>>>>> Hello Jian,
>>>>>>>> 
>>>>>>>> Thanks for your feedback,
>>>>>>>> 
>>>>>>>> Agreed, partition expansion is a common operational task, not an
>>> edge
>>>>>>>> case. I've updated the Motivation section accordingly.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>>>>>> 
>>>>>>>>> Hi Jiunn-Yang:
>>>>>>>>> 
>>>>>>>>> Thanks for the KIP. I think it would be useful to clarify that
>>> this
>>>>>> is a
>>>>>>>>> common scenario rather than an edge case, which further
>>> demonstrates
>>>>>> the
>>>>>>>>> need for this optimization. For example:
>>>>>>>>> A partition expansion is a common operational task in Kafka: To
>>>>>> balance
>>>>>>>>> resource utilization and cost, topics are typically created with a
>>>>>>> moderate
>>>>>>>>> default partition count. However, as traffic grows over time, it
>>> is
>>>>>>> often
>>>>>>>>> necessary to increase the number of partitions to accommodate the
>>>>>> higher
>>>>>>>>> workload.
>>>>>>>>> 
>>>>>>>>> Regards
>>>>>>>>> Jian
>>>>>>>>> 
>>>>>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>>>>>> 
>>>>>>>>>> Hello chia,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Jiunn-Yang,
>>>>>>>>>>> 
>>>>>>>>>>> Would you mind removing the terms "hot" and "cold" when
>>> describing
>>>>>>>>>>> partitions in the KIP? I understand you are using them to
>>> describe
>>>>>> the
>>>>>>>>>>> "freshness" or the users' need for the records, but applying
>>> these
>>>>>>> terms
>>>>>>>>>> to
>>>>>>>>>>> the partition itself feels a bit unnatural.
>>>>>>>>>>> 
>>>>>>>>>>> After all, in this scenario, users don't really care whether a
>>>>>>> partition
>>>>>>>>>> is
>>>>>>>>>>> newly expanded or not. Their only expectation is that they won't
>>>>>>> silently
>>>>>>>>>>> lose any live records produced to the topic during their active
>>>>>>>>>> consumption.
>>>>>>>>>>> 
>>>>>>>>>>> Best, Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>>>>>> 
>>>>>>>>>>>> Hello Jun,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the feedback, I have updated the KIP motivation
>>>> section.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12
>>> 寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the reply. I think we need a stronger motivation
>>> for
>>>>>> the
>>>>>>>>>> KIP.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The KIP says "The core insight is that not all partitions
>>> without
>>>>>> a
>>>>>>>>>>>>> committed offset are the same. A newly expanded partition
>>> (hot)
>>>> is
>>>>>>>>>>>>> fundamentally different from a partition the consumer has
>>> never
>>>>>> seen
>>>>>>>>>>>>> because it predates the group (cold)." Why is the hot
>>> partition
>>>>>>>>>>>>> fundamentally different from the cold?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The KIP says "The existing by_duration policy is also
>>>> insufficient
>>>>>>>>>>>> because:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> - The calculated seek time (now() - duration) varies across
>>> nodes
>>>>>>> due
>>>>>>>>>>>> to
>>>>>>>>>>>>> clock skew. To be safe, users must set an overly large
>>> duration,
>>>>>>>>>>>> causing
>>>>>>>>>>>>> unnecessary reprocessing.
>>>>>>>>>>>>> - On network errors, the client recalculates the seek time on
>>>>>> retry,
>>>>>>>>>>>>> shifting the target timestamp forward and risking data loss."
>>>>>>>>>>>>> 
>>>>>>>>>>>>> However, both of these situations are rare. If these issues
>>>>>> persist,
>>>>>>>>>> more
>>>>>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
>>>>>> need a
>>>>>>>>>>>> common
>>>>>>>>>>>>> solution. If users care about those rare situations, they can
>>>>>>> implement
>>>>>>>>>>>>> customized logic using
>>>>>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello chia,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>> always
>>>> be
>>>>>>>>>>>> greater
>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>> field
>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The earliest point at which we can detect the version
>>> mismatch
>>>> is
>>>>>>>>>> during
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> first metadata fetch after assignment, which occurs inside
>>>>>> poll().
>>>>>>>>>>>>>> Therefore, the
>>>>>>>>>>>>>> user would encounter an UnsupportedVersionException from
>>> poll().
>>>>>>> I’ll
>>>>>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50
>>> 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> hi Jiunn
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>>> partition
>>>>>> in
>>>>>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>>>>>> broker_current_time
>>>>>>>>>>>> -
>>>>>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
>>>>>> support
>>>>>>>>>> this
>>>>>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> If the creation time exists, the returned value should
>>> always
>>>> be
>>>>>>>>>>>> greater
>>>>>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>>>>>> field
>>>>>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
>>>>>>>>>> `poll()`?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
>>>>>>> discussion
>>>>>>>>>>>>>>>> feedback.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> The updated design decouples the new-partition reset
>>> behavior
>>>>>>> from
>>>>>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> - auto.offset.reset.max.age.ms
>>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>>> now applies to all
>>>>>>> auto.offset.reset
>>>>>>>>>>>>>> values
>>>>>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>>>>>>>>> auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>> .partitions
>>>>>>>>>>>>>>>> config setting
>>>>>>>>>>>>>>>> - For existing ("cold") partitions, the base
>>> auto.offset.reset
>>>>>>>>>> policy
>>>>>>>>>>>>>> continues
>>>>>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>>>>>> - The new-partition reset behavior is represented by a
>>>> separate
>>>>>>>>>>>>>> internal config
>>>>>>>>>>>>>>>> (auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions,
>>> currently fixed to
>>>>>> earliest).
>>>>>>>>>> This
>>>>>>>>>>>>>> decoupled design makes
>>>>>>>>>>>>>>>> it straightforward to promote the behavior to a public
>>>>>>> user-facing
>>>>>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
>>> 寫道:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
>>>> below:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 1) Add auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>> with a default value
>>>>>> of
>>>>>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
>>>>>> latest,
>>>>>>> and
>>>>>>>>>>>> it
>>>>>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>>>>>>>>>> 2) Mark auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>> as an internal
>>>>>>>>>>>>>> configuration. auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>> .partitions=earliest
>>>> already
>>>>>>>>>>>>>> addresses the issue, and we can discuss the use cases of
>>> other
>>>>>>> values
>>>>>>>>>>>> in a
>>>>>>>>>>>>>> separate KIP.
>>>>>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>> and
>>>>>>>>>>>>>> auto.offset.reset.latest.max.age.ms
>>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>,
>>> will be applied to all for
>>>>>>>>>>>>>> consistency.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
>>>>>> topic
>>>>>>> is
>>>>>>>>>>>>>> expanded
>>>>>>>>>>>>>>>>>> with new partitions, consumers using the latest auto
>>> offset
>>>>>>> reset
>>>>>>>>>>>>>> policy
>>>>>>>>>>>>>>>>>> will silently miss all records produced to those
>>> partitions
>>>>>>> before
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
>>>>>> issue
>>>>>>>>>> could
>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>>> will
>>>> take
>>>>>>> the
>>>>>>>>>>>> same
>>>>>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
>>>>>>> by_duration if
>>>>>>>>>>>>>> needed.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>>>>>> chia7712@apache.org
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
>>>>>>> important
>>>>>>>>>>>> from
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
>>> with
>>>>>> a
>>>>>>> few
>>>>>>>>>>>>>> questions:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic
>>> to
>>>>>>> other
>>>>>>>>>>>> reset
>>>>>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
>>>>>> by_duration
>>>>>>>>>>>> don't
>>>>>>>>>>>>>> seem
>>>>>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
>>>>>>> partition
>>>>>>>>>> is
>>>>>>>>>>>>>> expanded.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>>>>>>>>>> auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions?
>>> If they set it to
>>>>>> earliest
>>>>>>> or
>>>>>>>>>>>>>> latest,
>>>>>>>>>>>>>>>>>>> we might run into the exact same edge cases. For
>>> example,
>>>>>> if a
>>>>>>>>>>>>>> consumer is
>>>>>>>>>>>>>>>>>>> offline for a while and a new partition is created
>>> during
>>>>>> that
>>>>>>>>>>>>>> downtime,
>>>>>>>>>>>>>>>>>>> the user might actually want to skip to latest when
>>>>>> resuming,
>>>>>>>>>>>> rather
>>>>>>>>>>>>>> than
>>>>>>>>>>>>>>>>>>> reading from earliest just because the partition is
>>>>>>> technically
>>>>>>>>>>>>>> "new" to
>>>>>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
>>>>>>> threshold.
>>>>>>>>>>>> It
>>>>>>>>>>>>>> gives
>>>>>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
>>>> "hot/new"
>>>>>>> and
>>>>>>>>>>>>>> what is
>>>>>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
>>>>>> only
>>>>>>>>>>>>>> applies when
>>>>>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>>>>>>>>>> motivation
>>>>>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
>>> like
>>>>>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>> intention is that we want to have a separate way to
>>>> control
>>>>>>>>>> newly
>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>>>> partitions vs existing partitions when the group
>>> starts.
>>>>>>> Have we
>>>>>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>>>> .partitions?
>>>>>>> If
>>>>>>>>>>>> this
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
>>> the
>>>>>>>>>> policy
>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly
>>> to
>>>>>>>>>> customize
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7133700@gmail.com
>>>> 
>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>>>>>> PartitionMetadata
>>>>>>>>>>>>>> struct
>>>>>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
>>>>>>> server-side
>>>>>>>>>>>> by
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>>>>>> MembershipManager
>>>>>>>>>>>>>> detects
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
>>>> the
>>>>>>>>>>>> affected
>>>>>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
>>>>>> topic
>>>>>>> ID
>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
>>>>>>> negotiated
>>>>>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
>>> all
>>>>>> and
>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
>>> latest
>>>>>>> and
>>>>>>>>>>>>>> risking
>>>>>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
>>>> 下午4:04
>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
>>>>>> resolution
>>>>>>>>>> from
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
>>>>>> plane
>>>>>>>>>>>> clean.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>>>> inter-broker
>>>>>>>>>> clock
>>>>>>>>>>>>>> skew.
>>>>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
>>>>>> truth
>>>>>>>>>> for
>>>>>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
>>>>>>> negligible.
>>>>>>>>>>>>>> Given
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
>>>> minutes
>>>>>>> or
>>>>>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
>>> won't
>>>>>>> impact
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
>>>>>> 2026年4月29日
>>>>>>>>>>>> 下午3:29
>>>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
>>>>>>> conversation
>>>>>>>>>>>> but I
>>>>>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
>>> the
>>>>>>> KIP
>>>>>>>>>>>>>> states
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
>>> is
>>>>>>> set.
>>>>>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>>> partition,
>>>>>> it
>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
>>> ages.
>>>> Is
>>>>>>> my
>>>>>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
>>> also
>>>>>>> explain
>>>>>>>>>>>>>> how it
>>>>>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
>>>>>> offsets,
>>>>>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
>>> for
>>>>>>> this
>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
>>>>>> simply
>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
>>>> about
>>>>>>> not
>>>>>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
>>>> concepts.
>>>>>>> The
>>>>>>>>>>>> API
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>>>> partitions.
>>>>>>> The
>>>>>>>>>>>> fact
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
>>> API
>>>>>>> also
>>>>>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if
>>> we
>>>>>>> want to
>>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
>>>> via
>>>>>>> dev
>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
>>> the
>>>>>>> gap.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>>>>>> RequestPartitionAges
>>>>>>>>>>>>>> field.
>>>>>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges
>>> on
>>>>>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
>>>>>> value
>>>>>>> or
>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>>>>>> metadata.max.age.ms
>>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$>
>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
>>>>>> Streams,
>>>>>>>>>>>> would
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>>>>>> to add this new config
>>>>>> auto.offset.reset.latest.max.age
>>>>>>> to
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
>>> the
>>>>>>> most
>>>>>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
>>>>>>> configure
>>>>>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
>>> back
>>>>>> to
>>>>>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
>>> which
>>>>>>> isn't
>>>>>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
>>>> the
>>>>>>>>>>>> initial
>>>>>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest
>>> was
>>>>>>> applied
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
>>>>>>> unchanged.
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>>>>>> s7133700@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>>> 2026年4月28日
>>>>>>> 晚上8:03
>>>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
>>>> time
>>>>>>> via
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
>>> diagnose
>>>>>> and
>>>>>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
>>>>>>> Prevent
>>>>>>>>>> Hot
>>>>>>>>>>>>>> Data
>>>>>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
>>>>>> (hot)
>>>>>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>>>> automatically
>>>>>>> fall
>>>>>>>>>>>>>> back
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
>>>>>>> historical
>>>>>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping,

Thanks for the reply.

I agree that it's probably useful to allow a user to configure a different
offset policy for existing partitions vs new partitions. However, using
group creation time to capture that seems more intuitive. Here is another
proposal: remove auto.offset.reset.max.age.ms and categorize new partitions
based on group creation time. Introduce
a new config auto.offset.reset.new.partitions whose values can be earliest,
latest and by_duration, the same as auto.offset.reset. Users can set
`auto.offset.reset.new.partitions` to `earliest` if they want to guarantee
no data loss on new partitions. They can also use by_duration to set an
upper bound on the backlog replayed, which can be different from that of
the existing partitions. This will address your concern about too much
backlog being replayed when the offsets are lost. What do you think?

Jun


On Mon, Jun 15, 2026 at 10:39 AM Chia-Ping Tsai <ch...@gmail.com> wrote:

> hi Jun
>
> The most important part of this story is how users should expect the data
> they can see when using the latest or by_duration policy with expanded
> partitions.
>
> Yes, the by_duration policy can minimize data loss, but it is
> non-deterministic, which means users will either read too many historical
> records from existing partitions or lose some records from expanded
> partitions.
>
> Also, I agree that auto.offset.reset.max.age.ms
> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
> is a bit hard to understand, and that is why I preferred having a whole new
> policy based entirely on group creation time (KIP-1282)
>
> Best,
> Chia-Ping
>
> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:
>
>> Hi, Chia-Ping and Jiunn-Yang,
>>
>> Thanks for the reply. I am still trying to understand the value of the new
>> configs with the KIP.
>>
>> The motivation of the KIP is that a user doesn't want to miss the data if
>> the backlog is small. The backlog of the existing partition is easy to
>> understand because it relates to retention time. The backlog for the new
>> partition is a bit subtle to understand since it depends on the metadata
>> refresh delay. To set auto.offset.reset.max.age.ms
>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>,
>> the user needs to
>> understand the metadata refresh delay on the consumer side and use it to
>> set the config.
>>
>> Now, let's consider the alternative: setting the same value for the
>> existing by_duration policy. The KIP lists three issues with this
>> approach.
>> 1. It computes the seek target client-side as now() - duration, which
>> introduces clock skew across consumers and forces operators to choose
>> overly large durations, causing unnecessary reprocessing.
>> 2. The target timestamp is recomputed on each retry, so failed
>> ListOffsetsRequest retries can shift the target forward and potentially
>> miss records produced between attempts.
>> 3. It applies uniformly to all partitions without committed offsets, and
>> cannot distinguish newly expanded partitions from long-existing partitions
>> newly assigned to the group, leading to unnecessary replay.
>>
>> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
>> the metadata refresh delay. We could also consider improving the
>> implementation. For issue 3, the metadata refresh delay is typically low
>> (in the order of minutes with the classic consumer and tens of seconds
>> with
>> the new consumer). If a user is ok with reading that much backlog for new
>> partitions, it seems they will be ok doing the same for existing
>> partitions.
>>
>> So, instead of introducing a new config, could we just reuse the existing
>> config with better documentation and/or implementation?
>>
>> Jun
>>
>>
>> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
>>
>> > Hello Jun,
>> >
>> > You're right that group creation time is the more intuitive answer at
>> > first glance,
>> > the KIP's own motivation talks about partitions that "predate the group"
>> > vs partitions
>> > "created during group runtime," which directly points to a
>> group-lifecycle
>> > classifier.
>> > I'd like to walk through why we landed on partition age, and the
>> > trade-offs we considered.
>> >
>> > We evaluated three candidate signals:
>> >
>> > 1. `by_duration:5secs`
>> >
>> > This covers the metadata blindness window, but has issues the KIP
>> > currently documents
>> > under "Why not use `by_duration`?":
>> >
>> > - Client-side `now() - duration` introduces clock skew across consumers.
>> > - `ListOffsets` retries shift the target forward, potentially missing
>> > records produced between
>> > attempts.
>> > - It applies uniformly to all partitions without committed offsets,
>> > including pre-existing partitions
>> > newly assigned to the group, causing unnecessary replay.
>> >
>> > 2. Group creation time as classifier
>> >
>> > This works cleanly when the consumer is actively running. Our concern
>> > is the idle / late-rejoin case:
>> >
>> > T=0:         Group created.
>> > T=1..T=100:  Consumer idle (down, disconnected, etc.).
>> > T=50:        Partition added during the idle window.
>> > T=100:       Consumer resumes.
>> >
>> > Under group creation time, the new partition is classified as new
>> > (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
>> > But during `[T=1, T=100]`, base partitions also accumulated data that
>> > the consumer accepts as lost — that is precisely the contract of
>> > `auto.offset.reset=latest`. There is no principled reason to treat
>> > the new partition differently; both contain backlog accumulated during
>> > the same idle window.
>> >
>> > This aligns with the "backlog is backlog” principle you raised in
>> > the KIP-1282 thread: a `latest` user has tolerated some backlog on
>> > every other partition during the same idle period; forcing 0-backlog
>> > tolerance only on new partitions would be inconsistent with that
>> > tolerance.
>> >
>> > 3. Partition age vs threshold
>> >
>> > Partition age corresponds to the actual silent data loss window,
>> > the gap between partition creation and the consumer’s metadata
>> > refresh. Within this window, data loss is genuinely silent: the
>> > consumer had no opportunity to know about the partition. Outside this
>> > window, missing data reflects either:
>> >
>> > - (a) the user’s tolerated cost of running with idle consumers, or
>> > - (b) an operational issue to surface via monitoring, not via reset
>> policy.
>> >
>> > We did not choose partition age because it is more elegant than group
>> > creation time — we chose it because its failure mode (requires a
>> > threshold) is
>> > less invasive than the failure mode of group creation time (overrides
>> > user-stated
>> > `latest` intent during idle periods).
>> >
>> > Best Regards,
>> > Jiunn-Yang
>> >
>> > > Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
>> > >
>> > > Hi Jun,
>> > >
>> > > Relying on both creation times will create an inconsistent scenario. A
>> > > consumer that lost all offsets due to a long sleep will seek to the
>> > > beginning for the partitions created later than the group.
>> > >
>> > > That is why we initially proposed KIP-1282 to fix the inconsistency
>> > using a
>> > > whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
>> > goes
>> > > back to using flexible configurations to prevent users from falling
>> into
>> > > that pitfall.
>> > >
>> > > Best, Chia-Ping
>> > >
>> > > Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
>> > >
>> > >> Hi, Jiunn-Yang,
>> > >>
>> > >> Thanks for the reply and sorry for the late reply.
>> > >>
>> > >> JR1. The design of auto.offset.reset.max.age.ms
>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>> still feels weird to
>> > me.
>> > >> It
>> > >> categorizes partitions as new or existing based on the partition
>> > creation
>> > >> time. Intuitively, the categorization should be based on the group
>> > creation
>> > >> time: all partitions existing when the group is created are existing
>> and
>> > >> all partitions created after the group creation are new partitions.
>> > >>
>> > >> Jun
>> > >>
>> > >>
>> > >>
>> > >> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>> > >>
>> > >>> Hi all,
>> > >>>
>> > >>> Manually bumping this thread. If there is no further
>> > >>> discussion, I will close the vote.
>> > >>>
>> > >>> Best Regards,
>> > >>> Jiunn-Yang
>> > >>>
>> > >>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>> > >>>>
>> > >>>> Hello Jian,
>> > >>>>
>> > >>>> Thanks for your feedback,
>> > >>>>
>> > >>>> Agreed, partition expansion is a common operational task, not an
>> edge
>> > >>>> case. I've updated the Motivation section accordingly.
>> > >>>>
>> > >>>> Best Regards,
>> > >>>> Jiunn-Yang
>> > >>>>
>> > >>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>> > >>>>>
>> > >>>>> Hi Jiunn-Yang:
>> > >>>>>
>> > >>>>> Thanks for the KIP. I think it would be useful to clarify that
>> this
>> > >> is a
>> > >>>>> common scenario rather than an edge case, which further
>> demonstrates
>> > >> the
>> > >>>>> need for this optimization. For example:
>> > >>>>> A partition expansion is a common operational task in Kafka: To
>> > >> balance
>> > >>>>> resource utilization and cost, topics are typically created with a
>> > >>> moderate
>> > >>>>> default partition count. However, as traffic grows over time, it
>> is
>> > >>> often
>> > >>>>> necessary to increase the number of partitions to accommodate the
>> > >> higher
>> > >>>>> workload.
>> > >>>>>
>> > >>>>> Regards
>> > >>>>> Jian
>> > >>>>>
>> > >>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>> > >>>>>
>> > >>>>>> Hello chia,
>> > >>>>>>
>> > >>>>>> Thanks for the comments, I have updated the KIP!
>> > >>>>>>
>> > >>>>>> Best Regards,
>> > >>>>>> Jiunn-Yang
>> > >>>>>>
>> > >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>> > >>>>>>>
>> > >>>>>>> Hi Jiunn-Yang,
>> > >>>>>>>
>> > >>>>>>> Would you mind removing the terms "hot" and "cold" when
>> describing
>> > >>>>>>> partitions in the KIP? I understand you are using them to
>> describe
>> > >> the
>> > >>>>>>> "freshness" or the users' need for the records, but applying
>> these
>> > >>> terms
>> > >>>>>> to
>> > >>>>>>> the partition itself feels a bit unnatural.
>> > >>>>>>>
>> > >>>>>>> After all, in this scenario, users don't really care whether a
>> > >>> partition
>> > >>>>>> is
>> > >>>>>>> newly expanded or not. Their only expectation is that they won't
>> > >>> silently
>> > >>>>>>> lose any live records produced to the topic during their active
>> > >>>>>> consumption.
>> > >>>>>>>
>> > >>>>>>> Best, Chia-Ping
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>> > >>>>>>>
>> > >>>>>>>> Hello Jun,
>> > >>>>>>>>
>> > >>>>>>>> Thanks for the feedback, I have updated the KIP motivation
>> > section.
>> > >>>>>>>>
>> > >>>>>>>> Best Regards,
>> > >>>>>>>> Jiunn-Yang
>> > >>>>>>>>
>> > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12
>> 寫道:
>> > >>>>>>>>>
>> > >>>>>>>>> Hi, Jiunn-Yang,
>> > >>>>>>>>>
>> > >>>>>>>>> Thanks for the reply. I think we need a stronger motivation
>> for
>> > >> the
>> > >>>>>> KIP.
>> > >>>>>>>>>
>> > >>>>>>>>> The KIP says "The core insight is that not all partitions
>> without
>> > >> a
>> > >>>>>>>>> committed offset are the same. A newly expanded partition
>> (hot)
>> > is
>> > >>>>>>>>> fundamentally different from a partition the consumer has
>> never
>> > >> seen
>> > >>>>>>>>> because it predates the group (cold)." Why is the hot
>> partition
>> > >>>>>>>>> fundamentally different from the cold?
>> > >>>>>>>>>
>> > >>>>>>>>> The KIP says "The existing by_duration policy is also
>> > insufficient
>> > >>>>>>>> because:
>> > >>>>>>>>>
>> > >>>>>>>>> - The calculated seek time (now() - duration) varies across
>> nodes
>> > >>> due
>> > >>>>>>>> to
>> > >>>>>>>>> clock skew. To be safe, users must set an overly large
>> duration,
>> > >>>>>>>> causing
>> > >>>>>>>>> unnecessary reprocessing.
>> > >>>>>>>>> - On network errors, the client recalculates the seek time on
>> > >> retry,
>> > >>>>>>>>> shifting the target timestamp forward and risking data loss."
>> > >>>>>>>>>
>> > >>>>>>>>> However, both of these situations are rare. If these issues
>> > >> persist,
>> > >>>>>> more
>> > >>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
>> > >> need a
>> > >>>>>>>> common
>> > >>>>>>>>> solution. If users care about those rare situations, they can
>> > >>> implement
>> > >>>>>>>>> customized logic using
>> > >>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>> > >>>>>>>>>
>> > >>>>>>>>> Jun
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> Hello chia,
>> > >>>>>>>>>>
>> > >>>>>>>>>> Thanks for the feedback,
>> > >>>>>>>>>>
>> > >>>>>>>>>>> If the creation time exists, the returned value should
>> always
>> > be
>> > >>>>>>>> greater
>> > >>>>>>>>>> than or equal to zero, right?
>> > >>>>>>>>>> I have explicitly mentioned this in the KIP.
>> > >>>>>>>>>>
>> > >>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>> > >>> field
>> > >>>>>>>>>> absent    UnsupportedVersionException
>> > >>>>>>>>>>
>> > >>>>>>>>>> The earliest point at which we can detect the version
>> mismatch
>> > is
>> > >>>>>> during
>> > >>>>>>>>>> the
>> > >>>>>>>>>> first metadata fetch after assignment, which occurs inside
>> > >> poll().
>> > >>>>>>>>>> Therefore, the
>> > >>>>>>>>>> user would encounter an UnsupportedVersionException from
>> poll().
>> > >>> I’ll
>> > >>>>>>>>>> clarify this in the KIP.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Best Regards,
>> > >>>>>>>>>> Jiunn-Yang
>> > >>>>>>>>>>
>> > >>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50
>> 寫道:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> hi Jiunn
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
>> partition
>> > >> in
>> > >>>>>>>>>> milliseconds, computed server-side by the broker as
>> > >>>>>> broker_current_time
>> > >>>>>>>> -
>> > >>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
>> > >> support
>> > >>>>>> this
>> > >>>>>>>>>> feature or the partition creation time is unknown.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> If the creation time exists, the returned value should
>> always
>> > be
>> > >>>>>>>> greater
>> > >>>>>>>>>> than or equal to zero, right?
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>> > >>> field
>> > >>>>>>>>>> absent    UnsupportedVersionException
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
>> > >>>>>> `poll()`?
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> Best,
>> > >>>>>>>>>>> Chia-Ping
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>> > >>>>>>>>>>>> Hello Jun, chia,
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
>> > >>> discussion
>> > >>>>>>>>>>>> feedback.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> The updated design decouples the new-partition reset
>> behavior
>> > >>> from
>> > >>>>>>>>>>>> the base auto.offset.reset policy:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> - auto.offset.reset.max.age.ms
>> <https://urldefense.com/v3/__http://auto.offset.reset.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpwnwknH$>
>> now applies to all
>> > >>> auto.offset.reset
>> > >>>>>>>>>> values
>> > >>>>>>>>>>>> (latest, earliest, by_duration, none).
>> > >>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>> > >>>>>>>>>> auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>> .partitions
>> > >>>>>>>>>>>> config setting
>> > >>>>>>>>>>>> - For existing ("cold") partitions, the base
>> auto.offset.reset
>> > >>>>>> policy
>> > >>>>>>>>>> continues
>> > >>>>>>>>>>>> to apply unchanged.
>> > >>>>>>>>>>>> - The new-partition reset behavior is represented by a
>> > separate
>> > >>>>>>>>>> internal config
>> > >>>>>>>>>>>> (auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions,
>> currently fixed to
>> > >> earliest).
>> > >>>>>> This
>> > >>>>>>>>>> decoupled design makes
>> > >>>>>>>>>>>> it straightforward to promote the behavior to a public
>> > >>> user-facing
>> > >>>>>>>>>> configuration in a future KIP.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>> Best Regards,
>> > >>>>>>>>>>>> Jiunn-Yang
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
>> 寫道:
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> hi Jun
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
>> > below:
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> 1) Add auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>> with a default value
>> > >> of
>> > >>>>>>>>>> earliest. It fixes the data loss from both by_duration and
>> > >> latest,
>> > >>> and
>> > >>>>>>>> it
>> > >>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
>> > >>>>>>>>>>>>> 2) Mark auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>> as an internal
>> > >>>>>>>>>> configuration. auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>> .partitions=earliest
>> > already
>> > >>>>>>>>>> addresses the issue, and we can discuss the use cases of
>> other
>> > >>> values
>> > >>>>>>>> in a
>> > >>>>>>>>>> separate KIP.
>> > >>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>> and
>> > >>>>>>>>>> auto.offset.reset.latest.max.age.ms
>> <https://urldefense.com/v3/__http://auto.offset.reset.latest.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfu9JSP4l$>,
>> will be applied to all for
>> > >>>>>>>>>> consistency.
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> WDYT?
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>> > >>>>>>>>>>>>>> Hi, Chia-Ping,
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Thanks for the reply.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
>> > >> topic
>> > >>> is
>> > >>>>>>>>>> expanded
>> > >>>>>>>>>>>>>> with new partitions, consumers using the latest auto
>> offset
>> > >>> reset
>> > >>>>>>>>>> policy
>> > >>>>>>>>>>>>>> will silently miss all records produced to those
>> partitions
>> > >>> before
>> > >>>>>>>> the
>> > >>>>>>>>>>>>>> consumer discovers them.". If a user sets
>> > >>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
>> > >> issue
>> > >>>>>> could
>> > >>>>>>>>>> also
>> > >>>>>>>>>>>>>> happen, right?
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions
>> will
>> > take
>> > >>> the
>> > >>>>>>>> same
>> > >>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
>> > >>> by_duration if
>> > >>>>>>>>>> needed.
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> Jun
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>> > >>>>>> chia7712@apache.org
>> > >>>>>>>>>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> hi Jun
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
>> > >>> important
>> > >>>>>>>> from
>> > >>>>>>>>>> your
>> > >>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
>> with
>> > >> a
>> > >>> few
>> > >>>>>>>>>> questions:
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic
>> to
>> > >>> other
>> > >>>>>>>> reset
>> > >>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
>> > >> by_duration
>> > >>>>>>>> don't
>> > >>>>>>>>>> seem
>> > >>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
>> > >>> partition
>> > >>>>>> is
>> > >>>>>>>>>> expanded.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> 2. What values would we expect users to configure for
>> > >>>>>>>>>>>>>>> auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>.partitions?
>> If they set it to
>> > >> earliest
>> > >>> or
>> > >>>>>>>>>> latest,
>> > >>>>>>>>>>>>>>> we might run into the exact same edge cases. For
>> example,
>> > >> if a
>> > >>>>>>>>>> consumer is
>> > >>>>>>>>>>>>>>> offline for a while and a new partition is created
>> during
>> > >> that
>> > >>>>>>>>>> downtime,
>> > >>>>>>>>>>>>>>> the user might actually want to skip to latest when
>> > >> resuming,
>> > >>>>>>>> rather
>> > >>>>>>>>>> than
>> > >>>>>>>>>>>>>>> reading from earliest just because the partition is
>> > >>> technically
>> > >>>>>>>>>> "new" to
>> > >>>>>>>>>>>>>>> the group.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
>> > >>> threshold.
>> > >>>>>>>> It
>> > >>>>>>>>>> gives
>> > >>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
>> > "hot/new"
>> > >>> and
>> > >>>>>>>>>> what is
>> > >>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> Best,
>> > >>>>>>>>>>>>>>> Chia-Ping
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>> > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> Thanks for the KIP.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
>> > >> only
>> > >>>>>>>>>> applies when
>> > >>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>> > >>>>>> motivation
>> > >>>>>>>>>>>>>>> equally
>> > >>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
>> like
>> > >>>>>>>>>> by_duration.
>> > >>>>>>>>>>>>>>> The
>> > >>>>>>>>>>>>>>>> intention is that we want to have a separate way to
>> > control
>> > >>>>>> newly
>> > >>>>>>>>>> created
>> > >>>>>>>>>>>>>>>> partitions vs existing partitions when the group
>> starts.
>> > >>> Have we
>> > >>>>>>>>>>>>>>> considered
>> > >>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
>> <https://urldefense.com/v3/__http://auto.offset.reset.new__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkfpDMjdw3$>
>> > .partitions?
>> > >>> If
>> > >>>>>>>> this
>> > >>>>>>>>>> new
>> > >>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
>> the
>> > >>>>>> policy
>> > >>>>>>>>>> used
>> > >>>>>>>>>>>>>>> for
>> > >>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly
>> to
>> > >>>>>> customize
>> > >>>>>>>>>> the
>> > >>>>>>>>>>>>>>>> behavior for new partitions.
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> Jun
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7133700@gmail.com
>> >
>> > >>> wrote:
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> Hi all,
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>> Best Regards,
>> > >>>>>>>>>>>>>>>>> Jiunn-Yang
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> Hello all,
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> Thanks for the feedback.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> DJ01/DJ02:
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>> > >>> PartitionMetadata
>> > >>>>>>>>>> struct
>> > >>>>>>>>>>>>>>>>> gains a new
>> > >>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
>> > >>> server-side
>> > >>>>>>>> by
>> > >>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>> broker as
>> > >>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>> > >>> MembershipManager
>> > >>>>>>>>>> detects
>> > >>>>>>>>>>>>>>> a
>> > >>>>>>>>>>>>>>>>> newly assigned
>> > >>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
>> > the
>> > >>>>>>>> affected
>> > >>>>>>>>>>>>>>> topic
>> > >>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>> > >>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
>> > >> topic
>> > >>> ID
>> > >>>>>> is
>> > >>>>>>>>>>>>>>> already
>> > >>>>>>>>>>>>>>>>> in the cache.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> MB0:
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>> > >>>>>>>>>> MetadataResponse
>> > >>>>>>>>>>>>>>>>> version via the
>> > >>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
>> > >>> negotiated
>> > >>>>>>>>>>>>>>> version is
>> > >>>>>>>>>>>>>>>>> unsupported, the consumer
>> > >>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
>> all
>> > >> and
>> > >>>>>> can
>> > >>>>>>>>>>>>>>> throw an
>> > >>>>>>>>>>>>>>>>> UnsupportedVersionException
>> > >>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
>> latest
>> > >>> and
>> > >>>>>>>>>> risking
>> > >>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>> Best Regards,
>> > >>>>>>>>>>>>>>>>>> Jiunn-Yang
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
>> > 下午4:04
>> > >>> 寫道:
>> > >>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>> hi David
>> > >>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
>> > >> resolution
>> > >>>>>> from
>> > >>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
>> > >> plane
>> > >>>>>>>> clean.
>> > >>>>>>>>>> The
>> > >>>>>>>>>>>>>>> main
>> > >>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
>> > inter-broker
>> > >>>>>> clock
>> > >>>>>>>>>> skew.
>> > >>>>>>>>>>>>>>> The
>> > >>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
>> > >> truth
>> > >>>>>> for
>> > >>>>>>>>>> time.
>> > >>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
>> > >>> negligible.
>> > >>>>>>>>>> Given
>> > >>>>>>>>>>>>>>> that
>> > >>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
>> > minutes
>> > >>> or
>> > >>>>>>>>>> hours, a
>> > >>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
>> won't
>> > >>> impact
>> > >>>>>>>> the
>> > >>>>>>>>>>>>>>>>> fallback decision.
>> > >>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>> Best,
>> > >>>>>>>>>>>>>>>>>>> Chia-Ping
>> > >>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
>> > >> 2026年4月29日
>> > >>>>>>>> 下午3:29
>> > >>>>>>>>>> 寫道:
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> Hi all,
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
>> > >>> conversation
>> > >>>>>>>> but I
>> > >>>>>>>>>>>>>>> took a
>> > >>>>>>>>>>>>>>>>>>>> quick look at this one.
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>> > >>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>> > >>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
>> the
>> > >>> KIP
>> > >>>>>>>>>> states
>> > >>>>>>>>>>>>>>> that
>> > >>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
>> is
>> > >>> set.
>> > >>>>>>>>>>>>>>> Implicitly,
>> > >>>>>>>>>>>>>>>>> this
>> > >>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
>> partition,
>> > >> it
>> > >>>>>> will
>> > >>>>>>>>>>>>>>> issue a
>> > >>>>>>>>>>>>>>>>> new
>> > >>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the
>> ages.
>> > Is
>> > >>> my
>> > >>>>>>>>>>>>>>>>> understanding
>> > >>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and
>> also
>> > >>> explain
>> > >>>>>>>>>> how it
>> > >>>>>>>>>>>>>>>>> fits
>> > >>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
>> > >> offsets,
>> > >>>>>>>> etc.).
>> > >>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>> > >>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
>> for
>> > >>> this
>> > >>>>>>>> given
>> > >>>>>>>>>>>>>>> that
>> > >>>>>>>>>>>>>>>>> a new
>> > >>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
>> > >> simply
>> > >>>>>>>>>> include
>> > >>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
>> > about
>> > >>> not
>> > >>>>>>>>>>>>>>> overloading
>> > >>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
>> > concepts.
>> > >>> The
>> > >>>>>>>> API
>> > >>>>>>>>>> is
>> > >>>>>>>>>>>>>>> a
>> > >>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
>> > partitions.
>> > >>> The
>> > >>>>>>>> fact
>> > >>>>>>>>>>>>>>> that
>> > >>>>>>>>>>>>>>>>> we
>> > >>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
>> API
>> > >>> also
>> > >>>>>>>>>> suggests
>> > >>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if
>> we
>> > >>> want to
>> > >>>>>>>>>>>>>>> support
>> > >>>>>>>>>>>>>>>>>>>> Streams in the future?
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>> Best,
>> > >>>>>>>>>>>>>>>>>>>> David
>> > >>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
>> > via
>> > >>> dev
>> > >>>>>> <
>> > >>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
>> the
>> > >>> gap.
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>> > >>> RequestPartitionAges
>> > >>>>>>>>>> field.
>> > >>>>>>>>>>>>>>> Can a
>> > >>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges
>> on
>> > >>>>>>>>>>>>>>> TopicPartitions)
>> > >>>>>>>>>>>>>>>>> be
>> > >>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
>> > >> value
>> > >>> or
>> > >>>>>> a
>> > >>>>>>>>>>>>>>> ConfigDef
>> > >>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>> > >>>>>>>> metadata.max.age.ms
>> <https://urldefense.com/v3/__http://metadata.max.age.ms__;!!Ayb5sqE7!ryUSIElKDF-DJJHgYwYXwp4XEBXpXuBOnZd18PJoMNH4LZ1gc-pDbbdfb2eme_dRSvdvI3bkflKEb5SK$>
>> > >>>>>>>>>> ?
>> > >>>>>>>>>>>>>>>>> Sizing
>> > >>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
>> > >> Streams,
>> > >>>>>>>> would
>> > >>>>>>>>>> it
>> > >>>>>>>>>>>>>>> be
>> > >>>>>>>>>>>>>>>>> better
>> > >>>>>>>>>>>>>>>>>>>>> to add this new config
>> > >> auto.offset.reset.latest.max.age
>> > >>> to
>> > >>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>> > >>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>> > >>>>>>>>>>>>>>>>> for a
>> > >>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
>> the
>> > >>> most
>> > >>>>>>>>>>>>>>> familiar
>> > >>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
>> > >>> configure
>> > >>>>>>>>>> it. Or
>> > >>>>>>>>>>>>>>>>> may be
>> > >>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
>> back
>> > >> to
>> > >>>>>>>>>> earliest"
>> > >>>>>>>>>>>>>>>>> reads as
>> > >>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
>> which
>> > >>> isn't
>> > >>>>>>>>>>>>>>> supported.
>> > >>>>>>>>>>>>>>>>> May
>> > >>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
>> > the
>> > >>>>>>>> initial
>> > >>>>>>>>>>>>>>>>> position to
>> > >>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest
>> was
>> > >>> applied
>> > >>>>>>>> to
>> > >>>>>>>>>>>>>>> that
>> > >>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
>> > >>> unchanged.
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>> Thanks,
>> > >>>>>>>>>>>>>>>>>>>>> Murali
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>> > >>> s7133700@gmail.com>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> Hi chia,
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
>> > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
>> 2026年4月28日
>> > >>> 晚上8:03
>> > >>>>>>>> 寫道:
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
>> > time
>> > >>> via
>> > >>>>>>>> the
>> > >>>>>>>>>>>>>>> Admin
>> > >>>>>>>>>>>>>>>>> API?
>> > >>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
>> diagnose
>> > >> and
>> > >>>>>>>>>>>>>>> troubleshoot
>> > >>>>>>>>>>>>>>>>> the
>> > >>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> Best,
>> > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>> > >>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>> > >>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>> > >>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
>> > >>> Prevent
>> > >>>>>> Hot
>> > >>>>>>>>>> Data
>> > >>>>>>>>>>>>>>>>> Loss
>> > >>>>>>>>>>>>>>>>>>>>> on
>> > >>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>> > >>>>>>>>>>>>>>>>>>>>>>>> <
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>
>> > >>
>> >
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>> > >>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>> > >>>>>>>>>>>>>>>>> a
>> > >>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>> > >>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
>> > >> (hot)
>> > >>>>>>>>>> partitions
>> > >>>>>>>>>>>>>>>>> from
>> > >>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>> > >>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
>> > automatically
>> > >>> fall
>> > >>>>>>>>>> back
>> > >>>>>>>>>>>>>>> to
>> > >>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>> > >>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
>> > >>> historical
>> > >>>>>>>>>>>>>>> reprocess.
>> > >>>>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>> > >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>
>> > >>>
>> > >>>
>> > >>
>> >
>> >
>>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@gmail.com>.
hi Jun

The most important part of this story is how users should expect the data
they can see when using the latest or by_duration policy with expanded
partitions.

Yes, the by_duration policy can minimize data loss, but it is
non-deterministic, which means users will either read too many historical
records from existing partitions or lose some records from expanded
partitions.

Also, I agree that auto.offset.reset.max.age.ms is a bit hard to
understand, and that is why I preferred having a whole new policy based
entirely on group creation time (KIP-1282)

Best,
Chia-Ping

Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月16日週二 上午1:08寫道:

> Hi, Chia-Ping and Jiunn-Yang,
>
> Thanks for the reply. I am still trying to understand the value of the new
> configs with the KIP.
>
> The motivation of the KIP is that a user doesn't want to miss the data if
> the backlog is small. The backlog of the existing partition is easy to
> understand because it relates to retention time. The backlog for the new
> partition is a bit subtle to understand since it depends on the metadata
> refresh delay. To set auto.offset.reset.max.age.ms, the user needs to
> understand the metadata refresh delay on the consumer side and use it to
> set the config.
>
> Now, let's consider the alternative: setting the same value for the
> existing by_duration policy. The KIP lists three issues with this approach.
> 1. It computes the seek target client-side as now() - duration, which
> introduces clock skew across consumers and forces operators to choose
> overly large durations, causing unnecessary reprocessing.
> 2. The target timestamp is recomputed on each retry, so failed
> ListOffsetsRequest retries can shift the target forward and potentially
> miss records produced between attempts.
> 3. It applies uniformly to all partitions without committed offsets, and
> cannot distinguish newly expanded partitions from long-existing partitions
> newly assigned to the group, leading to unnecessary replay.
>
> Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
> the metadata refresh delay. We could also consider improving the
> implementation. For issue 3, the metadata refresh delay is typically low
> (in the order of minutes with the classic consumer and tens of seconds with
> the new consumer). If a user is ok with reading that much backlog for new
> partitions, it seems they will be ok doing the same for existing
> partitions.
>
> So, instead of introducing a new config, could we just reuse the existing
> config with better documentation and/or implementation?
>
> Jun
>
>
> On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hello Jun,
> >
> > You're right that group creation time is the more intuitive answer at
> > first glance,
> > the KIP's own motivation talks about partitions that "predate the group"
> > vs partitions
> > "created during group runtime," which directly points to a
> group-lifecycle
> > classifier.
> > I'd like to walk through why we landed on partition age, and the
> > trade-offs we considered.
> >
> > We evaluated three candidate signals:
> >
> > 1. `by_duration:5secs`
> >
> > This covers the metadata blindness window, but has issues the KIP
> > currently documents
> > under "Why not use `by_duration`?":
> >
> > - Client-side `now() - duration` introduces clock skew across consumers.
> > - `ListOffsets` retries shift the target forward, potentially missing
> > records produced between
> > attempts.
> > - It applies uniformly to all partitions without committed offsets,
> > including pre-existing partitions
> > newly assigned to the group, causing unnecessary replay.
> >
> > 2. Group creation time as classifier
> >
> > This works cleanly when the consumer is actively running. Our concern
> > is the idle / late-rejoin case:
> >
> > T=0:         Group created.
> > T=1..T=100:  Consumer idle (down, disconnected, etc.).
> > T=50:        Partition added during the idle window.
> > T=100:       Consumer resumes.
> >
> > Under group creation time, the new partition is classified as new
> > (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
> > But during `[T=1, T=100]`, base partitions also accumulated data that
> > the consumer accepts as lost — that is precisely the contract of
> > `auto.offset.reset=latest`. There is no principled reason to treat
> > the new partition differently; both contain backlog accumulated during
> > the same idle window.
> >
> > This aligns with the "backlog is backlog” principle you raised in
> > the KIP-1282 thread: a `latest` user has tolerated some backlog on
> > every other partition during the same idle period; forcing 0-backlog
> > tolerance only on new partitions would be inconsistent with that
> > tolerance.
> >
> > 3. Partition age vs threshold
> >
> > Partition age corresponds to the actual silent data loss window,
> > the gap between partition creation and the consumer’s metadata
> > refresh. Within this window, data loss is genuinely silent: the
> > consumer had no opportunity to know about the partition. Outside this
> > window, missing data reflects either:
> >
> > - (a) the user’s tolerated cost of running with idle consumers, or
> > - (b) an operational issue to surface via monitoring, not via reset
> policy.
> >
> > We did not choose partition age because it is more elegant than group
> > creation time — we chose it because its failure mode (requires a
> > threshold) is
> > less invasive than the failure mode of group creation time (overrides
> > user-stated
> > `latest` intent during idle periods).
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> > >
> > > Hi Jun,
> > >
> > > Relying on both creation times will create an inconsistent scenario. A
> > > consumer that lost all offsets due to a long sleep will seek to the
> > > beginning for the partitions created later than the group.
> > >
> > > That is why we initially proposed KIP-1282 to fix the inconsistency
> > using a
> > > whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
> > goes
> > > back to using flexible configurations to prevent users from falling
> into
> > > that pitfall.
> > >
> > > Best, Chia-Ping
> > >
> > > Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> > >
> > >> Hi, Jiunn-Yang,
> > >>
> > >> Thanks for the reply and sorry for the late reply.
> > >>
> > >> JR1. The design of auto.offset.reset.max.age.ms still feels weird to
> > me.
> > >> It
> > >> categorizes partitions as new or existing based on the partition
> > creation
> > >> time. Intuitively, the categorization should be based on the group
> > creation
> > >> time: all partitions existing when the group is created are existing
> and
> > >> all partitions created after the group creation are new partitions.
> > >>
> > >> Jun
> > >>
> > >>
> > >>
> > >> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> Manually bumping this thread. If there is no further
> > >>> discussion, I will close the vote.
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > >>>>
> > >>>> Hello Jian,
> > >>>>
> > >>>> Thanks for your feedback,
> > >>>>
> > >>>> Agreed, partition expansion is a common operational task, not an
> edge
> > >>>> case. I've updated the Motivation section accordingly.
> > >>>>
> > >>>> Best Regards,
> > >>>> Jiunn-Yang
> > >>>>
> > >>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > >>>>>
> > >>>>> Hi Jiunn-Yang:
> > >>>>>
> > >>>>> Thanks for the KIP. I think it would be useful to clarify that this
> > >> is a
> > >>>>> common scenario rather than an edge case, which further
> demonstrates
> > >> the
> > >>>>> need for this optimization. For example:
> > >>>>> A partition expansion is a common operational task in Kafka: To
> > >> balance
> > >>>>> resource utilization and cost, topics are typically created with a
> > >>> moderate
> > >>>>> default partition count. However, as traffic grows over time, it is
> > >>> often
> > >>>>> necessary to increase the number of partitions to accommodate the
> > >> higher
> > >>>>> workload.
> > >>>>>
> > >>>>> Regards
> > >>>>> Jian
> > >>>>>
> > >>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > >>>>>
> > >>>>>> Hello chia,
> > >>>>>>
> > >>>>>> Thanks for the comments, I have updated the KIP!
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>> Jiunn-Yang
> > >>>>>>
> > >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> > >>>>>>>
> > >>>>>>> Hi Jiunn-Yang,
> > >>>>>>>
> > >>>>>>> Would you mind removing the terms "hot" and "cold" when
> describing
> > >>>>>>> partitions in the KIP? I understand you are using them to
> describe
> > >> the
> > >>>>>>> "freshness" or the users' need for the records, but applying
> these
> > >>> terms
> > >>>>>> to
> > >>>>>>> the partition itself feels a bit unnatural.
> > >>>>>>>
> > >>>>>>> After all, in this scenario, users don't really care whether a
> > >>> partition
> > >>>>>> is
> > >>>>>>> newly expanded or not. Their only expectation is that they won't
> > >>> silently
> > >>>>>>> lose any live records produced to the topic during their active
> > >>>>>> consumption.
> > >>>>>>>
> > >>>>>>> Best, Chia-Ping
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > >>>>>>>
> > >>>>>>>> Hello Jun,
> > >>>>>>>>
> > >>>>>>>> Thanks for the feedback, I have updated the KIP motivation
> > section.
> > >>>>>>>>
> > >>>>>>>> Best Regards,
> > >>>>>>>> Jiunn-Yang
> > >>>>>>>>
> > >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> > >>>>>>>>>
> > >>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>
> > >>>>>>>>> Thanks for the reply. I think we need a stronger motivation for
> > >> the
> > >>>>>> KIP.
> > >>>>>>>>>
> > >>>>>>>>> The KIP says "The core insight is that not all partitions
> without
> > >> a
> > >>>>>>>>> committed offset are the same. A newly expanded partition (hot)
> > is
> > >>>>>>>>> fundamentally different from a partition the consumer has never
> > >> seen
> > >>>>>>>>> because it predates the group (cold)." Why is the hot partition
> > >>>>>>>>> fundamentally different from the cold?
> > >>>>>>>>>
> > >>>>>>>>> The KIP says "The existing by_duration policy is also
> > insufficient
> > >>>>>>>> because:
> > >>>>>>>>>
> > >>>>>>>>> - The calculated seek time (now() - duration) varies across
> nodes
> > >>> due
> > >>>>>>>> to
> > >>>>>>>>> clock skew. To be safe, users must set an overly large
> duration,
> > >>>>>>>> causing
> > >>>>>>>>> unnecessary reprocessing.
> > >>>>>>>>> - On network errors, the client recalculates the seek time on
> > >> retry,
> > >>>>>>>>> shifting the target timestamp forward and risking data loss."
> > >>>>>>>>>
> > >>>>>>>>> However, both of these situations are rare. If these issues
> > >> persist,
> > >>>>>> more
> > >>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
> > >> need a
> > >>>>>>>> common
> > >>>>>>>>> solution. If users care about those rare situations, they can
> > >>> implement
> > >>>>>>>>> customized logic using
> > >>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> > >>>>>>>>>
> > >>>>>>>>> Jun
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hello chia,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the feedback,
> > >>>>>>>>>>
> > >>>>>>>>>>> If the creation time exists, the returned value should always
> > be
> > >>>>>>>> greater
> > >>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>> I have explicitly mentioned this in the KIP.
> > >>>>>>>>>>
> > >>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> > >>> field
> > >>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>
> > >>>>>>>>>> The earliest point at which we can detect the version mismatch
> > is
> > >>>>>> during
> > >>>>>>>>>> the
> > >>>>>>>>>> first metadata fetch after assignment, which occurs inside
> > >> poll().
> > >>>>>>>>>> Therefore, the
> > >>>>>>>>>> user would encounter an UnsupportedVersionException from
> poll().
> > >>> I’ll
> > >>>>>>>>>> clarify this in the KIP.
> > >>>>>>>>>>
> > >>>>>>>>>> Best Regards,
> > >>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>
> > >>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> > >>>>>>>>>>>
> > >>>>>>>>>>> hi Jiunn
> > >>>>>>>>>>>
> > >>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this
> partition
> > >> in
> > >>>>>>>>>> milliseconds, computed server-side by the broker as
> > >>>>>> broker_current_time
> > >>>>>>>> -
> > >>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
> > >> support
> > >>>>>> this
> > >>>>>>>>>> feature or the partition creation time is unknown.
> > >>>>>>>>>>>
> > >>>>>>>>>>> If the creation time exists, the returned value should always
> > be
> > >>>>>>>> greater
> > >>>>>>>>>> than or equal to zero, right?
> > >>>>>>>>>>>
> > >>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> > >>> field
> > >>>>>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>>>>
> > >>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
> > >>>>>> `poll()`?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Best,
> > >>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > >>>>>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
> > >>> discussion
> > >>>>>>>>>>>> feedback.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The updated design decouples the new-partition reset
> behavior
> > >>> from
> > >>>>>>>>>>>> the base auto.offset.reset policy:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> > >>> auto.offset.reset
> > >>>>>>>>>> values
> > >>>>>>>>>>>> (latest, earliest, by_duration, none).
> > >>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> > >>>>>>>>>> auto.offset.reset.new.partitions
> > >>>>>>>>>>>> config setting
> > >>>>>>>>>>>> - For existing ("cold") partitions, the base
> auto.offset.reset
> > >>>>>> policy
> > >>>>>>>>>> continues
> > >>>>>>>>>>>> to apply unchanged.
> > >>>>>>>>>>>> - The new-partition reset behavior is represented by a
> > separate
> > >>>>>>>>>> internal config
> > >>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
> > >> earliest).
> > >>>>>> This
> > >>>>>>>>>> decoupled design makes
> > >>>>>>>>>>>> it straightforward to promote the behavior to a public
> > >>> user-facing
> > >>>>>>>>>> configuration in a future KIP.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46
> 寫道:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
> > below:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default
> value
> > >> of
> > >>>>>>>>>> earliest. It fixes the data loss from both by_duration and
> > >> latest,
> > >>> and
> > >>>>>>>> it
> > >>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
> > >>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
> > >>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest
> > already
> > >>>>>>>>>> addresses the issue, and we can discuss the use cases of other
> > >>> values
> > >>>>>>>> in a
> > >>>>>>>>>> separate KIP.
> > >>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> > >>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all
> for
> > >>>>>>>>>> consistency.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> WDYT?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
> > >> topic
> > >>> is
> > >>>>>>>>>> expanded
> > >>>>>>>>>>>>>> with new partitions, consumers using the latest auto
> offset
> > >>> reset
> > >>>>>>>>>> policy
> > >>>>>>>>>>>>>> will silently miss all records produced to those
> partitions
> > >>> before
> > >>>>>>>> the
> > >>>>>>>>>>>>>> consumer discovers them.". If a user sets
> > >>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
> > >> issue
> > >>>>>> could
> > >>>>>>>>>> also
> > >>>>>>>>>>>>>> happen, right?
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will
> > take
> > >>> the
> > >>>>>>>> same
> > >>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
> > >>> by_duration if
> > >>>>>>>>>> needed.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > >>>>>> chia7712@apache.org
> > >>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
> > >>> important
> > >>>>>>>> from
> > >>>>>>>>>> your
> > >>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify
> with
> > >> a
> > >>> few
> > >>>>>>>>>> questions:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic to
> > >>> other
> > >>>>>>>> reset
> > >>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> > >> by_duration
> > >>>>>>>> don't
> > >>>>>>>>>> seem
> > >>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
> > >>> partition
> > >>>>>> is
> > >>>>>>>>>> expanded.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 2. What values would we expect users to configure for
> > >>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
> > >> earliest
> > >>> or
> > >>>>>>>>>> latest,
> > >>>>>>>>>>>>>>> we might run into the exact same edge cases. For example,
> > >> if a
> > >>>>>>>>>> consumer is
> > >>>>>>>>>>>>>>> offline for a while and a new partition is created during
> > >> that
> > >>>>>>>>>> downtime,
> > >>>>>>>>>>>>>>> the user might actually want to skip to latest when
> > >> resuming,
> > >>>>>>>> rather
> > >>>>>>>>>> than
> > >>>>>>>>>>>>>>> reading from earliest just because the partition is
> > >>> technically
> > >>>>>>>>>> "new" to
> > >>>>>>>>>>>>>>> the group.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
> > >>> threshold.
> > >>>>>>>> It
> > >>>>>>>>>> gives
> > >>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
> > "hot/new"
> > >>> and
> > >>>>>>>>>> what is
> > >>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
> > >> only
> > >>>>>>>>>> applies when
> > >>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> > >>>>>> motivation
> > >>>>>>>>>>>>>>> equally
> > >>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values
> like
> > >>>>>>>>>> by_duration.
> > >>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>> intention is that we want to have a separate way to
> > control
> > >>>>>> newly
> > >>>>>>>>>> created
> > >>>>>>>>>>>>>>>> partitions vs existing partitions when the group starts.
> > >>> Have we
> > >>>>>>>>>>>>>>> considered
> > >>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> > .partitions?
> > >>> If
> > >>>>>>>> this
> > >>>>>>>>>> new
> > >>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to
> the
> > >>>>>> policy
> > >>>>>>>>>> used
> > >>>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly to
> > >>>>>> customize
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>> behavior for new partitions.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com>
> > >>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks for the feedback.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> DJ01/DJ02:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > >>> PartitionMetadata
> > >>>>>>>>>> struct
> > >>>>>>>>>>>>>>>>> gains a new
> > >>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> > >>> server-side
> > >>>>>>>> by
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> broker as
> > >>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > >>> MembershipManager
> > >>>>>>>>>> detects
> > >>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>> newly assigned
> > >>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
> > the
> > >>>>>>>> affected
> > >>>>>>>>>>>>>>> topic
> > >>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > >>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
> > >> topic
> > >>> ID
> > >>>>>> is
> > >>>>>>>>>>>>>>> already
> > >>>>>>>>>>>>>>>>> in the cache.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> MB0:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> > >>>>>>>>>> MetadataResponse
> > >>>>>>>>>>>>>>>>> version via the
> > >>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> > >>> negotiated
> > >>>>>>>>>>>>>>> version is
> > >>>>>>>>>>>>>>>>> unsupported, the consumer
> > >>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at
> all
> > >> and
> > >>>>>> can
> > >>>>>>>>>>>>>>> throw an
> > >>>>>>>>>>>>>>>>> UnsupportedVersionException
> > >>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to
> latest
> > >>> and
> > >>>>>>>>>> risking
> > >>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
> > 下午4:04
> > >>> 寫道:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> hi David
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> > >> resolution
> > >>>>>> from
> > >>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
> > >> plane
> > >>>>>>>> clean.
> > >>>>>>>>>> The
> > >>>>>>>>>>>>>>> main
> > >>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> > inter-broker
> > >>>>>> clock
> > >>>>>>>>>> skew.
> > >>>>>>>>>>>>>>> The
> > >>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
> > >> truth
> > >>>>>> for
> > >>>>>>>>>> time.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
> > >>> negligible.
> > >>>>>>>>>> Given
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
> > minutes
> > >>> or
> > >>>>>>>>>> hours, a
> > >>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers
> won't
> > >>> impact
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>>> fallback decision.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> > >> 2026年4月29日
> > >>>>>>>> 下午3:29
> > >>>>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> > >>> conversation
> > >>>>>>>> but I
> > >>>>>>>>>>>>>>> took a
> > >>>>>>>>>>>>>>>>>>>> quick look at this one.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> > >>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean;
> the
> > >>> KIP
> > >>>>>>>>>> states
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean
> is
> > >>> set.
> > >>>>>>>>>>>>>>> Implicitly,
> > >>>>>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new
> partition,
> > >> it
> > >>>>>> will
> > >>>>>>>>>>>>>>> issue a
> > >>>>>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages.
> > Is
> > >>> my
> > >>>>>>>>>>>>>>>>> understanding
> > >>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also
> > >>> explain
> > >>>>>>>>>> how it
> > >>>>>>>>>>>>>>>>> fits
> > >>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> > >> offsets,
> > >>>>>>>> etc.).
> > >>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> > >>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place
> for
> > >>> this
> > >>>>>>>> given
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> a new
> > >>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
> > >> simply
> > >>>>>>>>>> include
> > >>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
> > about
> > >>> not
> > >>>>>>>>>>>>>>> overloading
> > >>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
> > concepts.
> > >>> The
> > >>>>>>>> API
> > >>>>>>>>>> is
> > >>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> > partitions.
> > >>> The
> > >>>>>>>> fact
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams
> API
> > >>> also
> > >>>>>>>>>> suggests
> > >>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if we
> > >>> want to
> > >>>>>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>>>>> Streams in the future?
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
> > via
> > >>> dev
> > >>>>>> <
> > >>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about
> the
> > >>> gap.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > >>> RequestPartitionAges
> > >>>>>>>>>> field.
> > >>>>>>>>>>>>>>> Can a
> > >>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> > >>>>>>>>>>>>>>> TopicPartitions)
> > >>>>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
> > >> value
> > >>> or
> > >>>>>> a
> > >>>>>>>>>>>>>>> ConfigDef
> > >>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > >>>>>>>> metadata.max.age.ms
> > >>>>>>>>>> ?
> > >>>>>>>>>>>>>>>>> Sizing
> > >>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> > >> Streams,
> > >>>>>>>> would
> > >>>>>>>>>> it
> > >>>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>>>>>> to add this new config
> > >> auto.offset.reset.latest.max.age
> > >>> to
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> > >>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > >>>>>>>>>>>>>>>>> for a
> > >>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is
> the
> > >>> most
> > >>>>>>>>>>>>>>> familiar
> > >>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
> > >>> configure
> > >>>>>>>>>> it. Or
> > >>>>>>>>>>>>>>>>> may be
> > >>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls
> back
> > >> to
> > >>>>>>>>>> earliest"
> > >>>>>>>>>>>>>>>>> reads as
> > >>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition
> which
> > >>> isn't
> > >>>>>>>>>>>>>>> supported.
> > >>>>>>>>>>>>>>>>> May
> > >>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
> > the
> > >>>>>>>> initial
> > >>>>>>>>>>>>>>>>> position to
> > >>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was
> > >>> applied
> > >>>>>>>> to
> > >>>>>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> > >>> unchanged.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>>>> Murali
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > >>> s7133700@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Hi chia,
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於
> 2026年4月28日
> > >>> 晚上8:03
> > >>>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
> > time
> > >>> via
> > >>>>>>>> the
> > >>>>>>>>>>>>>>> Admin
> > >>>>>>>>>>>>>>>>> API?
> > >>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to
> diagnose
> > >> and
> > >>>>>>>>>>>>>>> troubleshoot
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
> > >>> Prevent
> > >>>>>> Hot
> > >>>>>>>>>> Data
> > >>>>>>>>>>>>>>>>> Loss
> > >>>>>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > >>>>>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>
> > >>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > >>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > >>>>>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> > >>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
> > >> (hot)
> > >>>>>>>>>> partitions
> > >>>>>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > >>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> > automatically
> > >>> fall
> > >>>>>>>>>> back
> > >>>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > >>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
> > >>> historical
> > >>>>>>>>>>>>>>> reprocess.
> > >>>>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping and Jiunn-Yang,

Thanks for the reply. I am still trying to understand the value of the new
configs with the KIP.

The motivation of the KIP is that a user doesn't want to miss the data if
the backlog is small. The backlog of the existing partition is easy to
understand because it relates to retention time. The backlog for the new
partition is a bit subtle to understand since it depends on the metadata
refresh delay. To set auto.offset.reset.max.age.ms, the user needs to
understand the metadata refresh delay on the consumer side and use it to
set the config.

Now, let's consider the alternative: setting the same value for the
existing by_duration policy. The KIP lists three issues with this approach.
1. It computes the seek target client-side as now() - duration, which
introduces clock skew across consumers and forces operators to choose
overly large durations, causing unnecessary reprocessing.
2. The target timestamp is recomputed on each retry, so failed
ListOffsetsRequest retries can shift the target forward and potentially
miss records produced between attempts.
3. It applies uniformly to all partitions without committed offsets, and
cannot distinguish newly expanded partitions from long-existing partitions
newly assigned to the group, leading to unnecessary replay.

Issues 1 and 2 are uncommon and can be mitigated by adding a bit buffer to
the metadata refresh delay. We could also consider improving the
implementation. For issue 3, the metadata refresh delay is typically low
(in the order of minutes with the classic consumer and tens of seconds with
the new consumer). If a user is ok with reading that much backlog for new
partitions, it seems they will be ok doing the same for existing partitions.

So, instead of introducing a new config, could we just reuse the existing
config with better documentation and/or implementation?

Jun


On Sat, Jun 13, 2026 at 12:19 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello Jun,
>
> You're right that group creation time is the more intuitive answer at
> first glance,
> the KIP's own motivation talks about partitions that "predate the group"
> vs partitions
> "created during group runtime," which directly points to a group-lifecycle
> classifier.
> I'd like to walk through why we landed on partition age, and the
> trade-offs we considered.
>
> We evaluated three candidate signals:
>
> 1. `by_duration:5secs`
>
> This covers the metadata blindness window, but has issues the KIP
> currently documents
> under "Why not use `by_duration`?":
>
> - Client-side `now() - duration` introduces clock skew across consumers.
> - `ListOffsets` retries shift the target forward, potentially missing
> records produced between
> attempts.
> - It applies uniformly to all partitions without committed offsets,
> including pre-existing partitions
> newly assigned to the group, causing unnecessary replay.
>
> 2. Group creation time as classifier
>
> This works cleanly when the consumer is actively running. Our concern
> is the idle / late-rejoin case:
>
> T=0:         Group created.
> T=1..T=100:  Consumer idle (down, disconnected, etc.).
> T=50:        Partition added during the idle window.
> T=100:       Consumer resumes.
>
> Under group creation time, the new partition is classified as new
> (`50 > 0`) and reset to `earliest`, replaying everything from T=50.
> But during `[T=1, T=100]`, base partitions also accumulated data that
> the consumer accepts as lost — that is precisely the contract of
> `auto.offset.reset=latest`. There is no principled reason to treat
> the new partition differently; both contain backlog accumulated during
> the same idle window.
>
> This aligns with the "backlog is backlog” principle you raised in
> the KIP-1282 thread: a `latest` user has tolerated some backlog on
> every other partition during the same idle period; forcing 0-backlog
> tolerance only on new partitions would be inconsistent with that
> tolerance.
>
> 3. Partition age vs threshold
>
> Partition age corresponds to the actual silent data loss window,
> the gap between partition creation and the consumer’s metadata
> refresh. Within this window, data loss is genuinely silent: the
> consumer had no opportunity to know about the partition. Outside this
> window, missing data reflects either:
>
> - (a) the user’s tolerated cost of running with idle consumers, or
> - (b) an operational issue to surface via monitoring, not via reset policy.
>
> We did not choose partition age because it is more elegant than group
> creation time — we chose it because its failure mode (requires a
> threshold) is
> less invasive than the failure mode of group creation time (overrides
> user-stated
> `latest` intent during idle periods).
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> >
> > Hi Jun,
> >
> > Relying on both creation times will create an inconsistent scenario. A
> > consumer that lost all offsets due to a long sleep will seek to the
> > beginning for the partitions created later than the group.
> >
> > That is why we initially proposed KIP-1282 to fix the inconsistency
> using a
> > whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327
> goes
> > back to using flexible configurations to prevent users from falling into
> > that pitfall.
> >
> > Best, Chia-Ping
> >
> > Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> >
> >> Hi, Jiunn-Yang,
> >>
> >> Thanks for the reply and sorry for the late reply.
> >>
> >> JR1. The design of auto.offset.reset.max.age.ms still feels weird to
> me.
> >> It
> >> categorizes partitions as new or existing based on the partition
> creation
> >> time. Intuitively, the categorization should be based on the group
> creation
> >> time: all partitions existing when the group is created are existing and
> >> all partitions created after the group creation are new partitions.
> >>
> >> Jun
> >>
> >>
> >>
> >> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Manually bumping this thread. If there is no further
> >>> discussion, I will close the vote.
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> >>>>
> >>>> Hello Jian,
> >>>>
> >>>> Thanks for your feedback,
> >>>>
> >>>> Agreed, partition expansion is a common operational task, not an edge
> >>>> case. I've updated the Motivation section accordingly.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> >>>>>
> >>>>> Hi Jiunn-Yang:
> >>>>>
> >>>>> Thanks for the KIP. I think it would be useful to clarify that this
> >> is a
> >>>>> common scenario rather than an edge case, which further demonstrates
> >> the
> >>>>> need for this optimization. For example:
> >>>>> A partition expansion is a common operational task in Kafka: To
> >> balance
> >>>>> resource utilization and cost, topics are typically created with a
> >>> moderate
> >>>>> default partition count. However, as traffic grows over time, it is
> >>> often
> >>>>> necessary to increase the number of partitions to accommodate the
> >> higher
> >>>>> workload.
> >>>>>
> >>>>> Regards
> >>>>> Jian
> >>>>>
> >>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> >>>>>
> >>>>>> Hello chia,
> >>>>>>
> >>>>>> Thanks for the comments, I have updated the KIP!
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> >>>>>>>
> >>>>>>> Hi Jiunn-Yang,
> >>>>>>>
> >>>>>>> Would you mind removing the terms "hot" and "cold" when describing
> >>>>>>> partitions in the KIP? I understand you are using them to describe
> >> the
> >>>>>>> "freshness" or the users' need for the records, but applying these
> >>> terms
> >>>>>> to
> >>>>>>> the partition itself feels a bit unnatural.
> >>>>>>>
> >>>>>>> After all, in this scenario, users don't really care whether a
> >>> partition
> >>>>>> is
> >>>>>>> newly expanded or not. Their only expectation is that they won't
> >>> silently
> >>>>>>> lose any live records produced to the topic during their active
> >>>>>> consumption.
> >>>>>>>
> >>>>>>> Best, Chia-Ping
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >>>>>>>
> >>>>>>>> Hello Jun,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback, I have updated the KIP motivation
> section.
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>>
> >>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> >>>>>>>>>
> >>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>
> >>>>>>>>> Thanks for the reply. I think we need a stronger motivation for
> >> the
> >>>>>> KIP.
> >>>>>>>>>
> >>>>>>>>> The KIP says "The core insight is that not all partitions without
> >> a
> >>>>>>>>> committed offset are the same. A newly expanded partition (hot)
> is
> >>>>>>>>> fundamentally different from a partition the consumer has never
> >> seen
> >>>>>>>>> because it predates the group (cold)." Why is the hot partition
> >>>>>>>>> fundamentally different from the cold?
> >>>>>>>>>
> >>>>>>>>> The KIP says "The existing by_duration policy is also
> insufficient
> >>>>>>>> because:
> >>>>>>>>>
> >>>>>>>>> - The calculated seek time (now() - duration) varies across nodes
> >>> due
> >>>>>>>> to
> >>>>>>>>> clock skew. To be safe, users must set an overly large duration,
> >>>>>>>> causing
> >>>>>>>>> unnecessary reprocessing.
> >>>>>>>>> - On network errors, the client recalculates the seek time on
> >> retry,
> >>>>>>>>> shifting the target timestamp forward and risking data loss."
> >>>>>>>>>
> >>>>>>>>> However, both of these situations are rare. If these issues
> >> persist,
> >>>>>> more
> >>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
> >> need a
> >>>>>>>> common
> >>>>>>>>> solution. If users care about those rare situations, they can
> >>> implement
> >>>>>>>>> customized logic using
> >>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>>>>>>>
> >>>>>>>>> Jun
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hello chia,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the feedback,
> >>>>>>>>>>
> >>>>>>>>>>> If the creation time exists, the returned value should always
> be
> >>>>>>>> greater
> >>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>> I have explicitly mentioned this in the KIP.
> >>>>>>>>>>
> >>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >>> field
> >>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>
> >>>>>>>>>> The earliest point at which we can detect the version mismatch
> is
> >>>>>> during
> >>>>>>>>>> the
> >>>>>>>>>> first metadata fetch after assignment, which occurs inside
> >> poll().
> >>>>>>>>>> Therefore, the
> >>>>>>>>>> user would encounter an UnsupportedVersionException from poll().
> >>> I’ll
> >>>>>>>>>> clarify this in the KIP.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>
> >>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> >>>>>>>>>>>
> >>>>>>>>>>> hi Jiunn
> >>>>>>>>>>>
> >>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this partition
> >> in
> >>>>>>>>>> milliseconds, computed server-side by the broker as
> >>>>>> broker_current_time
> >>>>>>>> -
> >>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
> >> support
> >>>>>> this
> >>>>>>>>>> feature or the partition creation time is unknown.
> >>>>>>>>>>>
> >>>>>>>>>>> If the creation time exists, the returned value should always
> be
> >>>>>>>> greater
> >>>>>>>>>> than or equal to zero, right?
> >>>>>>>>>>>
> >>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >>> field
> >>>>>>>>>> absent    UnsupportedVersionException
> >>>>>>>>>>>
> >>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
> >>>>>> `poll()`?
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>>>>>>>> Hello Jun, chia,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
> >>> discussion
> >>>>>>>>>>>> feedback.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The updated design decouples the new-partition reset behavior
> >>> from
> >>>>>>>>>>>> the base auto.offset.reset policy:
> >>>>>>>>>>>>
> >>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> >>> auto.offset.reset
> >>>>>>>>>> values
> >>>>>>>>>>>> (latest, earliest, by_duration, none).
> >>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
> >>>>>>>>>> auto.offset.reset.new.partitions
> >>>>>>>>>>>> config setting
> >>>>>>>>>>>> - For existing ("cold") partitions, the base auto.offset.reset
> >>>>>> policy
> >>>>>>>>>> continues
> >>>>>>>>>>>> to apply unchanged.
> >>>>>>>>>>>> - The new-partition reset behavior is represented by a
> separate
> >>>>>>>>>> internal config
> >>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
> >> earliest).
> >>>>>> This
> >>>>>>>>>> decoupled design makes
> >>>>>>>>>>>> it straightforward to promote the behavior to a public
> >>> user-facing
> >>>>>>>>>> configuration in a future KIP.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I see what you mean now. The proposal from me is listed
> below:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default value
> >> of
> >>>>>>>>>> earliest. It fixes the data loss from both by_duration and
> >> latest,
> >>> and
> >>>>>>>> it
> >>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
> >>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
> >>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest
> already
> >>>>>>>>>> addresses the issue, and we can discuss the use cases of other
> >>> values
> >>>>>>>> in a
> >>>>>>>>>> separate KIP.
> >>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> >>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all for
> >>>>>>>>>> consistency.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> WDYT?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
> >> topic
> >>> is
> >>>>>>>>>> expanded
> >>>>>>>>>>>>>> with new partitions, consumers using the latest auto offset
> >>> reset
> >>>>>>>>>> policy
> >>>>>>>>>>>>>> will silently miss all records produced to those partitions
> >>> before
> >>>>>>>> the
> >>>>>>>>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
> >> issue
> >>>>>> could
> >>>>>>>>>> also
> >>>>>>>>>>>>>> happen, right?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will
> take
> >>> the
> >>>>>>>> same
> >>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
> >>> by_duration if
> >>>>>>>>>> needed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> >>>>>> chia7712@apache.org
> >>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
> >>> important
> >>>>>>>> from
> >>>>>>>>>> your
> >>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify with
> >> a
> >>> few
> >>>>>>>>>> questions:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic to
> >>> other
> >>>>>>>> reset
> >>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> >> by_duration
> >>>>>>>> don't
> >>>>>>>>>> seem
> >>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
> >>> partition
> >>>>>> is
> >>>>>>>>>> expanded.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2. What values would we expect users to configure for
> >>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
> >> earliest
> >>> or
> >>>>>>>>>> latest,
> >>>>>>>>>>>>>>> we might run into the exact same edge cases. For example,
> >> if a
> >>>>>>>>>> consumer is
> >>>>>>>>>>>>>>> offline for a while and a new partition is created during
> >> that
> >>>>>>>>>> downtime,
> >>>>>>>>>>>>>>> the user might actually want to skip to latest when
> >> resuming,
> >>>>>>>> rather
> >>>>>>>>>> than
> >>>>>>>>>>>>>>> reading from earliest just because the partition is
> >>> technically
> >>>>>>>>>> "new" to
> >>>>>>>>>>>>>>> the group.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
> >>> threshold.
> >>>>>>>> It
> >>>>>>>>>> gives
> >>>>>>>>>>>>>>> users a time-bound way to define what is genuinely
> "hot/new"
> >>> and
> >>>>>>>>>> what is
> >>>>>>>>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
> >> only
> >>>>>>>>>> applies when
> >>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> >>>>>> motivation
> >>>>>>>>>>>>>>> equally
> >>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values like
> >>>>>>>>>> by_duration.
> >>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>> intention is that we want to have a separate way to
> control
> >>>>>> newly
> >>>>>>>>>> created
> >>>>>>>>>>>>>>>> partitions vs existing partitions when the group starts.
> >>> Have we
> >>>>>>>>>>>>>>> considered
> >>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new
> .partitions?
> >>> If
> >>>>>>>> this
> >>>>>>>>>> new
> >>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to the
> >>>>>> policy
> >>>>>>>>>> used
> >>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly to
> >>>>>> customize
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com>
> >>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> >>> PartitionMetadata
> >>>>>>>>>> struct
> >>>>>>>>>>>>>>>>> gains a new
> >>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> >>> server-side
> >>>>>>>> by
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> broker as
> >>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> >>> MembershipManager
> >>>>>>>>>> detects
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> newly assigned
> >>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for
> the
> >>>>>>>> affected
> >>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
> >> topic
> >>> ID
> >>>>>> is
> >>>>>>>>>>>>>>> already
> >>>>>>>>>>>>>>>>> in the cache.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> MB0:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> >>>>>>>>>> MetadataResponse
> >>>>>>>>>>>>>>>>> version via the
> >>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> >>> negotiated
> >>>>>>>>>>>>>>> version is
> >>>>>>>>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at all
> >> and
> >>>>>> can
> >>>>>>>>>>>>>>> throw an
> >>>>>>>>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to latest
> >>> and
> >>>>>>>>>> risking
> >>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日
> 下午4:04
> >>> 寫道:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> >> resolution
> >>>>>> from
> >>>>>>>>>> the
> >>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
> >> plane
> >>>>>>>> clean.
> >>>>>>>>>> The
> >>>>>>>>>>>>>>> main
> >>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing
> inter-broker
> >>>>>> clock
> >>>>>>>>>> skew.
> >>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
> >> truth
> >>>>>> for
> >>>>>>>>>> time.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
> >>> negligible.
> >>>>>>>>>> Given
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in
> minutes
> >>> or
> >>>>>>>>>> hours, a
> >>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't
> >>> impact
> >>>>>>>> the
> >>>>>>>>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> >> 2026年4月29日
> >>>>>>>> 下午3:29
> >>>>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> >>> conversation
> >>>>>>>> but I
> >>>>>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the
> >>> KIP
> >>>>>>>>>> states
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean is
> >>> set.
> >>>>>>>>>>>>>>> Implicitly,
> >>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new partition,
> >> it
> >>>>>> will
> >>>>>>>>>>>>>>> issue a
> >>>>>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages.
> Is
> >>> my
> >>>>>>>>>>>>>>>>> understanding
> >>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also
> >>> explain
> >>>>>>>>>> how it
> >>>>>>>>>>>>>>>>> fits
> >>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> >> offsets,
> >>>>>>>> etc.).
> >>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for
> >>> this
> >>>>>>>> given
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> a new
> >>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
> >> simply
> >>>>>>>>>> include
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious
> about
> >>> not
> >>>>>>>>>>>>>>> overloading
> >>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated
> concepts.
> >>> The
> >>>>>>>> API
> >>>>>>>>>> is
> >>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking
> partitions.
> >>> The
> >>>>>>>> fact
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams API
> >>> also
> >>>>>>>>>> suggests
> >>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if we
> >>> want to
> >>>>>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani
> via
> >>> dev
> >>>>>> <
> >>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about the
> >>> gap.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> >>> RequestPartitionAges
> >>>>>>>>>> field.
> >>>>>>>>>>>>>>> Can a
> >>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> >>>>>>>>>>>>>>> TopicPartitions)
> >>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
> >> value
> >>> or
> >>>>>> a
> >>>>>>>>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> >>>>>>>> metadata.max.age.ms
> >>>>>>>>>> ?
> >>>>>>>>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> >> Streams,
> >>>>>>>> would
> >>>>>>>>>> it
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>>>>> to add this new config
> >> auto.offset.reset.latest.max.age
> >>> to
> >>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is the
> >>> most
> >>>>>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
> >>> configure
> >>>>>>>>>> it. Or
> >>>>>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back
> >> to
> >>>>>>>>>> earliest"
> >>>>>>>>>>>>>>>>> reads as
> >>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition which
> >>> isn't
> >>>>>>>>>>>>>>> supported.
> >>>>>>>>>>>>>>>>> May
> >>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves
> the
> >>>>>>>> initial
> >>>>>>>>>>>>>>>>> position to
> >>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was
> >>> applied
> >>>>>>>> to
> >>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> >>> unchanged.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> >>> s7133700@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日
> >>> 晚上8:03
> >>>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation
> time
> >>> via
> >>>>>>>> the
> >>>>>>>>>>>>>>> Admin
> >>>>>>>>>>>>>>>>> API?
> >>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose
> >> and
> >>>>>>>>>>>>>>> troubleshoot
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
> >>> Prevent
> >>>>>> Hot
> >>>>>>>>>> Data
> >>>>>>>>>>>>>>>>> Loss
> >>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
> >> (hot)
> >>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold
> automatically
> >>> fall
> >>>>>>>>>> back
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
> >>> historical
> >>>>>>>>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello Jun,

You're right that group creation time is the more intuitive answer at first glance, 
the KIP's own motivation talks about partitions that "predate the group" vs partitions 
"created during group runtime," which directly points to a group-lifecycle classifier. 
I'd like to walk through why we landed on partition age, and the trade-offs we considered.

We evaluated three candidate signals:

1. `by_duration:5secs`

This covers the metadata blindness window, but has issues the KIP currently documents 
under "Why not use `by_duration`?": 

- Client-side `now() - duration` introduces clock skew across consumers. 
- `ListOffsets` retries shift the target forward, potentially missing records produced between 
attempts. 
- It applies uniformly to all partitions without committed offsets, including pre-existing partitions 
newly assigned to the group, causing unnecessary replay.

2. Group creation time as classifier

This works cleanly when the consumer is actively running. Our concern
is the idle / late-rejoin case:

T=0:         Group created.
T=1..T=100:  Consumer idle (down, disconnected, etc.).
T=50:        Partition added during the idle window.
T=100:       Consumer resumes.

Under group creation time, the new partition is classified as new
(`50 > 0`) and reset to `earliest`, replaying everything from T=50.
But during `[T=1, T=100]`, base partitions also accumulated data that
the consumer accepts as lost — that is precisely the contract of
`auto.offset.reset=latest`. There is no principled reason to treat
the new partition differently; both contain backlog accumulated during
the same idle window.

This aligns with the "backlog is backlog” principle you raised in
the KIP-1282 thread: a `latest` user has tolerated some backlog on
every other partition during the same idle period; forcing 0-backlog
tolerance only on new partitions would be inconsistent with that
tolerance.

3. Partition age vs threshold

Partition age corresponds to the actual silent data loss window, 
the gap between partition creation and the consumer’s metadata
refresh. Within this window, data loss is genuinely silent: the
consumer had no opportunity to know about the partition. Outside this
window, missing data reflects either:

- (a) the user’s tolerated cost of running with idle consumers, or
- (b) an operational issue to surface via monitoring, not via reset policy.

We did not choose partition age because it is more elegant than group 
creation time — we chose it because its failure mode (requires a threshold) is 
less invasive than the failure mode of group creation time (overrides user-stated 
`latest` intent during idle periods).

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2026年6月13日 上午11:52 寫道:
> 
> Hi Jun,
> 
> Relying on both creation times will create an inconsistent scenario. A
> consumer that lost all offsets due to a long sleep will seek to the
> beginning for the partitions created later than the group.
> 
> That is why we initially proposed KIP-1282 to fix the inconsistency using a
> whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327 goes
> back to using flexible configurations to prevent users from falling into
> that pitfall.
> 
> Best, Chia-Ping
> 
> Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:
> 
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the reply and sorry for the late reply.
>> 
>> JR1. The design of auto.offset.reset.max.age.ms still feels weird to me.
>> It
>> categorizes partitions as new or existing based on the partition creation
>> time. Intuitively, the categorization should be based on the group creation
>> time: all partitions existing when the group is created are existing and
>> all partitions created after the group creation are new partitions.
>> 
>> Jun
>> 
>> 
>> 
>> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hi all,
>>> 
>>> Manually bumping this thread. If there is no further
>>> discussion, I will close the vote.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
>>>> 
>>>> Hello Jian,
>>>> 
>>>> Thanks for your feedback,
>>>> 
>>>> Agreed, partition expansion is a common operational task, not an edge
>>>> case. I've updated the Motivation section accordingly.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>>>>> 
>>>>> Hi Jiunn-Yang:
>>>>> 
>>>>> Thanks for the KIP. I think it would be useful to clarify that this
>> is a
>>>>> common scenario rather than an edge case, which further demonstrates
>> the
>>>>> need for this optimization. For example:
>>>>> A partition expansion is a common operational task in Kafka: To
>> balance
>>>>> resource utilization and cost, topics are typically created with a
>>> moderate
>>>>> default partition count. However, as traffic grows over time, it is
>>> often
>>>>> necessary to increase the number of partitions to accommodate the
>> higher
>>>>> workload.
>>>>> 
>>>>> Regards
>>>>> Jian
>>>>> 
>>>>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>>>>> 
>>>>>> Hello chia,
>>>>>> 
>>>>>> Thanks for the comments, I have updated the KIP!
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>>>>>> 
>>>>>>> Hi Jiunn-Yang,
>>>>>>> 
>>>>>>> Would you mind removing the terms "hot" and "cold" when describing
>>>>>>> partitions in the KIP? I understand you are using them to describe
>> the
>>>>>>> "freshness" or the users' need for the records, but applying these
>>> terms
>>>>>> to
>>>>>>> the partition itself feels a bit unnatural.
>>>>>>> 
>>>>>>> After all, in this scenario, users don't really care whether a
>>> partition
>>>>>> is
>>>>>>> newly expanded or not. Their only expectation is that they won't
>>> silently
>>>>>>> lose any live records produced to the topic during their active
>>>>>> consumption.
>>>>>>> 
>>>>>>> Best, Chia-Ping
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>>>>> 
>>>>>>>> Hello Jun,
>>>>>>>> 
>>>>>>>> Thanks for the feedback, I have updated the KIP motivation section.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
>>>>>>>>> 
>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>> 
>>>>>>>>> Thanks for the reply. I think we need a stronger motivation for
>> the
>>>>>> KIP.
>>>>>>>>> 
>>>>>>>>> The KIP says "The core insight is that not all partitions without
>> a
>>>>>>>>> committed offset are the same. A newly expanded partition (hot) is
>>>>>>>>> fundamentally different from a partition the consumer has never
>> seen
>>>>>>>>> because it predates the group (cold)." Why is the hot partition
>>>>>>>>> fundamentally different from the cold?
>>>>>>>>> 
>>>>>>>>> The KIP says "The existing by_duration policy is also insufficient
>>>>>>>> because:
>>>>>>>>> 
>>>>>>>>> - The calculated seek time (now() - duration) varies across nodes
>>> due
>>>>>>>> to
>>>>>>>>> clock skew. To be safe, users must set an overly large duration,
>>>>>>>> causing
>>>>>>>>> unnecessary reprocessing.
>>>>>>>>> - On network errors, the client recalculates the seek time on
>> retry,
>>>>>>>>> shifting the target timestamp forward and risking data loss."
>>>>>>>>> 
>>>>>>>>> However, both of these situations are rare. If these issues
>> persist,
>>>>>> more
>>>>>>>>> severe problems likely exist elsewhere. Rare situations don't
>> need a
>>>>>>>> common
>>>>>>>>> solution. If users care about those rare situations, they can
>>> implement
>>>>>>>>> customized logic using
>>>>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>>>>> 
>>>>>>>>> Jun
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hello chia,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the feedback,
>>>>>>>>>> 
>>>>>>>>>>> If the creation time exists, the returned value should always be
>>>>>>>> greater
>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>>>>> 
>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>> field
>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>> 
>>>>>>>>>> The earliest point at which we can detect the version mismatch is
>>>>>> during
>>>>>>>>>> the
>>>>>>>>>> first metadata fetch after assignment, which occurs inside
>> poll().
>>>>>>>>>> Therefore, the
>>>>>>>>>> user would encounter an UnsupportedVersionException from poll().
>>> I’ll
>>>>>>>>>> clarify this in the KIP.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
>>>>>>>>>>> 
>>>>>>>>>>> hi Jiunn
>>>>>>>>>>> 
>>>>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this partition
>> in
>>>>>>>>>> milliseconds, computed server-side by the broker as
>>>>>> broker_current_time
>>>>>>>> -
>>>>>>>>>> partition_creation_time. Returns -1 if the broker does not
>> support
>>>>>> this
>>>>>>>>>> feature or the partition creation time is unknown.
>>>>>>>>>>> 
>>>>>>>>>>> If the creation time exists, the returned value should always be
>>>>>>>> greater
>>>>>>>>>> than or equal to zero, right?
>>>>>>>>>>> 
>>>>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>>> field
>>>>>>>>>> absent    UnsupportedVersionException
>>>>>>>>>>> 
>>>>>>>>>>> Will user encounter UnsupportedVersionException when calling
>>>>>> `poll()`?
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>>>>> Hello Jun, chia,
>>>>>>>>>>>> 
>>>>>>>>>>>> I've updated KIP-1327 with a design change based on the
>>> discussion
>>>>>>>>>>>> feedback.
>>>>>>>>>>>> 
>>>>>>>>>>>> The updated design decouples the new-partition reset behavior
>>> from
>>>>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>>>>> 
>>>>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
>>> auto.offset.reset
>>>>>>>>>> values
>>>>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>>>>> auto.offset.reset.new.partitions
>>>>>>>>>>>> config setting
>>>>>>>>>>>> - For existing ("cold") partitions, the base auto.offset.reset
>>>>>> policy
>>>>>>>>>> continues
>>>>>>>>>>>> to apply unchanged.
>>>>>>>>>>>> - The new-partition reset behavior is represented by a separate
>>>>>>>>>> internal config
>>>>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
>> earliest).
>>>>>> This
>>>>>>>>>> decoupled design makes
>>>>>>>>>>>> it straightforward to promote the behavior to a public
>>> user-facing
>>>>>>>>>> configuration in a future KIP.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I see what you mean now. The proposal from me is listed below:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default value
>> of
>>>>>>>>>> earliest. It fixes the data loss from both by_duration and
>> latest,
>>> and
>>>>>>>> it
>>>>>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
>>>>>>>>>> configuration. auto.offset.reset.new.partitions=earliest already
>>>>>>>>>> addresses the issue, and we can discuss the use cases of other
>>> values
>>>>>>>> in a
>>>>>>>>>> separate KIP.
>>>>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
>>>>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all for
>>>>>>>>>> consistency.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> WDYT?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
>> topic
>>> is
>>>>>>>>>> expanded
>>>>>>>>>>>>>> with new partitions, consumers using the latest auto offset
>>> reset
>>>>>>>>>> policy
>>>>>>>>>>>>>> will silently miss all records produced to those partitions
>>> before
>>>>>>>> the
>>>>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
>> issue
>>>>>> could
>>>>>>>>>> also
>>>>>>>>>>>>>> happen, right?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will take
>>> the
>>>>>>>> same
>>>>>>>>>>>>>> values as auto.offset.reset. So a user could set it
>>> by_duration if
>>>>>>>>>> needed.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>>>>> chia7712@apache.org
>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> hi Jun
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the feedback. I might be missing something
>>> important
>>>>>>>> from
>>>>>>>>>> your
>>>>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify with
>> a
>>> few
>>>>>>>>>> questions:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. Is there a strong use case for extending this logic to
>>> other
>>>>>>>> reset
>>>>>>>>>>>>>>> policies? Unlike latest, policies like earliest or
>> by_duration
>>>>>>>> don't
>>>>>>>>>> seem
>>>>>>>>>>>>>>> to suffer from the same silent data loss issue when a
>>> partition
>>>>>> is
>>>>>>>>>> expanded.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
>> earliest
>>> or
>>>>>>>>>> latest,
>>>>>>>>>>>>>>> we might run into the exact same edge cases. For example,
>> if a
>>>>>>>>>> consumer is
>>>>>>>>>>>>>>> offline for a while and a new partition is created during
>> that
>>>>>>>>>> downtime,
>>>>>>>>>>>>>>> the user might actually want to skip to latest when
>> resuming,
>>>>>>>> rather
>>>>>>>>>> than
>>>>>>>>>>>>>>> reading from earliest just because the partition is
>>> technically
>>>>>>>>>> "new" to
>>>>>>>>>>>>>>> the group.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This is exactly why we opted for introducing a max.age
>>> threshold.
>>>>>>>> It
>>>>>>>>>> gives
>>>>>>>>>>>>>>> users a time-bound way to define what is genuinely "hot/new"
>>> and
>>>>>>>>>> what is
>>>>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
>> only
>>>>>>>>>> applies when
>>>>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>>>>>> motivation
>>>>>>>>>>>>>>> equally
>>>>>>>>>>>>>>>> applies when auto.offset.reset is set to other values like
>>>>>>>>>> by_duration.
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>> intention is that we want to have a separate way to control
>>>>>> newly
>>>>>>>>>> created
>>>>>>>>>>>>>>>> partitions vs existing partitions when the group starts.
>>> Have we
>>>>>>>>>>>>>>> considered
>>>>>>>>>>>>>>>> adding a new config like auto.offset.reset.new.partitions?
>>> If
>>>>>>>> this
>>>>>>>>>> new
>>>>>>>>>>>>>>>> config is not set, the offset reset policy defaults to the
>>>>>> policy
>>>>>>>>>> used
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> existing partitions. The user could set it explicitly to
>>>>>> customize
>>>>>>>>>> the
>>>>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Jun
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com>
>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
>>> PartitionMetadata
>>>>>>>>>> struct
>>>>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
>>> server-side
>>>>>>>> by
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
>>> MembershipManager
>>>>>>>>>> detects
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for the
>>>>>>>> affected
>>>>>>>>>>>>>>> topic
>>>>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>>>>> before making the offset reset decision, even if the
>> topic
>>> ID
>>>>>> is
>>>>>>>>>>>>>>> already
>>>>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>>>>>> MetadataResponse
>>>>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
>>> negotiated
>>>>>>>>>>>>>>> version is
>>>>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at all
>> and
>>>>>> can
>>>>>>>>>>>>>>> throw an
>>>>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>>>>> immediately, rather than silently falling back to latest
>>> and
>>>>>>>>>> risking
>>>>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04
>>> 寫道:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
>> resolution
>>>>>> from
>>>>>>>>>> the
>>>>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
>> plane
>>>>>>>> clean.
>>>>>>>>>> The
>>>>>>>>>>>>>>> main
>>>>>>>>>>>>>>>>> trade-off, as we noted before, is introducing inter-broker
>>>>>> clock
>>>>>>>>>> skew.
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>> Group Coordinator approach provided a single source of
>> truth
>>>>>> for
>>>>>>>>>> time.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> However, realistically, this time skew should be
>>> negligible.
>>>>>>>>>> Given
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> the max.age threshold will likely be configured in minutes
>>> or
>>>>>>>>>> hours, a
>>>>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't
>>> impact
>>>>>>>> the
>>>>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
>> 2026年4月29日
>>>>>>>> 下午3:29
>>>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
>>> conversation
>>>>>>>> but I
>>>>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the
>>> KIP
>>>>>>>>>> states
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean is
>>> set.
>>>>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> means that when the consumer receives a new partition,
>> it
>>>>>> will
>>>>>>>>>>>>>>> issue a
>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages. Is
>>> my
>>>>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also
>>> explain
>>>>>>>>>> how it
>>>>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
>> offsets,
>>>>>>>> etc.).
>>>>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for
>>> this
>>>>>>>> given
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
>> simply
>>>>>>>>>> include
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious about
>>> not
>>>>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts.
>>> The
>>>>>>>> API
>>>>>>>>>> is
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>> control plane API for assigning or revoking partitions.
>>> The
>>>>>>>> fact
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams API
>>> also
>>>>>>>>>> suggests
>>>>>>>>>>>>>>>>>>>> something is not quite right. What would we do if we
>>> want to
>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via
>>> dev
>>>>>> <
>>>>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about the
>>> gap.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
>>> RequestPartitionAges
>>>>>>>>>> field.
>>>>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
>>>>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
>> value
>>> or
>>>>>> a
>>>>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>>>>> metadata.max.age.ms
>>>>>>>>>> ?
>>>>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
>> Streams,
>>>>>>>> would
>>>>>>>>>> it
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>>>> to add this new config
>> auto.offset.reset.latest.max.age
>>> to
>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is the
>>> most
>>>>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
>>> configure
>>>>>>>>>> it. Or
>>>>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back
>> to
>>>>>>>>>> earliest"
>>>>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>>>>> if the config were being changed per-partition which
>>> isn't
>>>>>>>>>>>>>>> supported.
>>>>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves the
>>>>>>>> initial
>>>>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was
>>> applied
>>>>>>>> to
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
>>> unchanged.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
>>> s7133700@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日
>>> 晚上8:03
>>>>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time
>>> via
>>>>>>>> the
>>>>>>>>>>>>>>> Admin
>>>>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose
>> and
>>>>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
>>> Prevent
>>>>>> Hot
>>>>>>>>>> Data
>>>>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>> 
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
>> (hot)
>>>>>>>>>> partitions
>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold automatically
>>> fall
>>>>>>>>>> back
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
>>> historical
>>>>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>> 
>>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@gmail.com>.
Hi Jun,

Relying on both creation times will create an inconsistent scenario. A
consumer that lost all offsets due to a long sleep will seek to the
beginning for the partitions created later than the group.

That is why we initially proposed KIP-1282 to fix the inconsistency using a
whole new policy. Since KIP-1282 couldn't reach a consensus, KIP-1327 goes
back to using flexible configurations to prevent users from falling into
that pitfall.

Best, Chia-Ping

Jun Rao via dev <de...@kafka.apache.org> 於 2026年6月13日週六 上午6:49寫道:

> Hi, Jiunn-Yang,
>
> Thanks for the reply and sorry for the late reply.
>
> JR1. The design of auto.offset.reset.max.age.ms still feels weird to me.
> It
> categorizes partitions as new or existing based on the partition creation
> time. Intuitively, the categorization should be based on the group creation
> time: all partitions existing when the group is created are existing and
> all partitions created after the group creation are new partitions.
>
> Jun
>
>
>
> On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:
>
> > Hi all,
> >
> > Manually bumping this thread. If there is no further
> > discussion, I will close the vote.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> > >
> > > Hello Jian,
> > >
> > > Thanks for your feedback,
> > >
> > > Agreed, partition expansion is a common operational task, not an edge
> > > case. I've updated the Motivation section accordingly.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > >> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> > >>
> > >> Hi Jiunn-Yang:
> > >>
> > >> Thanks for the KIP. I think it would be useful to clarify that this
> is a
> > >> common scenario rather than an edge case, which further demonstrates
> the
> > >> need for this optimization. For example:
> > >> A partition expansion is a common operational task in Kafka: To
> balance
> > >> resource utilization and cost, topics are typically created with a
> > moderate
> > >> default partition count. However, as traffic grows over time, it is
> > often
> > >> necessary to increase the number of partitions to accommodate the
> higher
> > >> workload.
> > >>
> > >> Regards
> > >> Jian
> > >>
> > >> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> > >>
> > >>> Hello chia,
> > >>>
> > >>> Thanks for the comments, I have updated the KIP!
> > >>>
> > >>> Best Regards,
> > >>> Jiunn-Yang
> > >>>
> > >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> > >>>>
> > >>>> Hi Jiunn-Yang,
> > >>>>
> > >>>> Would you mind removing the terms "hot" and "cold" when describing
> > >>>> partitions in the KIP? I understand you are using them to describe
> the
> > >>>> "freshness" or the users' need for the records, but applying these
> > terms
> > >>> to
> > >>>> the partition itself feels a bit unnatural.
> > >>>>
> > >>>> After all, in this scenario, users don't really care whether a
> > partition
> > >>> is
> > >>>> newly expanded or not. Their only expectation is that they won't
> > silently
> > >>>> lose any live records produced to the topic during their active
> > >>> consumption.
> > >>>>
> > >>>> Best, Chia-Ping
> > >>>>
> > >>>>
> > >>>>
> > >>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> > >>>>
> > >>>>> Hello Jun,
> > >>>>>
> > >>>>> Thanks for the feedback, I have updated the KIP motivation section.
> > >>>>>
> > >>>>> Best Regards,
> > >>>>> Jiunn-Yang
> > >>>>>
> > >>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> > >>>>>>
> > >>>>>> Hi, Jiunn-Yang,
> > >>>>>>
> > >>>>>> Thanks for the reply. I think we need a stronger motivation for
> the
> > >>> KIP.
> > >>>>>>
> > >>>>>> The KIP says "The core insight is that not all partitions without
> a
> > >>>>>> committed offset are the same. A newly expanded partition (hot) is
> > >>>>>> fundamentally different from a partition the consumer has never
> seen
> > >>>>>> because it predates the group (cold)." Why is the hot partition
> > >>>>>> fundamentally different from the cold?
> > >>>>>>
> > >>>>>> The KIP says "The existing by_duration policy is also insufficient
> > >>>>> because:
> > >>>>>>
> > >>>>>> - The calculated seek time (now() - duration) varies across nodes
> > due
> > >>>>> to
> > >>>>>> clock skew. To be safe, users must set an overly large duration,
> > >>>>> causing
> > >>>>>> unnecessary reprocessing.
> > >>>>>> - On network errors, the client recalculates the seek time on
> retry,
> > >>>>>> shifting the target timestamp forward and risking data loss."
> > >>>>>>
> > >>>>>> However, both of these situations are rare. If these issues
> persist,
> > >>> more
> > >>>>>> severe problems likely exist elsewhere. Rare situations don't
> need a
> > >>>>> common
> > >>>>>> solution. If users care about those rare situations, they can
> > implement
> > >>>>>> customized logic using
> > >>> ConsumerRebalanceListener.onPartitionsAssigned().
> > >>>>>>
> > >>>>>> Jun
> > >>>>>>
> > >>>>>>
> > >>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> Hello chia,
> > >>>>>>>
> > >>>>>>> Thanks for the feedback,
> > >>>>>>>
> > >>>>>>>> If the creation time exists, the returned value should always be
> > >>>>> greater
> > >>>>>>> than or equal to zero, right?
> > >>>>>>> I have explicitly mentioned this in the KIP.
> > >>>>>>>
> > >>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >  field
> > >>>>>>> absent    UnsupportedVersionException
> > >>>>>>>
> > >>>>>>> The earliest point at which we can detect the version mismatch is
> > >>> during
> > >>>>>>> the
> > >>>>>>> first metadata fetch after assignment, which occurs inside
> poll().
> > >>>>>>> Therefore, the
> > >>>>>>> user would encounter an UnsupportedVersionException from poll().
> > I’ll
> > >>>>>>> clarify this in the KIP.
> > >>>>>>>
> > >>>>>>> Best Regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>>>
> > >>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> > >>>>>>>>
> > >>>>>>>> hi Jiunn
> > >>>>>>>>
> > >>>>>>>>> PartitionAgeMs (int64, default -1): The age of this partition
> in
> > >>>>>>> milliseconds, computed server-side by the broker as
> > >>> broker_current_time
> > >>>>> -
> > >>>>>>> partition_creation_time. Returns -1 if the broker does not
> support
> > >>> this
> > >>>>>>> feature or the partition creation time is unknown.
> > >>>>>>>>
> > >>>>>>>> If the creation time exists, the returned value should always be
> > >>>>> greater
> > >>>>>>> than or equal to zero, right?
> > >>>>>>>>
> > >>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
> >  field
> > >>>>>>> absent    UnsupportedVersionException
> > >>>>>>>>
> > >>>>>>>> Will user encounter UnsupportedVersionException when calling
> > >>> `poll()`?
> > >>>>>>>>
> > >>>>>>>> Best,
> > >>>>>>>> Chia-Ping
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> > >>>>>>>>> Hello Jun, chia,
> > >>>>>>>>>
> > >>>>>>>>> I've updated KIP-1327 with a design change based on the
> > discussion
> > >>>>>>>>> feedback.
> > >>>>>>>>>
> > >>>>>>>>> The updated design decouples the new-partition reset behavior
> > from
> > >>>>>>>>> the base auto.offset.reset policy:
> > >>>>>>>>>
> > >>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> > auto.offset.reset
> > >>>>>>> values
> > >>>>>>>>> (latest, earliest, by_duration, none).
> > >>>>>>>>> - For new ("hot") partitions, the consumer resets to
> > >>>>>>> auto.offset.reset.new.partitions
> > >>>>>>>>> config setting
> > >>>>>>>>> - For existing ("cold") partitions, the base auto.offset.reset
> > >>> policy
> > >>>>>>> continues
> > >>>>>>>>> to apply unchanged.
> > >>>>>>>>> - The new-partition reset behavior is represented by a separate
> > >>>>>>> internal config
> > >>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to
> earliest).
> > >>> This
> > >>>>>>> decoupled design makes
> > >>>>>>>>> it straightforward to promote the behavior to a public
> > user-facing
> > >>>>>>> configuration in a future KIP.
> > >>>>>>>>>
> > >>>>>>>>> Best Regards,
> > >>>>>>>>> Jiunn-Yang
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
> > >>>>>>>>>>
> > >>>>>>>>>> hi Jun
> > >>>>>>>>>>
> > >>>>>>>>>> I see what you mean now. The proposal from me is listed below:
> > >>>>>>>>>>
> > >>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default value
> of
> > >>>>>>> earliest. It fixes the data loss from both by_duration and
> latest,
> > and
> > >>>>> it
> > >>>>>>> does not change the logic of auto.offset.reset=earliest.
> > >>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
> > >>>>>>> configuration. auto.offset.reset.new.partitions=earliest already
> > >>>>>>> addresses the issue, and we can discuss the use cases of other
> > values
> > >>>>> in a
> > >>>>>>> separate KIP.
> > >>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> > >>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all for
> > >>>>>>> consistency.
> > >>>>>>>>>>
> > >>>>>>>>>> WDYT?
> > >>>>>>>>>>
> > >>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> > >>>>>>>>>>> Hi, Chia-Ping,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks for the reply.
> > >>>>>>>>>>>
> > >>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka
> topic
> > is
> > >>>>>>> expanded
> > >>>>>>>>>>> with new partitions, consumers using the latest auto offset
> > reset
> > >>>>>>> policy
> > >>>>>>>>>>> will silently miss all records produced to those partitions
> > before
> > >>>>> the
> > >>>>>>>>>>> consumer discovers them.". If a user sets
> > >>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss
> issue
> > >>> could
> > >>>>>>> also
> > >>>>>>>>>>> happen, right?
> > >>>>>>>>>>>
> > >>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will take
> > the
> > >>>>> same
> > >>>>>>>>>>> values as auto.offset.reset. So a user could set it
> > by_duration if
> > >>>>>>> needed.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Jun
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> > >>> chia7712@apache.org
> > >>>>>>
> > >>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> hi Jun
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the feedback. I might be missing something
> > important
> > >>>>> from
> > >>>>>>> your
> > >>>>>>>>>>>> suggestion, so please bear with me as I try to clarify with
> a
> > few
> > >>>>>>> questions:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. Is there a strong use case for extending this logic to
> > other
> > >>>>> reset
> > >>>>>>>>>>>> policies? Unlike latest, policies like earliest or
> by_duration
> > >>>>> don't
> > >>>>>>> seem
> > >>>>>>>>>>>> to suffer from the same silent data loss issue when a
> > partition
> > >>> is
> > >>>>>>> expanded.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. What values would we expect users to configure for
> > >>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to
> earliest
> > or
> > >>>>>>> latest,
> > >>>>>>>>>>>> we might run into the exact same edge cases. For example,
> if a
> > >>>>>>> consumer is
> > >>>>>>>>>>>> offline for a while and a new partition is created during
> that
> > >>>>>>> downtime,
> > >>>>>>>>>>>> the user might actually want to skip to latest when
> resuming,
> > >>>>> rather
> > >>>>>>> than
> > >>>>>>>>>>>> reading from earliest just because the partition is
> > technically
> > >>>>>>> "new" to
> > >>>>>>>>>>>> the group.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This is exactly why we opted for introducing a max.age
> > threshold.
> > >>>>> It
> > >>>>>>> gives
> > >>>>>>>>>>>> users a time-bound way to define what is genuinely "hot/new"
> > and
> > >>>>>>> what is
> > >>>>>>>>>>>> just an old partition they haven't seen yet.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Best,
> > >>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > >>>>>>>>>>>>> Hi, Jiunn-Yang,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for the KIP.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It
> only
> > >>>>>>> applies when
> > >>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> > >>> motivation
> > >>>>>>>>>>>> equally
> > >>>>>>>>>>>>> applies when auto.offset.reset is set to other values like
> > >>>>>>> by_duration.
> > >>>>>>>>>>>> The
> > >>>>>>>>>>>>> intention is that we want to have a separate way to control
> > >>> newly
> > >>>>>>> created
> > >>>>>>>>>>>>> partitions vs existing partitions when the group starts.
> > Have we
> > >>>>>>>>>>>> considered
> > >>>>>>>>>>>>> adding a new config like auto.offset.reset.new.partitions?
> > If
> > >>>>> this
> > >>>>>>> new
> > >>>>>>>>>>>>> config is not set, the offset reset policy defaults to the
> > >>> policy
> > >>>>>>> used
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>> existing partitions. The user could set it explicitly to
> > >>> customize
> > >>>>>>> the
> > >>>>>>>>>>>>> behavior for new partitions.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Jun
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com>
> > wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I’d like to manually bump this thread.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks for the feedback.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> DJ01/DJ02:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> > PartitionMetadata
> > >>>>>>> struct
> > >>>>>>>>>>>>>> gains a new
> > >>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> > server-side
> > >>>>> by
> > >>>>>>> the
> > >>>>>>>>>>>>>> broker as
> > >>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> > MembershipManager
> > >>>>>>> detects
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>> newly assigned
> > >>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for the
> > >>>>> affected
> > >>>>>>>>>>>> topic
> > >>>>>>>>>>>>>> and forces a fresh MetadataRequest
> > >>>>>>>>>>>>>>> before making the offset reset decision, even if the
> topic
> > ID
> > >>> is
> > >>>>>>>>>>>> already
> > >>>>>>>>>>>>>> in the cache.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> MB0:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> > >>>>>>> MetadataResponse
> > >>>>>>>>>>>>>> version via the
> > >>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> > negotiated
> > >>>>>>>>>>>> version is
> > >>>>>>>>>>>>>> unsupported, the consumer
> > >>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at all
> and
> > >>> can
> > >>>>>>>>>>>> throw an
> > >>>>>>>>>>>>>> UnsupportedVersionException
> > >>>>>>>>>>>>>>> immediately, rather than silently falling back to latest
> > and
> > >>>>>>> risking
> > >>>>>>>>>>>>>> data loss without any operator-visible signal.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> MB1/MB2/MB3:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04
> > 寫道:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> hi David
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I agree with the direction of moving the 'age'
> resolution
> > >>> from
> > >>>>>>> the
> > >>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control
> plane
> > >>>>> clean.
> > >>>>>>> The
> > >>>>>>>>>>>> main
> > >>>>>>>>>>>>>> trade-off, as we noted before, is introducing inter-broker
> > >>> clock
> > >>>>>>> skew.
> > >>>>>>>>>>>> The
> > >>>>>>>>>>>>>> Group Coordinator approach provided a single source of
> truth
> > >>> for
> > >>>>>>> time.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> However, realistically, this time skew should be
> > negligible.
> > >>>>>>> Given
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>> the max.age threshold will likely be configured in minutes
> > or
> > >>>>>>> hours, a
> > >>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't
> > impact
> > >>>>> the
> > >>>>>>>>>>>>>> fallback decision.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於
> 2026年4月29日
> > >>>>> 下午3:29
> > >>>>>>> 寫道:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Thanks for the KIP!
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> > conversation
> > >>>>> but I
> > >>>>>>>>>>>> took a
> > >>>>>>>>>>>>>>>>> quick look at this one.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> > >>>>>>>>>>>>>> ConsumerGroupHeartbeat
> > >>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the
> > KIP
> > >>>>>>> states
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>> partition ages are returned only when this boolean is
> > set.
> > >>>>>>>>>>>> Implicitly,
> > >>>>>>>>>>>>>> this
> > >>>>>>>>>>>>>>>>> means that when the consumer receives a new partition,
> it
> > >>> will
> > >>>>>>>>>>>> issue a
> > >>>>>>>>>>>>>> new
> > >>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages. Is
> > my
> > >>>>>>>>>>>>>> understanding
> > >>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also
> > explain
> > >>>>>>> how it
> > >>>>>>>>>>>>>> fits
> > >>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch
> offsets,
> > >>>>> etc.).
> > >>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> > >>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for
> > this
> > >>>>> given
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>> a new
> > >>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could
> simply
> > >>>>>>> include
> > >>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious about
> > not
> > >>>>>>>>>>>> overloading
> > >>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts.
> > The
> > >>>>> API
> > >>>>>>> is
> > >>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>> control plane API for assigning or revoking partitions.
> > The
> > >>>>> fact
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>> we
> > >>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams API
> > also
> > >>>>>>> suggests
> > >>>>>>>>>>>>>>>>> something is not quite right. What would we do if we
> > want to
> > >>>>>>>>>>>> support
> > >>>>>>>>>>>>>>>>> Streams in the future?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>> David
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via
> > dev
> > >>> <
> > >>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi Jiunn,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about the
> > gap.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> > RequestPartitionAges
> > >>>>>>> field.
> > >>>>>>>>>>>> Can a
> > >>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> > >>>>>>>>>>>> TopicPartitions)
> > >>>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>> used here and avoid version bump?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended
> value
> > or
> > >>> a
> > >>>>>>>>>>>> ConfigDef
> > >>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> > >>>>> metadata.max.age.ms
> > >>>>>>> ?
> > >>>>>>>>>>>>>> Sizing
> > >>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka
> Streams,
> > >>>>> would
> > >>>>>>> it
> > >>>>>>>>>>>> be
> > >>>>>>>>>>>>>> better
> > >>>>>>>>>>>>>>>>>> to add this new config
> auto.offset.reset.latest.max.age
> > to
> > >>>>> the
> > >>>>>>>>>>>>>>>>>> StreamsConfig block list
> > >>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > >>>>>>>>>>>>>> for a
> > >>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is the
> > most
> > >>>>>>>>>>>> familiar
> > >>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
> > configure
> > >>>>>>> it. Or
> > >>>>>>>>>>>>>> may be
> > >>>>>>>>>>>>>>>>>> it's not worth it to add.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back
> to
> > >>>>>>> earliest"
> > >>>>>>>>>>>>>> reads as
> > >>>>>>>>>>>>>>>>>> if the config were being changed per-partition which
> > isn't
> > >>>>>>>>>>>> supported.
> > >>>>>>>>>>>>>> May
> > >>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves the
> > >>>>> initial
> > >>>>>>>>>>>>>> position to
> > >>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was
> > applied
> > >>>>> to
> > >>>>>>>>>>>> that
> > >>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> > unchanged.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>> Murali
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> > s7133700@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi chia,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Best Regards,
> > >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日
> > 晚上8:03
> > >>>>> 寫道:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time
> > via
> > >>>>> the
> > >>>>>>>>>>>> Admin
> > >>>>>>>>>>>>>> API?
> > >>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose
> and
> > >>>>>>>>>>>> troubleshoot
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>>>>>>> Chia-Ping
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>>>>>>>>>>>>>>>> Hello everyone,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
> > Prevent
> > >>> Hot
> > >>>>>>> Data
> > >>>>>>>>>>>>>> Loss
> > >>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> > >>>>>>>>>>>>>>>>>>>>> <
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> > >>>>>>>>>>>> auto.offset.reset.latest.max.age,
> > >>>>>>>>>>>>>> a
> > >>>>>>>>>>>>>>>>>>> consumer config that lets the
> > >>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded
> (hot)
> > >>>>>>> partitions
> > >>>>>>>>>>>>>> from
> > >>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> > >>>>>>>>>>>>>>>>>>>>> younger than the configured threshold automatically
> > fall
> > >>>>>>> back
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> > >>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
> > historical
> > >>>>>>>>>>>> reprocess.
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Best regards,
> > >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >
> >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Jiunn-Yang,

Thanks for the reply and sorry for the late reply.

JR1. The design of auto.offset.reset.max.age.ms still feels weird to me. It
categorizes partitions as new or existing based on the partition creation
time. Intuitively, the categorization should be based on the group creation
time: all partitions existing when the group is created are existing and
all partitions created after the group creation are new partitions.

Jun



On Tue, Jun 9, 2026 at 8:51 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hi all,
>
> Manually bumping this thread. If there is no further
> discussion, I will close the vote.
>
> Best Regards,
> Jiunn-Yang
>
> > 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> >
> > Hello Jian,
> >
> > Thanks for your feedback,
> >
> > Agreed, partition expansion is a common operational task, not an edge
> > case. I've updated the Motivation section accordingly.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> >> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> >>
> >> Hi Jiunn-Yang:
> >>
> >> Thanks for the KIP. I think it would be useful to clarify that this is a
> >> common scenario rather than an edge case, which further demonstrates the
> >> need for this optimization. For example:
> >> A partition expansion is a common operational task in Kafka: To balance
> >> resource utilization and cost, topics are typically created with a
> moderate
> >> default partition count. However, as traffic grows over time, it is
> often
> >> necessary to increase the number of partitions to accommodate the higher
> >> workload.
> >>
> >> Regards
> >> Jian
> >>
> >> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> >>
> >>> Hello chia,
> >>>
> >>> Thanks for the comments, I have updated the KIP!
> >>>
> >>> Best Regards,
> >>> Jiunn-Yang
> >>>
> >>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> >>>>
> >>>> Hi Jiunn-Yang,
> >>>>
> >>>> Would you mind removing the terms "hot" and "cold" when describing
> >>>> partitions in the KIP? I understand you are using them to describe the
> >>>> "freshness" or the users' need for the records, but applying these
> terms
> >>> to
> >>>> the partition itself feels a bit unnatural.
> >>>>
> >>>> After all, in this scenario, users don't really care whether a
> partition
> >>> is
> >>>> newly expanded or not. Their only expectation is that they won't
> silently
> >>>> lose any live records produced to the topic during their active
> >>> consumption.
> >>>>
> >>>> Best, Chia-Ping
> >>>>
> >>>>
> >>>>
> >>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >>>>
> >>>>> Hello Jun,
> >>>>>
> >>>>> Thanks for the feedback, I have updated the KIP motivation section.
> >>>>>
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> >>>>>>
> >>>>>> Hi, Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the reply. I think we need a stronger motivation for the
> >>> KIP.
> >>>>>>
> >>>>>> The KIP says "The core insight is that not all partitions without a
> >>>>>> committed offset are the same. A newly expanded partition (hot) is
> >>>>>> fundamentally different from a partition the consumer has never seen
> >>>>>> because it predates the group (cold)." Why is the hot partition
> >>>>>> fundamentally different from the cold?
> >>>>>>
> >>>>>> The KIP says "The existing by_duration policy is also insufficient
> >>>>> because:
> >>>>>>
> >>>>>> - The calculated seek time (now() - duration) varies across nodes
> due
> >>>>> to
> >>>>>> clock skew. To be safe, users must set an overly large duration,
> >>>>> causing
> >>>>>> unnecessary reprocessing.
> >>>>>> - On network errors, the client recalculates the seek time on retry,
> >>>>>> shifting the target timestamp forward and risking data loss."
> >>>>>>
> >>>>>> However, both of these situations are rare. If these issues persist,
> >>> more
> >>>>>> severe problems likely exist elsewhere. Rare situations don't need a
> >>>>> common
> >>>>>> solution. If users care about those rare situations, they can
> implement
> >>>>>> customized logic using
> >>> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>>
> >>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hello chia,
> >>>>>>>
> >>>>>>> Thanks for the feedback,
> >>>>>>>
> >>>>>>>> If the creation time exists, the returned value should always be
> >>>>> greater
> >>>>>>> than or equal to zero, right?
> >>>>>>> I have explicitly mentioned this in the KIP.
> >>>>>>>
> >>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>  field
> >>>>>>> absent    UnsupportedVersionException
> >>>>>>>
> >>>>>>> The earliest point at which we can detect the version mismatch is
> >>> during
> >>>>>>> the
> >>>>>>> first metadata fetch after assignment, which occurs inside poll().
> >>>>>>> Therefore, the
> >>>>>>> user would encounter an UnsupportedVersionException from poll().
> I’ll
> >>>>>>> clarify this in the KIP.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> >>>>>>>>
> >>>>>>>> hi Jiunn
> >>>>>>>>
> >>>>>>>>> PartitionAgeMs (int64, default -1): The age of this partition in
> >>>>>>> milliseconds, computed server-side by the broker as
> >>> broker_current_time
> >>>>> -
> >>>>>>> partition_creation_time. Returns -1 if the broker does not support
> >>> this
> >>>>>>> feature or the partition creation time is unknown.
> >>>>>>>>
> >>>>>>>> If the creation time exists, the returned value should always be
> >>>>> greater
> >>>>>>> than or equal to zero, right?
> >>>>>>>>
> >>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any
>  field
> >>>>>>> absent    UnsupportedVersionException
> >>>>>>>>
> >>>>>>>> Will user encounter UnsupportedVersionException when calling
> >>> `poll()`?
> >>>>>>>>
> >>>>>>>> Best,
> >>>>>>>> Chia-Ping
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>>>>> Hello Jun, chia,
> >>>>>>>>>
> >>>>>>>>> I've updated KIP-1327 with a design change based on the
> discussion
> >>>>>>>>> feedback.
> >>>>>>>>>
> >>>>>>>>> The updated design decouples the new-partition reset behavior
> from
> >>>>>>>>> the base auto.offset.reset policy:
> >>>>>>>>>
> >>>>>>>>> - auto.offset.reset.max.age.ms now applies to all
> auto.offset.reset
> >>>>>>> values
> >>>>>>>>> (latest, earliest, by_duration, none).
> >>>>>>>>> - For new ("hot") partitions, the consumer resets to
> >>>>>>> auto.offset.reset.new.partitions
> >>>>>>>>> config setting
> >>>>>>>>> - For existing ("cold") partitions, the base auto.offset.reset
> >>> policy
> >>>>>>> continues
> >>>>>>>>> to apply unchanged.
> >>>>>>>>> - The new-partition reset behavior is represented by a separate
> >>>>>>> internal config
> >>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to earliest).
> >>> This
> >>>>>>> decoupled design makes
> >>>>>>>>> it straightforward to promote the behavior to a public
> user-facing
> >>>>>>> configuration in a future KIP.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
> >>>>>>>>>>
> >>>>>>>>>> hi Jun
> >>>>>>>>>>
> >>>>>>>>>> I see what you mean now. The proposal from me is listed below:
> >>>>>>>>>>
> >>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default value of
> >>>>>>> earliest. It fixes the data loss from both by_duration and latest,
> and
> >>>>> it
> >>>>>>> does not change the logic of auto.offset.reset=earliest.
> >>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
> >>>>>>> configuration. auto.offset.reset.new.partitions=earliest already
> >>>>>>> addresses the issue, and we can discuss the use cases of other
> values
> >>>>> in a
> >>>>>>> separate KIP.
> >>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> >>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all for
> >>>>>>> consistency.
> >>>>>>>>>>
> >>>>>>>>>> WDYT?
> >>>>>>>>>>
> >>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>>>>> Hi, Chia-Ping,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the reply.
> >>>>>>>>>>>
> >>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka topic
> is
> >>>>>>> expanded
> >>>>>>>>>>> with new partitions, consumers using the latest auto offset
> reset
> >>>>>>> policy
> >>>>>>>>>>> will silently miss all records produced to those partitions
> before
> >>>>> the
> >>>>>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss issue
> >>> could
> >>>>>>> also
> >>>>>>>>>>> happen, right?
> >>>>>>>>>>>
> >>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will take
> the
> >>>>> same
> >>>>>>>>>>> values as auto.offset.reset. So a user could set it
> by_duration if
> >>>>>>> needed.
> >>>>>>>>>>>
> >>>>>>>>>>> Jun
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> >>> chia7712@apache.org
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> hi Jun
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the feedback. I might be missing something
> important
> >>>>> from
> >>>>>>> your
> >>>>>>>>>>>> suggestion, so please bear with me as I try to clarify with a
> few
> >>>>>>> questions:
> >>>>>>>>>>>>
> >>>>>>>>>>>> 1. Is there a strong use case for extending this logic to
> other
> >>>>> reset
> >>>>>>>>>>>> policies? Unlike latest, policies like earliest or by_duration
> >>>>> don't
> >>>>>>> seem
> >>>>>>>>>>>> to suffer from the same silent data loss issue when a
> partition
> >>> is
> >>>>>>> expanded.
> >>>>>>>>>>>>
> >>>>>>>>>>>> 2. What values would we expect users to configure for
> >>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to earliest
> or
> >>>>>>> latest,
> >>>>>>>>>>>> we might run into the exact same edge cases. For example, if a
> >>>>>>> consumer is
> >>>>>>>>>>>> offline for a while and a new partition is created during that
> >>>>>>> downtime,
> >>>>>>>>>>>> the user might actually want to skip to latest when resuming,
> >>>>> rather
> >>>>>>> than
> >>>>>>>>>>>> reading from earliest just because the partition is
> technically
> >>>>>>> "new" to
> >>>>>>>>>>>> the group.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This is exactly why we opted for introducing a max.age
> threshold.
> >>>>> It
> >>>>>>> gives
> >>>>>>>>>>>> users a time-bound way to define what is genuinely "hot/new"
> and
> >>>>>>> what is
> >>>>>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only
> >>>>>>> applies when
> >>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> >>> motivation
> >>>>>>>>>>>> equally
> >>>>>>>>>>>>> applies when auto.offset.reset is set to other values like
> >>>>>>> by_duration.
> >>>>>>>>>>>> The
> >>>>>>>>>>>>> intention is that we want to have a separate way to control
> >>> newly
> >>>>>>> created
> >>>>>>>>>>>>> partitions vs existing partitions when the group starts.
> Have we
> >>>>>>>>>>>> considered
> >>>>>>>>>>>>> adding a new config like auto.offset.reset.new.partitions?
> If
> >>>>> this
> >>>>>>> new
> >>>>>>>>>>>>> config is not set, the offset reset policy defaults to the
> >>> policy
> >>>>>>> used
> >>>>>>>>>>>> for
> >>>>>>>>>>>>> existing partitions. The user could set it explicitly to
> >>> customize
> >>>>>>> the
> >>>>>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Jun
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com>
> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The
> PartitionMetadata
> >>>>>>> struct
> >>>>>>>>>>>>>> gains a new
> >>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed
> server-side
> >>>>> by
> >>>>>>> the
> >>>>>>>>>>>>>> broker as
> >>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when
> MembershipManager
> >>>>>>> detects
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>> newly assigned
> >>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for the
> >>>>> affected
> >>>>>>>>>>>> topic
> >>>>>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>>>>> before making the offset reset decision, even if the topic
> ID
> >>> is
> >>>>>>>>>>>> already
> >>>>>>>>>>>>>> in the cache.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> MB0:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
> >>>>>>> MetadataResponse
> >>>>>>>>>>>>>> version via the
> >>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the
> negotiated
> >>>>>>>>>>>> version is
> >>>>>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at all and
> >>> can
> >>>>>>>>>>>> throw an
> >>>>>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>>>>> immediately, rather than silently falling back to latest
> and
> >>>>>>> risking
> >>>>>>>>>>>>>> data loss without any operator-visible signal.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04
> 寫道:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I agree with the direction of moving the 'age' resolution
> >>> from
> >>>>>>> the
> >>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control plane
> >>>>> clean.
> >>>>>>> The
> >>>>>>>>>>>> main
> >>>>>>>>>>>>>> trade-off, as we noted before, is introducing inter-broker
> >>> clock
> >>>>>>> skew.
> >>>>>>>>>>>> The
> >>>>>>>>>>>>>> Group Coordinator approach provided a single source of truth
> >>> for
> >>>>>>> time.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> However, realistically, this time skew should be
> negligible.
> >>>>>>> Given
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>> the max.age threshold will likely be configured in minutes
> or
> >>>>>>> hours, a
> >>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't
> impact
> >>>>> the
> >>>>>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日
> >>>>> 下午3:29
> >>>>>>> 寫道:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous
> conversation
> >>>>> but I
> >>>>>>>>>>>> took a
> >>>>>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the
> KIP
> >>>>>>> states
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> partition ages are returned only when this boolean is
> set.
> >>>>>>>>>>>> Implicitly,
> >>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> means that when the consumer receives a new partition, it
> >>> will
> >>>>>>>>>>>> issue a
> >>>>>>>>>>>>>> new
> >>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages. Is
> my
> >>>>>>>>>>>>>> understanding
> >>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also
> explain
> >>>>>>> how it
> >>>>>>>>>>>>>> fits
> >>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets,
> >>>>> etc.).
> >>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for
> this
> >>>>> given
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>> a new
> >>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could simply
> >>>>>>> include
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious about
> not
> >>>>>>>>>>>> overloading
> >>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts.
> The
> >>>>> API
> >>>>>>> is
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>> control plane API for assigning or revoking partitions.
> The
> >>>>> fact
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams API
> also
> >>>>>>> suggests
> >>>>>>>>>>>>>>>>> something is not quite right. What would we do if we
> want to
> >>>>>>>>>>>> support
> >>>>>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via
> dev
> >>> <
> >>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about the
> gap.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for
> RequestPartitionAges
> >>>>>>> field.
> >>>>>>>>>>>> Can a
> >>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> >>>>>>>>>>>> TopicPartitions)
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended value
> or
> >>> a
> >>>>>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>>>>> validator? Probably it should based on the
> >>>>> metadata.max.age.ms
> >>>>>>> ?
> >>>>>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams,
> >>>>> would
> >>>>>>> it
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>> better
> >>>>>>>>>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age
> to
> >>>>> the
> >>>>>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>>>>> for a
> >>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is the
> most
> >>>>>>>>>>>> familiar
> >>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly
> configure
> >>>>>>> it. Or
> >>>>>>>>>>>>>> may be
> >>>>>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to
> >>>>>>> earliest"
> >>>>>>>>>>>>>> reads as
> >>>>>>>>>>>>>>>>>> if the config were being changed per-partition which
> isn't
> >>>>>>>>>>>> supported.
> >>>>>>>>>>>>>> May
> >>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves the
> >>>>> initial
> >>>>>>>>>>>>>> position to
> >>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was
> applied
> >>>>> to
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is
> unchanged.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <
> s7133700@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日
> 晚上8:03
> >>>>> 寫道:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time
> via
> >>>>> the
> >>>>>>>>>>>> Admin
> >>>>>>>>>>>>>> API?
> >>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose and
> >>>>>>>>>>>> troubleshoot
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327
> Prevent
> >>> Hot
> >>>>>>> Data
> >>>>>>>>>>>>>> Loss
> >>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot)
> >>>>>>> partitions
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>>>>> younger than the configured threshold automatically
> fall
> >>>>>>> back
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full
> historical
> >>>>>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hi all,

Manually bumping this thread. If there is no further 
discussion, I will close the vote.

Best Regards,
Jiunn-Yang

> 黃竣陽 <s7...@gmail.com> 於 2026年6月1日 晚上7:16 寫道:
> 
> Hello Jian,
> 
> Thanks for your feedback,
> 
> Agreed, partition expansion is a common operational task, not an edge 
> case. I've updated the Motivation section accordingly.
> 
> Best Regards,
> Jiunn-Yang
> 
>> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
>> 
>> Hi Jiunn-Yang:
>> 
>> Thanks for the KIP. I think it would be useful to clarify that this is a
>> common scenario rather than an edge case, which further demonstrates the
>> need for this optimization. For example:
>> A partition expansion is a common operational task in Kafka: To balance
>> resource utilization and cost, topics are typically created with a moderate
>> default partition count. However, as traffic grows over time, it is often
>> necessary to increase the number of partitions to accommodate the higher
>> workload.
>> 
>> Regards
>> Jian
>> 
>> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
>> 
>>> Hello chia,
>>> 
>>> Thanks for the comments, I have updated the KIP!
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>>> 
>>>> Hi Jiunn-Yang,
>>>> 
>>>> Would you mind removing the terms "hot" and "cold" when describing
>>>> partitions in the KIP? I understand you are using them to describe the
>>>> "freshness" or the users' need for the records, but applying these terms
>>> to
>>>> the partition itself feels a bit unnatural.
>>>> 
>>>> After all, in this scenario, users don't really care whether a partition
>>> is
>>>> newly expanded or not. Their only expectation is that they won't silently
>>>> lose any live records produced to the topic during their active
>>> consumption.
>>>> 
>>>> Best, Chia-Ping
>>>> 
>>>> 
>>>> 
>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>>> 
>>>>> Hello Jun,
>>>>> 
>>>>> Thanks for the feedback, I have updated the KIP motivation section.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
>>>>>> 
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the reply. I think we need a stronger motivation for the
>>> KIP.
>>>>>> 
>>>>>> The KIP says "The core insight is that not all partitions without a
>>>>>> committed offset are the same. A newly expanded partition (hot) is
>>>>>> fundamentally different from a partition the consumer has never seen
>>>>>> because it predates the group (cold)." Why is the hot partition
>>>>>> fundamentally different from the cold?
>>>>>> 
>>>>>> The KIP says "The existing by_duration policy is also insufficient
>>>>> because:
>>>>>> 
>>>>>> - The calculated seek time (now() - duration) varies across nodes due
>>>>> to
>>>>>> clock skew. To be safe, users must set an overly large duration,
>>>>> causing
>>>>>> unnecessary reprocessing.
>>>>>> - On network errors, the client recalculates the seek time on retry,
>>>>>> shifting the target timestamp forward and risking data loss."
>>>>>> 
>>>>>> However, both of these situations are rare. If these issues persist,
>>> more
>>>>>> severe problems likely exist elsewhere. Rare situations don't need a
>>>>> common
>>>>>> solution. If users care about those rare situations, they can implement
>>>>>> customized logic using
>>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> 
>>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hello chia,
>>>>>>> 
>>>>>>> Thanks for the feedback,
>>>>>>> 
>>>>>>>> If the creation time exists, the returned value should always be
>>>>> greater
>>>>>>> than or equal to zero, right?
>>>>>>> I have explicitly mentioned this in the KIP.
>>>>>>> 
>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any     field
>>>>>>> absent    UnsupportedVersionException
>>>>>>> 
>>>>>>> The earliest point at which we can detect the version mismatch is
>>> during
>>>>>>> the
>>>>>>> first metadata fetch after assignment, which occurs inside poll().
>>>>>>> Therefore, the
>>>>>>> user would encounter an UnsupportedVersionException from poll(). I’ll
>>>>>>> clarify this in the KIP.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
>>>>>>>> 
>>>>>>>> hi Jiunn
>>>>>>>> 
>>>>>>>>> PartitionAgeMs (int64, default -1): The age of this partition in
>>>>>>> milliseconds, computed server-side by the broker as
>>> broker_current_time
>>>>> -
>>>>>>> partition_creation_time. Returns -1 if the broker does not support
>>> this
>>>>>>> feature or the partition creation time is unknown.
>>>>>>>> 
>>>>>>>> If the creation time exists, the returned value should always be
>>>>> greater
>>>>>>> than or equal to zero, right?
>>>>>>>> 
>>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any     field
>>>>>>> absent    UnsupportedVersionException
>>>>>>>> 
>>>>>>>> Will user encounter UnsupportedVersionException when calling
>>> `poll()`?
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Chia-Ping
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>>> Hello Jun, chia,
>>>>>>>>> 
>>>>>>>>> I've updated KIP-1327 with a design change based on the discussion
>>>>>>>>> feedback.
>>>>>>>>> 
>>>>>>>>> The updated design decouples the new-partition reset behavior from
>>>>>>>>> the base auto.offset.reset policy:
>>>>>>>>> 
>>>>>>>>> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset
>>>>>>> values
>>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>>> auto.offset.reset.new.partitions
>>>>>>>>> config setting
>>>>>>>>> - For existing ("cold") partitions, the base auto.offset.reset
>>> policy
>>>>>>> continues
>>>>>>>>> to apply unchanged.
>>>>>>>>> - The new-partition reset behavior is represented by a separate
>>>>>>> internal config
>>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to earliest).
>>> This
>>>>>>> decoupled design makes
>>>>>>>>> it straightforward to promote the behavior to a public user-facing
>>>>>>> configuration in a future KIP.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
>>>>>>>>>> 
>>>>>>>>>> hi Jun
>>>>>>>>>> 
>>>>>>>>>> I see what you mean now. The proposal from me is listed below:
>>>>>>>>>> 
>>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default value of
>>>>>>> earliest. It fixes the data loss from both by_duration and latest, and
>>>>> it
>>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
>>>>>>> configuration. auto.offset.reset.new.partitions=earliest already
>>>>>>> addresses the issue, and we can discuss the use cases of other values
>>>>> in a
>>>>>>> separate KIP.
>>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
>>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all for
>>>>>>> consistency.
>>>>>>>>>> 
>>>>>>>>>> WDYT?
>>>>>>>>>> 
>>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>> 
>>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka topic is
>>>>>>> expanded
>>>>>>>>>>> with new partitions, consumers using the latest auto offset reset
>>>>>>> policy
>>>>>>>>>>> will silently miss all records produced to those partitions before
>>>>> the
>>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss issue
>>> could
>>>>>>> also
>>>>>>>>>>> happen, right?
>>>>>>>>>>> 
>>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will take the
>>>>> same
>>>>>>>>>>> values as auto.offset.reset. So a user could set it by_duration if
>>>>>>> needed.
>>>>>>>>>>> 
>>>>>>>>>>> Jun
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>>> chia7712@apache.org
>>>>>> 
>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> hi Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the feedback. I might be missing something important
>>>>> from
>>>>>>> your
>>>>>>>>>>>> suggestion, so please bear with me as I try to clarify with a few
>>>>>>> questions:
>>>>>>>>>>>> 
>>>>>>>>>>>> 1. Is there a strong use case for extending this logic to other
>>>>> reset
>>>>>>>>>>>> policies? Unlike latest, policies like earliest or by_duration
>>>>> don't
>>>>>>> seem
>>>>>>>>>>>> to suffer from the same silent data loss issue when a partition
>>> is
>>>>>>> expanded.
>>>>>>>>>>>> 
>>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to earliest or
>>>>>>> latest,
>>>>>>>>>>>> we might run into the exact same edge cases. For example, if a
>>>>>>> consumer is
>>>>>>>>>>>> offline for a while and a new partition is created during that
>>>>>>> downtime,
>>>>>>>>>>>> the user might actually want to skip to latest when resuming,
>>>>> rather
>>>>>>> than
>>>>>>>>>>>> reading from earliest just because the partition is technically
>>>>>>> "new" to
>>>>>>>>>>>> the group.
>>>>>>>>>>>> 
>>>>>>>>>>>> This is exactly why we opted for introducing a max.age threshold.
>>>>> It
>>>>>>> gives
>>>>>>>>>>>> users a time-bound way to define what is genuinely "hot/new" and
>>>>>>> what is
>>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>> 
>>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only
>>>>>>> applies when
>>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>>> motivation
>>>>>>>>>>>> equally
>>>>>>>>>>>>> applies when auto.offset.reset is set to other values like
>>>>>>> by_duration.
>>>>>>>>>>>> The
>>>>>>>>>>>>> intention is that we want to have a separate way to control
>>> newly
>>>>>>> created
>>>>>>>>>>>>> partitions vs existing partitions when the group starts. Have we
>>>>>>>>>>>> considered
>>>>>>>>>>>>> adding a new config like auto.offset.reset.new.partitions? If
>>>>> this
>>>>>>> new
>>>>>>>>>>>>> config is not set, the offset reset policy defaults to the
>>> policy
>>>>>>> used
>>>>>>>>>>>> for
>>>>>>>>>>>>> existing partitions. The user could set it explicitly to
>>> customize
>>>>>>> the
>>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Jun
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata
>>>>>>> struct
>>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed server-side
>>>>> by
>>>>>>> the
>>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when MembershipManager
>>>>>>> detects
>>>>>>>>>>>> a
>>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for the
>>>>> affected
>>>>>>>>>>>> topic
>>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>>> before making the offset reset decision, even if the topic ID
>>> is
>>>>>>>>>>>> already
>>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>>> MetadataResponse
>>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the negotiated
>>>>>>>>>>>> version is
>>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at all and
>>> can
>>>>>>>>>>>> throw an
>>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>>> immediately, rather than silently falling back to latest and
>>>>>>> risking
>>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I agree with the direction of moving the 'age' resolution
>>> from
>>>>>>> the
>>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control plane
>>>>> clean.
>>>>>>> The
>>>>>>>>>>>> main
>>>>>>>>>>>>>> trade-off, as we noted before, is introducing inter-broker
>>> clock
>>>>>>> skew.
>>>>>>>>>>>> The
>>>>>>>>>>>>>> Group Coordinator approach provided a single source of truth
>>> for
>>>>>>> time.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> However, realistically, this time skew should be negligible.
>>>>>>> Given
>>>>>>>>>>>> that
>>>>>>>>>>>>>> the max.age threshold will likely be configured in minutes or
>>>>>>> hours, a
>>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't impact
>>>>> the
>>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日
>>>>> 下午3:29
>>>>>>> 寫道:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous conversation
>>>>> but I
>>>>>>>>>>>> took a
>>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the KIP
>>>>>>> states
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> partition ages are returned only when this boolean is set.
>>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> means that when the consumer receives a new partition, it
>>> will
>>>>>>>>>>>> issue a
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages. Is my
>>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also explain
>>>>>>> how it
>>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets,
>>>>> etc.).
>>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this
>>>>> given
>>>>>>>>>>>> that
>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could simply
>>>>>>> include
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious about not
>>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The
>>>>> API
>>>>>>> is
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> control plane API for assigning or revoking partitions. The
>>>>> fact
>>>>>>>>>>>> that
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams API also
>>>>>>> suggests
>>>>>>>>>>>>>>>>> something is not quite right. What would we do if we want to
>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev
>>> <
>>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about the gap.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges
>>>>>>> field.
>>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
>>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended value or
>>> a
>>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>>> metadata.max.age.ms
>>>>>>> ?
>>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams,
>>>>> would
>>>>>>> it
>>>>>>>>>>>> be
>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to
>>>>> the
>>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is the most
>>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly configure
>>>>>>> it. Or
>>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to
>>>>>>> earliest"
>>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>>> if the config were being changed per-partition which isn't
>>>>>>>>>>>> supported.
>>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves the
>>>>> initial
>>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was applied
>>>>> to
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is unchanged.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03
>>>>> 寫道:
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time via
>>>>> the
>>>>>>>>>>>> Admin
>>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose and
>>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent
>>> Hot
>>>>>>> Data
>>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>> 
>>>>> 
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot)
>>>>>>> partitions
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>>> younger than the configured threshold automatically fall
>>>>>>> back
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full historical
>>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello Jian,

Thanks for your feedback,

Agreed, partition expansion is a common operational task, not an edge 
case. I've updated the Motivation section accordingly.

Best Regards,
Jiunn-Yang

> jian fu <fu...@gmail.com> 於 2026年6月1日 下午5:49 寫道:
> 
> Hi Jiunn-Yang:
> 
> Thanks for the KIP. I think it would be useful to clarify that this is a
> common scenario rather than an edge case, which further demonstrates the
> need for this optimization. For example:
> A partition expansion is a common operational task in Kafka: To balance
> resource utilization and cost, topics are typically created with a moderate
> default partition count. However, as traffic grows over time, it is often
> necessary to increase the number of partitions to accommodate the higher
> workload.
> 
> Regards
> Jian
> 
> 黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:
> 
>> Hello chia,
>> 
>> Thanks for the comments, I have updated the KIP!
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
>>> 
>>> Hi Jiunn-Yang,
>>> 
>>> Would you mind removing the terms "hot" and "cold" when describing
>>> partitions in the KIP? I understand you are using them to describe the
>>> "freshness" or the users' need for the records, but applying these terms
>> to
>>> the partition itself feels a bit unnatural.
>>> 
>>> After all, in this scenario, users don't really care whether a partition
>> is
>>> newly expanded or not. Their only expectation is that they won't silently
>>> lose any live records produced to the topic during their active
>> consumption.
>>> 
>>> Best, Chia-Ping
>>> 
>>> 
>>> 
>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
>>> 
>>>> Hello Jun,
>>>> 
>>>> Thanks for the feedback, I have updated the KIP motivation section.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
>>>>> 
>>>>> Hi, Jiunn-Yang,
>>>>> 
>>>>> Thanks for the reply. I think we need a stronger motivation for the
>> KIP.
>>>>> 
>>>>> The KIP says "The core insight is that not all partitions without a
>>>>> committed offset are the same. A newly expanded partition (hot) is
>>>>> fundamentally different from a partition the consumer has never seen
>>>>> because it predates the group (cold)." Why is the hot partition
>>>>> fundamentally different from the cold?
>>>>> 
>>>>> The KIP says "The existing by_duration policy is also insufficient
>>>> because:
>>>>> 
>>>>> - The calculated seek time (now() - duration) varies across nodes due
>>>> to
>>>>> clock skew. To be safe, users must set an overly large duration,
>>>> causing
>>>>> unnecessary reprocessing.
>>>>> - On network errors, the client recalculates the seek time on retry,
>>>>> shifting the target timestamp forward and risking data loss."
>>>>> 
>>>>> However, both of these situations are rare. If these issues persist,
>> more
>>>>> severe problems likely exist elsewhere. Rare situations don't need a
>>>> common
>>>>> solution. If users care about those rare situations, they can implement
>>>>> customized logic using
>> ConsumerRebalanceListener.onPartitionsAssigned().
>>>>> 
>>>>> Jun
>>>>> 
>>>>> 
>>>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>>> Hello chia,
>>>>>> 
>>>>>> Thanks for the feedback,
>>>>>> 
>>>>>>> If the creation time exists, the returned value should always be
>>>> greater
>>>>>> than or equal to zero, right?
>>>>>> I have explicitly mentioned this in the KIP.
>>>>>> 
>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any     field
>>>>>> absent    UnsupportedVersionException
>>>>>> 
>>>>>> The earliest point at which we can detect the version mismatch is
>> during
>>>>>> the
>>>>>> first metadata fetch after assignment, which occurs inside poll().
>>>>>> Therefore, the
>>>>>> user would encounter an UnsupportedVersionException from poll(). I’ll
>>>>>> clarify this in the KIP.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
>>>>>>> 
>>>>>>> hi Jiunn
>>>>>>> 
>>>>>>>> PartitionAgeMs (int64, default -1): The age of this partition in
>>>>>> milliseconds, computed server-side by the broker as
>> broker_current_time
>>>> -
>>>>>> partition_creation_time. Returns -1 if the broker does not support
>> this
>>>>>> feature or the partition creation time is unknown.
>>>>>>> 
>>>>>>> If the creation time exists, the returned value should always be
>>>> greater
>>>>>> than or equal to zero, right?
>>>>>>> 
>>>>>>>> New  Old (MetadataResponse v0–13)    positive        any     field
>>>>>> absent    UnsupportedVersionException
>>>>>>> 
>>>>>>> Will user encounter UnsupportedVersionException when calling
>> `poll()`?
>>>>>>> 
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>> 
>>>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>>>> Hello Jun, chia,
>>>>>>>> 
>>>>>>>> I've updated KIP-1327 with a design change based on the discussion
>>>>>>>> feedback.
>>>>>>>> 
>>>>>>>> The updated design decouples the new-partition reset behavior from
>>>>>>>> the base auto.offset.reset policy:
>>>>>>>> 
>>>>>>>> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset
>>>>>> values
>>>>>>>> (latest, earliest, by_duration, none).
>>>>>>>> - For new ("hot") partitions, the consumer resets to
>>>>>> auto.offset.reset.new.partitions
>>>>>>>> config setting
>>>>>>>> - For existing ("cold") partitions, the base auto.offset.reset
>> policy
>>>>>> continues
>>>>>>>> to apply unchanged.
>>>>>>>> - The new-partition reset behavior is represented by a separate
>>>>>> internal config
>>>>>>>> (auto.offset.reset.new.partitions, currently fixed to earliest).
>> This
>>>>>> decoupled design makes
>>>>>>>> it straightforward to promote the behavior to a public user-facing
>>>>>> configuration in a future KIP.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
>>>>>>>>> 
>>>>>>>>> hi Jun
>>>>>>>>> 
>>>>>>>>> I see what you mean now. The proposal from me is listed below:
>>>>>>>>> 
>>>>>>>>> 1) Add auto.offset.reset.new.partitions with a default value of
>>>>>> earliest. It fixes the data loss from both by_duration and latest, and
>>>> it
>>>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
>>>>>> configuration. auto.offset.reset.new.partitions=earliest already
>>>>>> addresses the issue, and we can discuss the use cases of other values
>>>> in a
>>>>>> separate KIP.
>>>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
>>>>>> auto.offset.reset.latest.max.age.ms, will be applied to all for
>>>>>> consistency.
>>>>>>>>> 
>>>>>>>>> WDYT?
>>>>>>>>> 
>>>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>>>> Hi, Chia-Ping,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the reply.
>>>>>>>>>> 
>>>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka topic is
>>>>>> expanded
>>>>>>>>>> with new partitions, consumers using the latest auto offset reset
>>>>>> policy
>>>>>>>>>> will silently miss all records produced to those partitions before
>>>> the
>>>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss issue
>> could
>>>>>> also
>>>>>>>>>> happen, right?
>>>>>>>>>> 
>>>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will take the
>>>> same
>>>>>>>>>> values as auto.offset.reset. So a user could set it by_duration if
>>>>>> needed.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
>> chia7712@apache.org
>>>>> 
>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> hi Jun
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the feedback. I might be missing something important
>>>> from
>>>>>> your
>>>>>>>>>>> suggestion, so please bear with me as I try to clarify with a few
>>>>>> questions:
>>>>>>>>>>> 
>>>>>>>>>>> 1. Is there a strong use case for extending this logic to other
>>>> reset
>>>>>>>>>>> policies? Unlike latest, policies like earliest or by_duration
>>>> don't
>>>>>> seem
>>>>>>>>>>> to suffer from the same silent data loss issue when a partition
>> is
>>>>>> expanded.
>>>>>>>>>>> 
>>>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>>>> auto.offset.reset.new.partitions? If they set it to earliest or
>>>>>> latest,
>>>>>>>>>>> we might run into the exact same edge cases. For example, if a
>>>>>> consumer is
>>>>>>>>>>> offline for a while and a new partition is created during that
>>>>>> downtime,
>>>>>>>>>>> the user might actually want to skip to latest when resuming,
>>>> rather
>>>>>> than
>>>>>>>>>>> reading from earliest just because the partition is technically
>>>>>> "new" to
>>>>>>>>>>> the group.
>>>>>>>>>>> 
>>>>>>>>>>> This is exactly why we opted for introducing a max.age threshold.
>>>> It
>>>>>> gives
>>>>>>>>>>> users a time-bound way to define what is genuinely "hot/new" and
>>>>>> what is
>>>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>>>> 
>>>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only
>>>>>> applies when
>>>>>>>>>>>> auto.offset.reset is latest. However, it seems that the
>> motivation
>>>>>>>>>>> equally
>>>>>>>>>>>> applies when auto.offset.reset is set to other values like
>>>>>> by_duration.
>>>>>>>>>>> The
>>>>>>>>>>>> intention is that we want to have a separate way to control
>> newly
>>>>>> created
>>>>>>>>>>>> partitions vs existing partitions when the group starts. Have we
>>>>>>>>>>> considered
>>>>>>>>>>>> adding a new config like auto.offset.reset.new.partitions? If
>>>> this
>>>>>> new
>>>>>>>>>>>> config is not set, the offset reset policy defaults to the
>> policy
>>>>>> used
>>>>>>>>>>> for
>>>>>>>>>>>> existing partitions. The user could set it explicitly to
>> customize
>>>>>> the
>>>>>>>>>>>> behavior for new partitions.
>>>>>>>>>>>> 
>>>>>>>>>>>> Jun
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata
>>>>>> struct
>>>>>>>>>>>>> gains a new
>>>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed server-side
>>>> by
>>>>>> the
>>>>>>>>>>>>> broker as
>>>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Also add the consumer heartbeat flow. when MembershipManager
>>>>>> detects
>>>>>>>>>>> a
>>>>>>>>>>>>> newly assigned
>>>>>>>>>>>>>> partition, it explicitly invalidates the metadata for the
>>>> affected
>>>>>>>>>>> topic
>>>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>>>> before making the offset reset decision, even if the topic ID
>> is
>>>>>>>>>>> already
>>>>>>>>>>>>> in the cache.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> MB0:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>>>> MetadataResponse
>>>>>>>>>>>>> version via the
>>>>>>>>>>>>>> ApiVersions negotiation at connection time. If the negotiated
>>>>>>>>>>> version is
>>>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at all and
>> can
>>>>>>>>>>> throw an
>>>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>>>> immediately, rather than silently falling back to latest and
>>>>>> risking
>>>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> hi David
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I agree with the direction of moving the 'age' resolution
>> from
>>>>>> the
>>>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control plane
>>>> clean.
>>>>>> The
>>>>>>>>>>> main
>>>>>>>>>>>>> trade-off, as we noted before, is introducing inter-broker
>> clock
>>>>>> skew.
>>>>>>>>>>> The
>>>>>>>>>>>>> Group Coordinator approach provided a single source of truth
>> for
>>>>>> time.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> However, realistically, this time skew should be negligible.
>>>>>> Given
>>>>>>>>>>> that
>>>>>>>>>>>>> the max.age threshold will likely be configured in minutes or
>>>>>> hours, a
>>>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't impact
>>>> the
>>>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日
>>>> 下午3:29
>>>>>> 寫道:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Sorry, I haven't really followed the previous conversation
>>>> but I
>>>>>>>>>>> took a
>>>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the KIP
>>>>>> states
>>>>>>>>>>> that
>>>>>>>>>>>>>>>> partition ages are returned only when this boolean is set.
>>>>>>>>>>> Implicitly,
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> means that when the consumer receives a new partition, it
>> will
>>>>>>>>>>> issue a
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>> HB request with the boolean set to receive the ages. Is my
>>>>>>>>>>>>> understanding
>>>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also explain
>>>>>> how it
>>>>>>>>>>>>> fits
>>>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets,
>>>> etc.).
>>>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this
>>>> given
>>>>>>>>>>> that
>>>>>>>>>>>>> a new
>>>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could simply
>>>>>> include
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious about not
>>>>>>>>>>> overloading
>>>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The
>>>> API
>>>>>> is
>>>>>>>>>>> a
>>>>>>>>>>>>>>>> control plane API for assigning or revoking partitions. The
>>>> fact
>>>>>>>>>>> that
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> don't want to add it to the corresponding Streams API also
>>>>>> suggests
>>>>>>>>>>>>>>>> something is not quite right. What would we do if we want to
>>>>>>>>>>> support
>>>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>> David
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev
>> <
>>>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about the gap.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges
>>>>>> field.
>>>>>>>>>>> Can a
>>>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
>>>>>>>>>>> TopicPartitions)
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended value or
>> a
>>>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>>>> validator? Probably it should based on the
>>>> metadata.max.age.ms
>>>>>> ?
>>>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams,
>>>> would
>>>>>> it
>>>>>>>>>>> be
>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to
>>>> the
>>>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>>>> for a
>>>>>>>>>>>>>>>>> clear warning, incase users configure it? This is the most
>>>>>>>>>>> familiar
>>>>>>>>>>>>>>>>> consumer config and users might easily mistakenly configure
>>>>>> it. Or
>>>>>>>>>>>>> may be
>>>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to
>>>>>> earliest"
>>>>>>>>>>>>> reads as
>>>>>>>>>>>>>>>>> if the config were being changed per-partition which isn't
>>>>>>>>>>> supported.
>>>>>>>>>>>>> May
>>>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves the
>>>> initial
>>>>>>>>>>>>> position to
>>>>>>>>>>>>>>>>> start offset for that partition" as if earliest was applied
>>>> to
>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> partition only and auto.offset.reset config is unchanged.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03
>>>> 寫道:
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time via
>>>> the
>>>>>>>>>>> Admin
>>>>>>>>>>>>> API?
>>>>>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose and
>>>>>>>>>>> troubleshoot
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent
>> Hot
>>>>>> Data
>>>>>>>>>>>>> Loss
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>> 
>>>> 
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot)
>>>>>> partitions
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>>>> younger than the configured threshold automatically fall
>>>>>> back
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full historical
>>>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by jian fu <fu...@gmail.com>.
Hi Jiunn-Yang:

Thanks for the KIP. I think it would be useful to clarify that this is a
common scenario rather than an edge case, which further demonstrates the
need for this optimization. For example:
A partition expansion is a common operational task in Kafka: To balance
resource utilization and cost, topics are typically created with a moderate
default partition count. However, as traffic grows over time, it is often
necessary to increase the number of partitions to accommodate the higher
workload.

Regards
Jian

黃竣陽 <s7...@gmail.com> 于2026年5月30日周六 22:31写道:

> Hello chia,
>
> Thanks for the comments, I have updated the KIP!
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> >
> > Hi Jiunn-Yang,
> >
> > Would you mind removing the terms "hot" and "cold" when describing
> > partitions in the KIP? I understand you are using them to describe the
> > "freshness" or the users' need for the records, but applying these terms
> to
> > the partition itself feels a bit unnatural.
> >
> > After all, in this scenario, users don't really care whether a partition
> is
> > newly expanded or not. Their only expectation is that they won't silently
> > lose any live records produced to the topic during their active
> consumption.
> >
> > Best, Chia-Ping
> >
> >
> >
> > 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> >
> >> Hello Jun,
> >>
> >> Thanks for the feedback, I have updated the KIP motivation section.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> >>>
> >>> Hi, Jiunn-Yang,
> >>>
> >>> Thanks for the reply. I think we need a stronger motivation for the
> KIP.
> >>>
> >>> The KIP says "The core insight is that not all partitions without a
> >>> committed offset are the same. A newly expanded partition (hot) is
> >>> fundamentally different from a partition the consumer has never seen
> >>> because it predates the group (cold)." Why is the hot partition
> >>> fundamentally different from the cold?
> >>>
> >>> The KIP says "The existing by_duration policy is also insufficient
> >> because:
> >>>
> >>>  - The calculated seek time (now() - duration) varies across nodes due
> >> to
> >>>  clock skew. To be safe, users must set an overly large duration,
> >> causing
> >>>  unnecessary reprocessing.
> >>>  - On network errors, the client recalculates the seek time on retry,
> >>>  shifting the target timestamp forward and risking data loss."
> >>>
> >>> However, both of these situations are rare. If these issues persist,
> more
> >>> severe problems likely exist elsewhere. Rare situations don't need a
> >> common
> >>> solution. If users care about those rare situations, they can implement
> >>> customized logic using
> ConsumerRebalanceListener.onPartitionsAssigned().
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>
> >>>> Hello chia,
> >>>>
> >>>> Thanks for the feedback,
> >>>>
> >>>>> If the creation time exists, the returned value should always be
> >> greater
> >>>> than or equal to zero, right?
> >>>> I have explicitly mentioned this in the KIP.
> >>>>
> >>>>>> New  Old (MetadataResponse v0–13)    positive        any     field
> >>>> absent    UnsupportedVersionException
> >>>>
> >>>> The earliest point at which we can detect the version mismatch is
> during
> >>>> the
> >>>> first metadata fetch after assignment, which occurs inside poll().
> >>>> Therefore, the
> >>>> user would encounter an UnsupportedVersionException from poll(). I’ll
> >>>> clarify this in the KIP.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> >>>>>
> >>>>> hi Jiunn
> >>>>>
> >>>>>> PartitionAgeMs (int64, default -1): The age of this partition in
> >>>> milliseconds, computed server-side by the broker as
> broker_current_time
> >> -
> >>>> partition_creation_time. Returns -1 if the broker does not support
> this
> >>>> feature or the partition creation time is unknown.
> >>>>>
> >>>>> If the creation time exists, the returned value should always be
> >> greater
> >>>> than or equal to zero, right?
> >>>>>
> >>>>>> New  Old (MetadataResponse v0–13)    positive        any     field
> >>>> absent    UnsupportedVersionException
> >>>>>
> >>>>> Will user encounter UnsupportedVersionException when calling
> `poll()`?
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>>
> >>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>>>> Hello Jun, chia,
> >>>>>>
> >>>>>> I've updated KIP-1327 with a design change based on the discussion
> >>>>>> feedback.
> >>>>>>
> >>>>>> The updated design decouples the new-partition reset behavior from
> >>>>>> the base auto.offset.reset policy:
> >>>>>>
> >>>>>> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset
> >>>> values
> >>>>>> (latest, earliest, by_duration, none).
> >>>>>> - For new ("hot") partitions, the consumer resets to
> >>>> auto.offset.reset.new.partitions
> >>>>>> config setting
> >>>>>> - For existing ("cold") partitions, the base auto.offset.reset
> policy
> >>>> continues
> >>>>>> to apply unchanged.
> >>>>>> - The new-partition reset behavior is represented by a separate
> >>>> internal config
> >>>>>> (auto.offset.reset.new.partitions, currently fixed to earliest).
> This
> >>>> decoupled design makes
> >>>>>> it straightforward to promote the behavior to a public user-facing
> >>>> configuration in a future KIP.
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>>
> >>>>>>
> >>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
> >>>>>>>
> >>>>>>> hi Jun
> >>>>>>>
> >>>>>>> I see what you mean now. The proposal from me is listed below:
> >>>>>>>
> >>>>>>> 1) Add auto.offset.reset.new.partitions with a default value of
> >>>> earliest. It fixes the data loss from both by_duration and latest, and
> >> it
> >>>> does not change the logic of auto.offset.reset=earliest.
> >>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
> >>>> configuration. auto.offset.reset.new.partitions=earliest already
> >>>> addresses the issue, and we can discuss the use cases of other values
> >> in a
> >>>> separate KIP.
> >>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
> >>>> auto.offset.reset.latest.max.age.ms, will be applied to all for
> >>>> consistency.
> >>>>>>>
> >>>>>>> WDYT?
> >>>>>>>
> >>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>>>> Hi, Chia-Ping,
> >>>>>>>>
> >>>>>>>> Thanks for the reply.
> >>>>>>>>
> >>>>>>>> 1. In the motivation section, the KIP says "When a Kafka topic is
> >>>> expanded
> >>>>>>>> with new partitions, consumers using the latest auto offset reset
> >>>> policy
> >>>>>>>> will silently miss all records produced to those partitions before
> >> the
> >>>>>>>> consumer discovers them.". If a user sets
> >>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss issue
> could
> >>>> also
> >>>>>>>> happen, right?
> >>>>>>>>
> >>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will take the
> >> same
> >>>>>>>> values as auto.offset.reset. So a user could set it by_duration if
> >>>> needed.
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <
> chia7712@apache.org
> >>>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> hi Jun
> >>>>>>>>>
> >>>>>>>>> Thanks for the feedback. I might be missing something important
> >> from
> >>>> your
> >>>>>>>>> suggestion, so please bear with me as I try to clarify with a few
> >>>> questions:
> >>>>>>>>>
> >>>>>>>>> 1. Is there a strong use case for extending this logic to other
> >> reset
> >>>>>>>>> policies? Unlike latest, policies like earliest or by_duration
> >> don't
> >>>> seem
> >>>>>>>>> to suffer from the same silent data loss issue when a partition
> is
> >>>> expanded.
> >>>>>>>>>
> >>>>>>>>> 2. What values would we expect users to configure for
> >>>>>>>>> auto.offset.reset.new.partitions? If they set it to earliest or
> >>>> latest,
> >>>>>>>>> we might run into the exact same edge cases. For example, if a
> >>>> consumer is
> >>>>>>>>> offline for a while and a new partition is created during that
> >>>> downtime,
> >>>>>>>>> the user might actually want to skip to latest when resuming,
> >> rather
> >>>> than
> >>>>>>>>> reading from earliest just because the partition is technically
> >>>> "new" to
> >>>>>>>>> the group.
> >>>>>>>>>
> >>>>>>>>> This is exactly why we opted for introducing a max.age threshold.
> >> It
> >>>> gives
> >>>>>>>>> users a time-bound way to define what is genuinely "hot/new" and
> >>>> what is
> >>>>>>>>> just an old partition they haven't seen yet.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Chia-Ping
> >>>>>>>>>
> >>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the KIP.
> >>>>>>>>>>
> >>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only
> >>>> applies when
> >>>>>>>>>> auto.offset.reset is latest. However, it seems that the
> motivation
> >>>>>>>>> equally
> >>>>>>>>>> applies when auto.offset.reset is set to other values like
> >>>> by_duration.
> >>>>>>>>> The
> >>>>>>>>>> intention is that we want to have a separate way to control
> newly
> >>>> created
> >>>>>>>>>> partitions vs existing partitions when the group starts. Have we
> >>>>>>>>> considered
> >>>>>>>>>> adding a new config like auto.offset.reset.new.partitions? If
> >> this
> >>>> new
> >>>>>>>>>> config is not set, the offset reset policy defaults to the
> policy
> >>>> used
> >>>>>>>>> for
> >>>>>>>>>> existing partitions. The user could set it explicitly to
> customize
> >>>> the
> >>>>>>>>>> behavior for new partitions.
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi all,
> >>>>>>>>>>>
> >>>>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>>>
> >>>>>>>>>>> Best Regards,
> >>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>
> >>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>
> >>>>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>>>
> >>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata
> >>>> struct
> >>>>>>>>>>> gains a new
> >>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed server-side
> >> by
> >>>> the
> >>>>>>>>>>> broker as
> >>>>>>>>>>>> broker_current_time - partition_creation_time.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Also add the consumer heartbeat flow. when MembershipManager
> >>>> detects
> >>>>>>>>> a
> >>>>>>>>>>> newly assigned
> >>>>>>>>>>>> partition, it explicitly invalidates the metadata for the
> >> affected
> >>>>>>>>> topic
> >>>>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>>>> before making the offset reset decision, even if the topic ID
> is
> >>>>>>>>> already
> >>>>>>>>>>> in the cache.
> >>>>>>>>>>>>
> >>>>>>>>>>>> MB0:
> >>>>>>>>>>>>
> >>>>>>>>>>>> The consumer learns the broker's maximum supported
> >>>> MetadataResponse
> >>>>>>>>>>> version via the
> >>>>>>>>>>>> ApiVersions negotiation at connection time. If the negotiated
> >>>>>>>>> version is
> >>>>>>>>>>> unsupported, the consumer
> >>>>>>>>>>>> knows the broker does not support PartitionAgeMs at all and
> can
> >>>>>>>>> throw an
> >>>>>>>>>>> UnsupportedVersionException
> >>>>>>>>>>>> immediately, rather than silently falling back to latest and
> >>>> risking
> >>>>>>>>>>> data loss without any operator-visible signal.
> >>>>>>>>>>>>
> >>>>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hi David
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I agree with the direction of moving the 'age' resolution
> from
> >>>> the
> >>>>>>>>>>> Heartbeat API to the Metadata API to keep the control plane
> >> clean.
> >>>> The
> >>>>>>>>> main
> >>>>>>>>>>> trade-off, as we noted before, is introducing inter-broker
> clock
> >>>> skew.
> >>>>>>>>> The
> >>>>>>>>>>> Group Coordinator approach provided a single source of truth
> for
> >>>> time.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> However, realistically, this time skew should be negligible.
> >>>> Given
> >>>>>>>>> that
> >>>>>>>>>>> the max.age threshold will likely be configured in minutes or
> >>>> hours, a
> >>>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't impact
> >> the
> >>>>>>>>>>> fallback decision.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日
> >> 下午3:29
> >>>> 寫道:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sorry, I haven't really followed the previous conversation
> >> but I
> >>>>>>>>> took a
> >>>>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the KIP
> >>>> states
> >>>>>>>>> that
> >>>>>>>>>>>>>> partition ages are returned only when this boolean is set.
> >>>>>>>>> Implicitly,
> >>>>>>>>>>> this
> >>>>>>>>>>>>>> means that when the consumer receives a new partition, it
> will
> >>>>>>>>> issue a
> >>>>>>>>>>> new
> >>>>>>>>>>>>>> HB request with the boolean set to receive the ages. Is my
> >>>>>>>>>>> understanding
> >>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also explain
> >>>> how it
> >>>>>>>>>>> fits
> >>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets,
> >> etc.).
> >>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this
> >> given
> >>>>>>>>> that
> >>>>>>>>>>> a new
> >>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could simply
> >>>> include
> >>>>>>>>> the
> >>>>>>>>>>>>>> metadata. Generally, we should be rather cautious about not
> >>>>>>>>> overloading
> >>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The
> >> API
> >>>> is
> >>>>>>>>> a
> >>>>>>>>>>>>>> control plane API for assigning or revoking partitions. The
> >> fact
> >>>>>>>>> that
> >>>>>>>>>>> we
> >>>>>>>>>>>>>> don't want to add it to the corresponding Streams API also
> >>>> suggests
> >>>>>>>>>>>>>> something is not quite right. What would we do if we want to
> >>>>>>>>> support
> >>>>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>> David
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev
> <
> >>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thank you for this great kip. Good to know about the gap.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges
> >>>> field.
> >>>>>>>>> Can a
> >>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> >>>>>>>>> TopicPartitions)
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended value or
> a
> >>>>>>>>> ConfigDef
> >>>>>>>>>>>>>>> validator? Probably it should based on the
> >> metadata.max.age.ms
> >>>> ?
> >>>>>>>>>>> Sizing
> >>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams,
> >> would
> >>>> it
> >>>>>>>>> be
> >>>>>>>>>>> better
> >>>>>>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to
> >> the
> >>>>>>>>>>>>>>> StreamsConfig block list
> >>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>>>> for a
> >>>>>>>>>>>>>>> clear warning, incase users configure it? This is the most
> >>>>>>>>> familiar
> >>>>>>>>>>>>>>> consumer config and users might easily mistakenly configure
> >>>> it. Or
> >>>>>>>>>>> may be
> >>>>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to
> >>>> earliest"
> >>>>>>>>>>> reads as
> >>>>>>>>>>>>>>> if the config were being changed per-partition which isn't
> >>>>>>>>> supported.
> >>>>>>>>>>> May
> >>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves the
> >> initial
> >>>>>>>>>>> position to
> >>>>>>>>>>>>>>> start offset for that partition" as if earliest was applied
> >> to
> >>>>>>>>> that
> >>>>>>>>>>>>>>> partition only and auto.offset.reset config is unchanged.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03
> >> 寫道:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time via
> >> the
> >>>>>>>>> Admin
> >>>>>>>>>>> API?
> >>>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose and
> >>>>>>>>> troubleshoot
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent
> Hot
> >>>> Data
> >>>>>>>>>>> Loss
> >>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>>>> a
> >>>>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot)
> >>>> partitions
> >>>>>>>>>>> from
> >>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>>>> younger than the configured threshold automatically fall
> >>>> back
> >>>>>>>>> to
> >>>>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>>>> during topic expansion without forcing a full historical
> >>>>>>>>> reprocess.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello chia, 

Thanks for the comments, I have updated the KIP!

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@gmail.com> 於 2026年5月30日 晚上8:29 寫道:
> 
> Hi Jiunn-Yang,
> 
> Would you mind removing the terms "hot" and "cold" when describing
> partitions in the KIP? I understand you are using them to describe the
> "freshness" or the users' need for the records, but applying these terms to
> the partition itself feels a bit unnatural.
> 
> After all, in this scenario, users don't really care whether a partition is
> newly expanded or not. Their only expectation is that they won't silently
> lose any live records produced to the topic during their active consumption.
> 
> Best, Chia-Ping
> 
> 
> 
> 黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:
> 
>> Hello Jun,
>> 
>> Thanks for the feedback, I have updated the KIP motivation section.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
>>> 
>>> Hi, Jiunn-Yang,
>>> 
>>> Thanks for the reply. I think we need a stronger motivation for the KIP.
>>> 
>>> The KIP says "The core insight is that not all partitions without a
>>> committed offset are the same. A newly expanded partition (hot) is
>>> fundamentally different from a partition the consumer has never seen
>>> because it predates the group (cold)." Why is the hot partition
>>> fundamentally different from the cold?
>>> 
>>> The KIP says "The existing by_duration policy is also insufficient
>> because:
>>> 
>>>  - The calculated seek time (now() - duration) varies across nodes due
>> to
>>>  clock skew. To be safe, users must set an overly large duration,
>> causing
>>>  unnecessary reprocessing.
>>>  - On network errors, the client recalculates the seek time on retry,
>>>  shifting the target timestamp forward and risking data loss."
>>> 
>>> However, both of these situations are rare. If these issues persist, more
>>> severe problems likely exist elsewhere. Rare situations don't need a
>> common
>>> solution. If users care about those rare situations, they can implement
>>> customized logic using ConsumerRebalanceListener.onPartitionsAssigned().
>>> 
>>> Jun
>>> 
>>> 
>>> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
>>> 
>>>> Hello chia,
>>>> 
>>>> Thanks for the feedback,
>>>> 
>>>>> If the creation time exists, the returned value should always be
>> greater
>>>> than or equal to zero, right?
>>>> I have explicitly mentioned this in the KIP.
>>>> 
>>>>>> New  Old (MetadataResponse v0–13)    positive        any     field
>>>> absent    UnsupportedVersionException
>>>> 
>>>> The earliest point at which we can detect the version mismatch is during
>>>> the
>>>> first metadata fetch after assignment, which occurs inside poll().
>>>> Therefore, the
>>>> user would encounter an UnsupportedVersionException from poll(). I’ll
>>>> clarify this in the KIP.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
>>>>> 
>>>>> hi Jiunn
>>>>> 
>>>>>> PartitionAgeMs (int64, default -1): The age of this partition in
>>>> milliseconds, computed server-side by the broker as broker_current_time
>> -
>>>> partition_creation_time. Returns -1 if the broker does not support this
>>>> feature or the partition creation time is unknown.
>>>>> 
>>>>> If the creation time exists, the returned value should always be
>> greater
>>>> than or equal to zero, right?
>>>>> 
>>>>>> New  Old (MetadataResponse v0–13)    positive        any     field
>>>> absent    UnsupportedVersionException
>>>>> 
>>>>> Will user encounter UnsupportedVersionException when calling `poll()`?
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> 
>>>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>>>> Hello Jun, chia,
>>>>>> 
>>>>>> I've updated KIP-1327 with a design change based on the discussion
>>>>>> feedback.
>>>>>> 
>>>>>> The updated design decouples the new-partition reset behavior from
>>>>>> the base auto.offset.reset policy:
>>>>>> 
>>>>>> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset
>>>> values
>>>>>> (latest, earliest, by_duration, none).
>>>>>> - For new ("hot") partitions, the consumer resets to
>>>> auto.offset.reset.new.partitions
>>>>>> config setting
>>>>>> - For existing ("cold") partitions, the base auto.offset.reset policy
>>>> continues
>>>>>> to apply unchanged.
>>>>>> - The new-partition reset behavior is represented by a separate
>>>> internal config
>>>>>> (auto.offset.reset.new.partitions, currently fixed to earliest). This
>>>> decoupled design makes
>>>>>> it straightforward to promote the behavior to a public user-facing
>>>> configuration in a future KIP.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
>>>>>>> 
>>>>>>> hi Jun
>>>>>>> 
>>>>>>> I see what you mean now. The proposal from me is listed below:
>>>>>>> 
>>>>>>> 1) Add auto.offset.reset.new.partitions with a default value of
>>>> earliest. It fixes the data loss from both by_duration and latest, and
>> it
>>>> does not change the logic of auto.offset.reset=earliest.
>>>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
>>>> configuration. auto.offset.reset.new.partitions=earliest already
>>>> addresses the issue, and we can discuss the use cases of other values
>> in a
>>>> separate KIP.
>>>>>>> 3) Both configs, auto.offset.reset.new.partitions and
>>>> auto.offset.reset.latest.max.age.ms, will be applied to all for
>>>> consistency.
>>>>>>> 
>>>>>>> WDYT?
>>>>>>> 
>>>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>>>> Hi, Chia-Ping,
>>>>>>>> 
>>>>>>>> Thanks for the reply.
>>>>>>>> 
>>>>>>>> 1. In the motivation section, the KIP says "When a Kafka topic is
>>>> expanded
>>>>>>>> with new partitions, consumers using the latest auto offset reset
>>>> policy
>>>>>>>> will silently miss all records produced to those partitions before
>> the
>>>>>>>> consumer discovers them.". If a user sets
>>>>>>>> auto.offset.reset=by_duration=1sec, the same record loss issue could
>>>> also
>>>>>>>> happen, right?
>>>>>>>> 
>>>>>>>> 2. I was thinking auto.offset.reset.new.partitions will take the
>> same
>>>>>>>> values as auto.offset.reset. So a user could set it by_duration if
>>>> needed.
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <chia7712@apache.org
>>> 
>>>> wrote:
>>>>>>>> 
>>>>>>>>> hi Jun
>>>>>>>>> 
>>>>>>>>> Thanks for the feedback. I might be missing something important
>> from
>>>> your
>>>>>>>>> suggestion, so please bear with me as I try to clarify with a few
>>>> questions:
>>>>>>>>> 
>>>>>>>>> 1. Is there a strong use case for extending this logic to other
>> reset
>>>>>>>>> policies? Unlike latest, policies like earliest or by_duration
>> don't
>>>> seem
>>>>>>>>> to suffer from the same silent data loss issue when a partition is
>>>> expanded.
>>>>>>>>> 
>>>>>>>>> 2. What values would we expect users to configure for
>>>>>>>>> auto.offset.reset.new.partitions? If they set it to earliest or
>>>> latest,
>>>>>>>>> we might run into the exact same edge cases. For example, if a
>>>> consumer is
>>>>>>>>> offline for a while and a new partition is created during that
>>>> downtime,
>>>>>>>>> the user might actually want to skip to latest when resuming,
>> rather
>>>> than
>>>>>>>>> reading from earliest just because the partition is technically
>>>> "new" to
>>>>>>>>> the group.
>>>>>>>>> 
>>>>>>>>> This is exactly why we opted for introducing a max.age threshold.
>> It
>>>> gives
>>>>>>>>> users a time-bound way to define what is genuinely "hot/new" and
>>>> what is
>>>>>>>>> just an old partition they haven't seen yet.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the KIP.
>>>>>>>>>> 
>>>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only
>>>> applies when
>>>>>>>>>> auto.offset.reset is latest. However, it seems that the motivation
>>>>>>>>> equally
>>>>>>>>>> applies when auto.offset.reset is set to other values like
>>>> by_duration.
>>>>>>>>> The
>>>>>>>>>> intention is that we want to have a separate way to control newly
>>>> created
>>>>>>>>>> partitions vs existing partitions when the group starts. Have we
>>>>>>>>> considered
>>>>>>>>>> adding a new config like auto.offset.reset.new.partitions? If
>> this
>>>> new
>>>>>>>>>> config is not set, the offset reset policy defaults to the policy
>>>> used
>>>>>>>>> for
>>>>>>>>>> existing partitions. The user could set it explicitly to customize
>>>> the
>>>>>>>>>> behavior for new partitions.
>>>>>>>>>> 
>>>>>>>>>> Jun
>>>>>>>>>> 
>>>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi all,
>>>>>>>>>>> 
>>>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>>>> 
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hello all,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>> 
>>>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>>>> 
>>>>>>>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata
>>>> struct
>>>>>>>>>>> gains a new
>>>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed server-side
>> by
>>>> the
>>>>>>>>>>> broker as
>>>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>>>> 
>>>>>>>>>>>> Also add the consumer heartbeat flow. when MembershipManager
>>>> detects
>>>>>>>>> a
>>>>>>>>>>> newly assigned
>>>>>>>>>>>> partition, it explicitly invalidates the metadata for the
>> affected
>>>>>>>>> topic
>>>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>>>> before making the offset reset decision, even if the topic ID is
>>>>>>>>> already
>>>>>>>>>>> in the cache.
>>>>>>>>>>>> 
>>>>>>>>>>>> MB0:
>>>>>>>>>>>> 
>>>>>>>>>>>> The consumer learns the broker's maximum supported
>>>> MetadataResponse
>>>>>>>>>>> version via the
>>>>>>>>>>>> ApiVersions negotiation at connection time. If the negotiated
>>>>>>>>> version is
>>>>>>>>>>> unsupported, the consumer
>>>>>>>>>>>> knows the broker does not support PartitionAgeMs at all and can
>>>>>>>>> throw an
>>>>>>>>>>> UnsupportedVersionException
>>>>>>>>>>>> immediately, rather than silently falling back to latest and
>>>> risking
>>>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>>>> 
>>>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>>>> 
>>>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> hi David
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I agree with the direction of moving the 'age' resolution from
>>>> the
>>>>>>>>>>> Heartbeat API to the Metadata API to keep the control plane
>> clean.
>>>> The
>>>>>>>>> main
>>>>>>>>>>> trade-off, as we noted before, is introducing inter-broker clock
>>>> skew.
>>>>>>>>> The
>>>>>>>>>>> Group Coordinator approach provided a single source of truth for
>>>> time.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> However, realistically, this time skew should be negligible.
>>>> Given
>>>>>>>>> that
>>>>>>>>>>> the max.age threshold will likely be configured in minutes or
>>>> hours, a
>>>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't impact
>> the
>>>>>>>>>>> fallback decision.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日
>> 下午3:29
>>>> 寫道:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Sorry, I haven't really followed the previous conversation
>> but I
>>>>>>>>> took a
>>>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the KIP
>>>> states
>>>>>>>>> that
>>>>>>>>>>>>>> partition ages are returned only when this boolean is set.
>>>>>>>>> Implicitly,
>>>>>>>>>>> this
>>>>>>>>>>>>>> means that when the consumer receives a new partition, it will
>>>>>>>>> issue a
>>>>>>>>>>> new
>>>>>>>>>>>>>> HB request with the boolean set to receive the ages. Is my
>>>>>>>>>>> understanding
>>>>>>>>>>>>>> correct? We should perhaps clarify the flow and also explain
>>>> how it
>>>>>>>>>>> fits
>>>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets,
>> etc.).
>>>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this
>> given
>>>>>>>>> that
>>>>>>>>>>> a new
>>>>>>>>>>>>>> round trip is done anyway. Alternatively, it could simply
>>>> include
>>>>>>>>> the
>>>>>>>>>>>>>> metadata. Generally, we should be rather cautious about not
>>>>>>>>> overloading
>>>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The
>> API
>>>> is
>>>>>>>>> a
>>>>>>>>>>>>>> control plane API for assigning or revoking partitions. The
>> fact
>>>>>>>>> that
>>>>>>>>>>> we
>>>>>>>>>>>>>> don't want to add it to the corresponding Streams API also
>>>> suggests
>>>>>>>>>>>>>> something is not quite right. What would we do if we want to
>>>>>>>>> support
>>>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>> David
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
>>>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thank you for this great kip. Good to know about the gap.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges
>>>> field.
>>>>>>>>> Can a
>>>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
>>>>>>>>> TopicPartitions)
>>>>>>>>>>> be
>>>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended value or a
>>>>>>>>> ConfigDef
>>>>>>>>>>>>>>> validator? Probably it should based on the
>> metadata.max.age.ms
>>>> ?
>>>>>>>>>>> Sizing
>>>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams,
>> would
>>>> it
>>>>>>>>> be
>>>>>>>>>>> better
>>>>>>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to
>> the
>>>>>>>>>>>>>>> StreamsConfig block list
>>>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>>>> for a
>>>>>>>>>>>>>>> clear warning, incase users configure it? This is the most
>>>>>>>>> familiar
>>>>>>>>>>>>>>> consumer config and users might easily mistakenly configure
>>>> it. Or
>>>>>>>>>>> may be
>>>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to
>>>> earliest"
>>>>>>>>>>> reads as
>>>>>>>>>>>>>>> if the config were being changed per-partition which isn't
>>>>>>>>> supported.
>>>>>>>>>>> May
>>>>>>>>>>>>>>> be rephrasing to something like "consumer resolves the
>> initial
>>>>>>>>>>> position to
>>>>>>>>>>>>>>> start offset for that partition" as if earliest was applied
>> to
>>>>>>>>> that
>>>>>>>>>>>>>>> partition only and auto.offset.reset config is unchanged.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Murali
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com>
>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03
>> 寫道:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time via
>> the
>>>>>>>>> Admin
>>>>>>>>>>> API?
>>>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose and
>>>>>>>>> troubleshoot
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot
>>>> Data
>>>>>>>>>>> Loss
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>> 
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>>>> a
>>>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot)
>>>> partitions
>>>>>>>>>>> from
>>>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>>>> younger than the configured threshold automatically fall
>>>> back
>>>>>>>>> to
>>>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>>>> during topic expansion without forcing a full historical
>>>>>>>>> reprocess.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@gmail.com>.
Hi Jiunn-Yang,

Would you mind removing the terms "hot" and "cold" when describing
partitions in the KIP? I understand you are using them to describe the
"freshness" or the users' need for the records, but applying these terms to
the partition itself feels a bit unnatural.

After all, in this scenario, users don't really care whether a partition is
newly expanded or not. Their only expectation is that they won't silently
lose any live records produced to the topic during their active consumption.

Best, Chia-Ping



黃竣陽 <s7...@gmail.com> 於 2026年5月30日週六 下午12:30寫道:

> Hello Jun,
>
> Thanks for the feedback, I have updated the KIP motivation section.
>
> Best Regards,
> Jiunn-Yang
>
> > Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> >
> > Hi, Jiunn-Yang,
> >
> > Thanks for the reply. I think we need a stronger motivation for the KIP.
> >
> > The KIP says "The core insight is that not all partitions without a
> > committed offset are the same. A newly expanded partition (hot) is
> > fundamentally different from a partition the consumer has never seen
> > because it predates the group (cold)." Why is the hot partition
> > fundamentally different from the cold?
> >
> > The KIP says "The existing by_duration policy is also insufficient
> because:
> >
> >   - The calculated seek time (now() - duration) varies across nodes due
> to
> >   clock skew. To be safe, users must set an overly large duration,
> causing
> >   unnecessary reprocessing.
> >   - On network errors, the client recalculates the seek time on retry,
> >   shifting the target timestamp forward and risking data loss."
> >
> > However, both of these situations are rare. If these issues persist, more
> > severe problems likely exist elsewhere. Rare situations don't need a
> common
> > solution. If users care about those rare situations, they can implement
> > customized logic using ConsumerRebalanceListener.onPartitionsAssigned().
> >
> > Jun
> >
> >
> > On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> >> Hello chia,
> >>
> >> Thanks for the feedback,
> >>
> >>> If the creation time exists, the returned value should always be
> greater
> >> than or equal to zero, right?
> >> I have explicitly mentioned this in the KIP.
> >>
> >>>> New  Old (MetadataResponse v0–13)    positive        any     field
> >> absent    UnsupportedVersionException
> >>
> >> The earliest point at which we can detect the version mismatch is during
> >> the
> >> first metadata fetch after assignment, which occurs inside poll().
> >> Therefore, the
> >> user would encounter an UnsupportedVersionException from poll(). I’ll
> >> clarify this in the KIP.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> >>>
> >>> hi Jiunn
> >>>
> >>>> PartitionAgeMs (int64, default -1): The age of this partition in
> >> milliseconds, computed server-side by the broker as broker_current_time
> -
> >> partition_creation_time. Returns -1 if the broker does not support this
> >> feature or the partition creation time is unknown.
> >>>
> >>> If the creation time exists, the returned value should always be
> greater
> >> than or equal to zero, right?
> >>>
> >>>> New  Old (MetadataResponse v0–13)    positive        any     field
> >> absent    UnsupportedVersionException
> >>>
> >>> Will user encounter UnsupportedVersionException when calling `poll()`?
> >>>
> >>> Best,
> >>> Chia-Ping
> >>>
> >>>
> >>> On 2026/05/16 04:30:49 黃竣陽 wrote:
> >>>> Hello Jun, chia,
> >>>>
> >>>> I've updated KIP-1327 with a design change based on the discussion
> >>>> feedback.
> >>>>
> >>>> The updated design decouples the new-partition reset behavior from
> >>>> the base auto.offset.reset policy:
> >>>>
> >>>> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset
> >> values
> >>>> (latest, earliest, by_duration, none).
> >>>> - For new ("hot") partitions, the consumer resets to
> >> auto.offset.reset.new.partitions
> >>>> config setting
> >>>> - For existing ("cold") partitions, the base auto.offset.reset policy
> >> continues
> >>>> to apply unchanged.
> >>>> - The new-partition reset behavior is represented by a separate
> >> internal config
> >>>> (auto.offset.reset.new.partitions, currently fixed to earliest). This
> >> decoupled design makes
> >>>> it straightforward to promote the behavior to a public user-facing
> >> configuration in a future KIP.
> >>>>
> >>>> Best Regards,
> >>>> Jiunn-Yang
> >>>>
> >>>>
> >>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
> >>>>>
> >>>>> hi Jun
> >>>>>
> >>>>> I see what you mean now. The proposal from me is listed below:
> >>>>>
> >>>>> 1) Add auto.offset.reset.new.partitions with a default value of
> >> earliest. It fixes the data loss from both by_duration and latest, and
> it
> >> does not change the logic of auto.offset.reset=earliest.
> >>>>> 2) Mark auto.offset.reset.new.partitions as an internal
> >> configuration. auto.offset.reset.new.partitions=earliest already
> >> addresses the issue, and we can discuss the use cases of other values
> in a
> >> separate KIP.
> >>>>> 3) Both configs, auto.offset.reset.new.partitions and
> >> auto.offset.reset.latest.max.age.ms, will be applied to all for
> >> consistency.
> >>>>>
> >>>>> WDYT?
> >>>>>
> >>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>>>> Hi, Chia-Ping,
> >>>>>>
> >>>>>> Thanks for the reply.
> >>>>>>
> >>>>>> 1. In the motivation section, the KIP says "When a Kafka topic is
> >> expanded
> >>>>>> with new partitions, consumers using the latest auto offset reset
> >> policy
> >>>>>> will silently miss all records produced to those partitions before
> the
> >>>>>> consumer discovers them.". If a user sets
> >>>>>> auto.offset.reset=by_duration=1sec, the same record loss issue could
> >> also
> >>>>>> happen, right?
> >>>>>>
> >>>>>> 2. I was thinking auto.offset.reset.new.partitions will take the
> same
> >>>>>> values as auto.offset.reset. So a user could set it by_duration if
> >> needed.
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <chia7712@apache.org
> >
> >> wrote:
> >>>>>>
> >>>>>>> hi Jun
> >>>>>>>
> >>>>>>> Thanks for the feedback. I might be missing something important
> from
> >> your
> >>>>>>> suggestion, so please bear with me as I try to clarify with a few
> >> questions:
> >>>>>>>
> >>>>>>> 1. Is there a strong use case for extending this logic to other
> reset
> >>>>>>> policies? Unlike latest, policies like earliest or by_duration
> don't
> >> seem
> >>>>>>> to suffer from the same silent data loss issue when a partition is
> >> expanded.
> >>>>>>>
> >>>>>>> 2. What values would we expect users to configure for
> >>>>>>> auto.offset.reset.new.partitions? If they set it to earliest or
> >> latest,
> >>>>>>> we might run into the exact same edge cases. For example, if a
> >> consumer is
> >>>>>>> offline for a while and a new partition is created during that
> >> downtime,
> >>>>>>> the user might actually want to skip to latest when resuming,
> rather
> >> than
> >>>>>>> reading from earliest just because the partition is technically
> >> "new" to
> >>>>>>> the group.
> >>>>>>>
> >>>>>>> This is exactly why we opted for introducing a max.age threshold.
> It
> >> gives
> >>>>>>> users a time-bound way to define what is genuinely "hot/new" and
> >> what is
> >>>>>>> just an old partition they haven't seen yet.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Chia-Ping
> >>>>>>>
> >>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>>>> Hi, Jiunn-Yang,
> >>>>>>>>
> >>>>>>>> Thanks for the KIP.
> >>>>>>>>
> >>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only
> >> applies when
> >>>>>>>> auto.offset.reset is latest. However, it seems that the motivation
> >>>>>>> equally
> >>>>>>>> applies when auto.offset.reset is set to other values like
> >> by_duration.
> >>>>>>> The
> >>>>>>>> intention is that we want to have a separate way to control newly
> >> created
> >>>>>>>> partitions vs existing partitions when the group starts. Have we
> >>>>>>> considered
> >>>>>>>> adding a new config like auto.offset.reset.new.partitions? If
> this
> >> new
> >>>>>>>> config is not set, the offset reset policy defaults to the policy
> >> used
> >>>>>>> for
> >>>>>>>> existing partitions. The user could set it explicitly to customize
> >> the
> >>>>>>>> behavior for new partitions.
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi all,
> >>>>>>>>>
> >>>>>>>>> I’d like to manually bump this thread.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Jiunn-Yang
> >>>>>>>>>
> >>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>>>>>>
> >>>>>>>>>> Hello all,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>
> >>>>>>>>>> DJ01/DJ02:
> >>>>>>>>>>
> >>>>>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata
> >> struct
> >>>>>>>>> gains a new
> >>>>>>>>>> field PartitionAgeMs (int64, default -1), computed server-side
> by
> >> the
> >>>>>>>>> broker as
> >>>>>>>>>> broker_current_time - partition_creation_time.
> >>>>>>>>>>
> >>>>>>>>>> Also add the consumer heartbeat flow. when MembershipManager
> >> detects
> >>>>>>> a
> >>>>>>>>> newly assigned
> >>>>>>>>>> partition, it explicitly invalidates the metadata for the
> affected
> >>>>>>> topic
> >>>>>>>>> and forces a fresh MetadataRequest
> >>>>>>>>>> before making the offset reset decision, even if the topic ID is
> >>>>>>> already
> >>>>>>>>> in the cache.
> >>>>>>>>>>
> >>>>>>>>>> MB0:
> >>>>>>>>>>
> >>>>>>>>>> The consumer learns the broker's maximum supported
> >> MetadataResponse
> >>>>>>>>> version via the
> >>>>>>>>>> ApiVersions negotiation at connection time. If the negotiated
> >>>>>>> version is
> >>>>>>>>> unsupported, the consumer
> >>>>>>>>>> knows the broker does not support PartitionAgeMs at all and can
> >>>>>>> throw an
> >>>>>>>>> UnsupportedVersionException
> >>>>>>>>>> immediately, rather than silently falling back to latest and
> >> risking
> >>>>>>>>> data loss without any operator-visible signal.
> >>>>>>>>>>
> >>>>>>>>>> MB1/MB2/MB3:
> >>>>>>>>>>
> >>>>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>
> >>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> >>>>>>>>>>>
> >>>>>>>>>>> hi David
> >>>>>>>>>>>
> >>>>>>>>>>> I agree with the direction of moving the 'age' resolution from
> >> the
> >>>>>>>>> Heartbeat API to the Metadata API to keep the control plane
> clean.
> >> The
> >>>>>>> main
> >>>>>>>>> trade-off, as we noted before, is introducing inter-broker clock
> >> skew.
> >>>>>>> The
> >>>>>>>>> Group Coordinator approach provided a single source of truth for
> >> time.
> >>>>>>>>>>>
> >>>>>>>>>>> However, realistically, this time skew should be negligible.
> >> Given
> >>>>>>> that
> >>>>>>>>> the max.age threshold will likely be configured in minutes or
> >> hours, a
> >>>>>>>>> typical NTP skew (in milliseconds) between brokers won't impact
> the
> >>>>>>>>> fallback decision.
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>
> >>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日
> 下午3:29
> >> 寫道:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sorry, I haven't really followed the previous conversation
> but I
> >>>>>>> took a
> >>>>>>>>>>>> quick look at this one.
> >>>>>>>>>>>>
> >>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>>>> API after reading the KIP. There is a new boolean; the KIP
> >> states
> >>>>>>> that
> >>>>>>>>>>>> partition ages are returned only when this boolean is set.
> >>>>>>> Implicitly,
> >>>>>>>>> this
> >>>>>>>>>>>> means that when the consumer receives a new partition, it will
> >>>>>>> issue a
> >>>>>>>>> new
> >>>>>>>>>>>> HB request with the boolean set to receive the ages. Is my
> >>>>>>>>> understanding
> >>>>>>>>>>>> correct? We should perhaps clarify the flow and also explain
> >> how it
> >>>>>>>>> fits
> >>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets,
> etc.).
> >>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this
> given
> >>>>>>> that
> >>>>>>>>> a new
> >>>>>>>>>>>> round trip is done anyway. Alternatively, it could simply
> >> include
> >>>>>>> the
> >>>>>>>>>>>> metadata. Generally, we should be rather cautious about not
> >>>>>>> overloading
> >>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The
> API
> >> is
> >>>>>>> a
> >>>>>>>>>>>> control plane API for assigning or revoking partitions. The
> fact
> >>>>>>> that
> >>>>>>>>> we
> >>>>>>>>>>>> don't want to add it to the corresponding Streams API also
> >> suggests
> >>>>>>>>>>>> something is not quite right. What would we do if we want to
> >>>>>>> support
> >>>>>>>>>>>> Streams in the future?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best,
> >>>>>>>>>>>> David
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
> >>>>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you for this great kip. Good to know about the gap.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges
> >> field.
> >>>>>>> Can a
> >>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> >>>>>>> TopicPartitions)
> >>>>>>>>> be
> >>>>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> mb-1 - For the new config, is there a recommended value or a
> >>>>>>> ConfigDef
> >>>>>>>>>>>>> validator? Probably it should based on the
> metadata.max.age.ms
> >> ?
> >>>>>>>>> Sizing
> >>>>>>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams,
> would
> >> it
> >>>>>>> be
> >>>>>>>>> better
> >>>>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to
> the
> >>>>>>>>>>>>> StreamsConfig block list
> >>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>>>> for a
> >>>>>>>>>>>>> clear warning, incase users configure it? This is the most
> >>>>>>> familiar
> >>>>>>>>>>>>> consumer config and users might easily mistakenly configure
> >> it. Or
> >>>>>>>>> may be
> >>>>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to
> >> earliest"
> >>>>>>>>> reads as
> >>>>>>>>>>>>> if the config were being changed per-partition which isn't
> >>>>>>> supported.
> >>>>>>>>> May
> >>>>>>>>>>>>> be rephrasing to something like "consumer resolves the
> initial
> >>>>>>>>> position to
> >>>>>>>>>>>>> start offset for that partition" as if earliest was applied
> to
> >>>>>>> that
> >>>>>>>>>>>>> partition only and auto.offset.reset config is unchanged.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Murali
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com>
> >> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03
> 寫道:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time via
> the
> >>>>>>> Admin
> >>>>>>>>> API?
> >>>>>>>>>>>>>> I assume it would be valuable for users to diagnose and
> >>>>>>> troubleshoot
> >>>>>>>>> the
> >>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot
> >> Data
> >>>>>>>>> Loss
> >>>>>>>>>>>>> on
> >>>>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> This proposal aims to introduces
> >>>>>>> auto.offset.reset.latest.max.age,
> >>>>>>>>> a
> >>>>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot)
> >> partitions
> >>>>>>>>> from
> >>>>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>>>> younger than the configured threshold automatically fall
> >> back
> >>>>>>> to
> >>>>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>>>> during topic expansion without forcing a full historical
> >>>>>>> reprocess.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello Jun,

Thanks for the feedback, I have updated the KIP motivation section.

Best Regards,
Jiunn-Yang

> Jun Rao via dev <de...@kafka.apache.org> 於 2026年5月30日 凌晨1:12 寫道:
> 
> Hi, Jiunn-Yang,
> 
> Thanks for the reply. I think we need a stronger motivation for the KIP.
> 
> The KIP says "The core insight is that not all partitions without a
> committed offset are the same. A newly expanded partition (hot) is
> fundamentally different from a partition the consumer has never seen
> because it predates the group (cold)." Why is the hot partition
> fundamentally different from the cold?
> 
> The KIP says "The existing by_duration policy is also insufficient because:
> 
>   - The calculated seek time (now() - duration) varies across nodes due to
>   clock skew. To be safe, users must set an overly large duration, causing
>   unnecessary reprocessing.
>   - On network errors, the client recalculates the seek time on retry,
>   shifting the target timestamp forward and risking data loss."
> 
> However, both of these situations are rare. If these issues persist, more
> severe problems likely exist elsewhere. Rare situations don't need a common
> solution. If users care about those rare situations, they can implement
> customized logic using ConsumerRebalanceListener.onPartitionsAssigned().
> 
> Jun
> 
> 
> On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
>> Hello chia,
>> 
>> Thanks for the feedback,
>> 
>>> If the creation time exists, the returned value should always be greater
>> than or equal to zero, right?
>> I have explicitly mentioned this in the KIP.
>> 
>>>> New  Old (MetadataResponse v0–13)    positive        any     field
>> absent    UnsupportedVersionException
>> 
>> The earliest point at which we can detect the version mismatch is during
>> the
>> first metadata fetch after assignment, which occurs inside poll().
>> Therefore, the
>> user would encounter an UnsupportedVersionException from poll(). I’ll
>> clarify this in the KIP.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
>>> 
>>> hi Jiunn
>>> 
>>>> PartitionAgeMs (int64, default -1): The age of this partition in
>> milliseconds, computed server-side by the broker as broker_current_time -
>> partition_creation_time. Returns -1 if the broker does not support this
>> feature or the partition creation time is unknown.
>>> 
>>> If the creation time exists, the returned value should always be greater
>> than or equal to zero, right?
>>> 
>>>> New  Old (MetadataResponse v0–13)    positive        any     field
>> absent    UnsupportedVersionException
>>> 
>>> Will user encounter UnsupportedVersionException when calling `poll()`?
>>> 
>>> Best,
>>> Chia-Ping
>>> 
>>> 
>>> On 2026/05/16 04:30:49 黃竣陽 wrote:
>>>> Hello Jun, chia,
>>>> 
>>>> I've updated KIP-1327 with a design change based on the discussion
>>>> feedback.
>>>> 
>>>> The updated design decouples the new-partition reset behavior from
>>>> the base auto.offset.reset policy:
>>>> 
>>>> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset
>> values
>>>> (latest, earliest, by_duration, none).
>>>> - For new ("hot") partitions, the consumer resets to
>> auto.offset.reset.new.partitions
>>>> config setting
>>>> - For existing ("cold") partitions, the base auto.offset.reset policy
>> continues
>>>> to apply unchanged.
>>>> - The new-partition reset behavior is represented by a separate
>> internal config
>>>> (auto.offset.reset.new.partitions, currently fixed to earliest). This
>> decoupled design makes
>>>> it straightforward to promote the behavior to a public user-facing
>> configuration in a future KIP.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>> 
>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
>>>>> 
>>>>> hi Jun
>>>>> 
>>>>> I see what you mean now. The proposal from me is listed below:
>>>>> 
>>>>> 1) Add auto.offset.reset.new.partitions with a default value of
>> earliest. It fixes the data loss from both by_duration and latest, and it
>> does not change the logic of auto.offset.reset=earliest.
>>>>> 2) Mark auto.offset.reset.new.partitions as an internal
>> configuration. auto.offset.reset.new.partitions=earliest already
>> addresses the issue, and we can discuss the use cases of other values in a
>> separate KIP.
>>>>> 3) Both configs, auto.offset.reset.new.partitions and
>> auto.offset.reset.latest.max.age.ms, will be applied to all for
>> consistency.
>>>>> 
>>>>> WDYT?
>>>>> 
>>>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>>>> Hi, Chia-Ping,
>>>>>> 
>>>>>> Thanks for the reply.
>>>>>> 
>>>>>> 1. In the motivation section, the KIP says "When a Kafka topic is
>> expanded
>>>>>> with new partitions, consumers using the latest auto offset reset
>> policy
>>>>>> will silently miss all records produced to those partitions before the
>>>>>> consumer discovers them.". If a user sets
>>>>>> auto.offset.reset=by_duration=1sec, the same record loss issue could
>> also
>>>>>> happen, right?
>>>>>> 
>>>>>> 2. I was thinking auto.offset.reset.new.partitions will take the same
>>>>>> values as auto.offset.reset. So a user could set it by_duration if
>> needed.
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <ch...@apache.org>
>> wrote:
>>>>>> 
>>>>>>> hi Jun
>>>>>>> 
>>>>>>> Thanks for the feedback. I might be missing something important from
>> your
>>>>>>> suggestion, so please bear with me as I try to clarify with a few
>> questions:
>>>>>>> 
>>>>>>> 1. Is there a strong use case for extending this logic to other reset
>>>>>>> policies? Unlike latest, policies like earliest or by_duration don't
>> seem
>>>>>>> to suffer from the same silent data loss issue when a partition is
>> expanded.
>>>>>>> 
>>>>>>> 2. What values would we expect users to configure for
>>>>>>> auto.offset.reset.new.partitions? If they set it to earliest or
>> latest,
>>>>>>> we might run into the exact same edge cases. For example, if a
>> consumer is
>>>>>>> offline for a while and a new partition is created during that
>> downtime,
>>>>>>> the user might actually want to skip to latest when resuming, rather
>> than
>>>>>>> reading from earliest just because the partition is technically
>> "new" to
>>>>>>> the group.
>>>>>>> 
>>>>>>> This is exactly why we opted for introducing a max.age threshold. It
>> gives
>>>>>>> users a time-bound way to define what is genuinely "hot/new" and
>> what is
>>>>>>> just an old partition they haven't seen yet.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>>>> Hi, Jiunn-Yang,
>>>>>>>> 
>>>>>>>> Thanks for the KIP.
>>>>>>>> 
>>>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only
>> applies when
>>>>>>>> auto.offset.reset is latest. However, it seems that the motivation
>>>>>>> equally
>>>>>>>> applies when auto.offset.reset is set to other values like
>> by_duration.
>>>>>>> The
>>>>>>>> intention is that we want to have a separate way to control newly
>> created
>>>>>>>> partitions vs existing partitions when the group starts. Have we
>>>>>>> considered
>>>>>>>> adding a new config like auto.offset.reset.new.partitions? If this
>> new
>>>>>>>> config is not set, the offset reset policy defaults to the policy
>> used
>>>>>>> for
>>>>>>>> existing partitions. The user could set it explicitly to customize
>> the
>>>>>>>> behavior for new partitions.
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> Hi all,
>>>>>>>>> 
>>>>>>>>> I’d like to manually bump this thread.
>>>>>>>>> 
>>>>>>>>> Best Regards,
>>>>>>>>> Jiunn-Yang
>>>>>>>>> 
>>>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>>>> 
>>>>>>>>>> Hello all,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>> 
>>>>>>>>>> DJ01/DJ02:
>>>>>>>>>> 
>>>>>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata
>> struct
>>>>>>>>> gains a new
>>>>>>>>>> field PartitionAgeMs (int64, default -1), computed server-side by
>> the
>>>>>>>>> broker as
>>>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>>>> 
>>>>>>>>>> Also add the consumer heartbeat flow. when MembershipManager
>> detects
>>>>>>> a
>>>>>>>>> newly assigned
>>>>>>>>>> partition, it explicitly invalidates the metadata for the affected
>>>>>>> topic
>>>>>>>>> and forces a fresh MetadataRequest
>>>>>>>>>> before making the offset reset decision, even if the topic ID is
>>>>>>> already
>>>>>>>>> in the cache.
>>>>>>>>>> 
>>>>>>>>>> MB0:
>>>>>>>>>> 
>>>>>>>>>> The consumer learns the broker's maximum supported
>> MetadataResponse
>>>>>>>>> version via the
>>>>>>>>>> ApiVersions negotiation at connection time. If the negotiated
>>>>>>> version is
>>>>>>>>> unsupported, the consumer
>>>>>>>>>> knows the broker does not support PartitionAgeMs at all and can
>>>>>>> throw an
>>>>>>>>> UnsupportedVersionException
>>>>>>>>>> immediately, rather than silently falling back to latest and
>> risking
>>>>>>>>> data loss without any operator-visible signal.
>>>>>>>>>> 
>>>>>>>>>> MB1/MB2/MB3:
>>>>>>>>>> 
>>>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
>>>>>>>>>>> 
>>>>>>>>>>> hi David
>>>>>>>>>>> 
>>>>>>>>>>> I agree with the direction of moving the 'age' resolution from
>> the
>>>>>>>>> Heartbeat API to the Metadata API to keep the control plane clean.
>> The
>>>>>>> main
>>>>>>>>> trade-off, as we noted before, is introducing inter-broker clock
>> skew.
>>>>>>> The
>>>>>>>>> Group Coordinator approach provided a single source of truth for
>> time.
>>>>>>>>>>> 
>>>>>>>>>>> However, realistically, this time skew should be negligible.
>> Given
>>>>>>> that
>>>>>>>>> the max.age threshold will likely be configured in minutes or
>> hours, a
>>>>>>>>> typical NTP skew (in milliseconds) between brokers won't impact the
>>>>>>>>> fallback decision.
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29
>> 寫道:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>> 
>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>> 
>>>>>>>>>>>> Sorry, I haven't really followed the previous conversation but I
>>>>>>> took a
>>>>>>>>>>>> quick look at this one.
>>>>>>>>>>>> 
>>>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>>>> API after reading the KIP. There is a new boolean; the KIP
>> states
>>>>>>> that
>>>>>>>>>>>> partition ages are returned only when this boolean is set.
>>>>>>> Implicitly,
>>>>>>>>> this
>>>>>>>>>>>> means that when the consumer receives a new partition, it will
>>>>>>> issue a
>>>>>>>>> new
>>>>>>>>>>>> HB request with the boolean set to receive the ages. Is my
>>>>>>>>> understanding
>>>>>>>>>>>> correct? We should perhaps clarify the flow and also explain
>> how it
>>>>>>>>> fits
>>>>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
>>>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this given
>>>>>>> that
>>>>>>>>> a new
>>>>>>>>>>>> round trip is done anyway. Alternatively, it could simply
>> include
>>>>>>> the
>>>>>>>>>>>> metadata. Generally, we should be rather cautious about not
>>>>>>> overloading
>>>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The API
>> is
>>>>>>> a
>>>>>>>>>>>> control plane API for assigning or revoking partitions. The fact
>>>>>>> that
>>>>>>>>> we
>>>>>>>>>>>> don't want to add it to the corresponding Streams API also
>> suggests
>>>>>>>>>>>> something is not quite right. What would we do if we want to
>>>>>>> support
>>>>>>>>>>>> Streams in the future?
>>>>>>>>>>>> 
>>>>>>>>>>>> Best,
>>>>>>>>>>>> David
>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
>>>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you for this great kip. Good to know about the gap.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges
>> field.
>>>>>>> Can a
>>>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
>>>>>>> TopicPartitions)
>>>>>>>>> be
>>>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> mb-1 - For the new config, is there a recommended value or a
>>>>>>> ConfigDef
>>>>>>>>>>>>> validator? Probably it should based on the metadata.max.age.ms
>> ?
>>>>>>>>> Sizing
>>>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams, would
>> it
>>>>>>> be
>>>>>>>>> better
>>>>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to the
>>>>>>>>>>>>> StreamsConfig block list
>>>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>>>> for a
>>>>>>>>>>>>> clear warning, incase users configure it? This is the most
>>>>>>> familiar
>>>>>>>>>>>>> consumer config and users might easily mistakenly configure
>> it. Or
>>>>>>>>> may be
>>>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to
>> earliest"
>>>>>>>>> reads as
>>>>>>>>>>>>> if the config were being changed per-partition which isn't
>>>>>>> supported.
>>>>>>>>> May
>>>>>>>>>>>>> be rephrasing to something like "consumer resolves the initial
>>>>>>>>> position to
>>>>>>>>>>>>> start offset for that partition" as if earliest was applied to
>>>>>>> that
>>>>>>>>>>>>> partition only and auto.offset.reset config is unchanged.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Murali
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com>
>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> chia_0: Should we expose the partition creation time via the
>>>>>>> Admin
>>>>>>>>> API?
>>>>>>>>>>>>>> I assume it would be valuable for users to diagnose and
>>>>>>> troubleshoot
>>>>>>>>> the
>>>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot
>> Data
>>>>>>>>> Loss
>>>>>>>>>>>>> on
>>>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>>>> <
>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> This proposal aims to introduces
>>>>>>> auto.offset.reset.latest.max.age,
>>>>>>>>> a
>>>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot)
>> partitions
>>>>>>>>> from
>>>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>>>> younger than the configured threshold automatically fall
>> back
>>>>>>> to
>>>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>>>> during topic expansion without forcing a full historical
>>>>>>> reprocess.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Jiunn-Yang,

Thanks for the reply. I think we need a stronger motivation for the KIP.

The KIP says "The core insight is that not all partitions without a
committed offset are the same. A newly expanded partition (hot) is
fundamentally different from a partition the consumer has never seen
because it predates the group (cold)." Why is the hot partition
fundamentally different from the cold?

The KIP says "The existing by_duration policy is also insufficient because:

   - The calculated seek time (now() - duration) varies across nodes due to
   clock skew. To be safe, users must set an overly large duration, causing
   unnecessary reprocessing.
   - On network errors, the client recalculates the seek time on retry,
   shifting the target timestamp forward and risking data loss."

However, both of these situations are rare. If these issues persist, more
severe problems likely exist elsewhere. Rare situations don't need a common
solution. If users care about those rare situations, they can implement
customized logic using ConsumerRebalanceListener.onPartitionsAssigned().

Jun


On Sun, May 17, 2026 at 6:50 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hello chia,
>
> Thanks for the feedback,
>
> > If the creation time exists, the returned value should always be greater
> than or equal to zero, right?
> I have explicitly mentioned this in the KIP.
>
> >> New  Old (MetadataResponse v0–13)    positive        any     field
> absent    UnsupportedVersionException
>
> The earliest point at which we can detect the version mismatch is during
> the
> first metadata fetch after assignment, which occurs inside poll().
> Therefore, the
> user would encounter an UnsupportedVersionException from poll(). I’ll
> clarify this in the KIP.
>
> Best Regards,
> Jiunn-Yang
>
> > Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> >
> > hi Jiunn
> >
> >> PartitionAgeMs (int64, default -1): The age of this partition in
> milliseconds, computed server-side by the broker as broker_current_time -
> partition_creation_time. Returns -1 if the broker does not support this
> feature or the partition creation time is unknown.
> >
> > If the creation time exists, the returned value should always be greater
> than or equal to zero, right?
> >
> >> New  Old (MetadataResponse v0–13)    positive        any     field
> absent    UnsupportedVersionException
> >
> > Will user encounter UnsupportedVersionException when calling `poll()`?
> >
> > Best,
> > Chia-Ping
> >
> >
> > On 2026/05/16 04:30:49 黃竣陽 wrote:
> >> Hello Jun, chia,
> >>
> >> I've updated KIP-1327 with a design change based on the discussion
> >> feedback.
> >>
> >> The updated design decouples the new-partition reset behavior from
> >> the base auto.offset.reset policy:
> >>
> >> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset
> values
> >> (latest, earliest, by_duration, none).
> >> - For new ("hot") partitions, the consumer resets to
> auto.offset.reset.new.partitions
> >> config setting
> >> - For existing ("cold") partitions, the base auto.offset.reset policy
> continues
> >> to apply unchanged.
> >> - The new-partition reset behavior is represented by a separate
> internal config
> >> (auto.offset.reset.new.partitions, currently fixed to earliest). This
> decoupled design makes
> >> it straightforward to promote the behavior to a public user-facing
> configuration in a future KIP.
> >>
> >> Best Regards,
> >> Jiunn-Yang
> >>
> >>
> >>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
> >>>
> >>> hi Jun
> >>>
> >>> I see what you mean now. The proposal from me is listed below:
> >>>
> >>> 1) Add auto.offset.reset.new.partitions with a default value of
> earliest. It fixes the data loss from both by_duration and latest, and it
> does not change the logic of auto.offset.reset=earliest.
> >>> 2) Mark auto.offset.reset.new.partitions as an internal
> configuration. auto.offset.reset.new.partitions=earliest already
> addresses the issue, and we can discuss the use cases of other values in a
> separate KIP.
> >>> 3) Both configs, auto.offset.reset.new.partitions and
> auto.offset.reset.latest.max.age.ms, will be applied to all for
> consistency.
> >>>
> >>> WDYT?
> >>>
> >>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >>>> Hi, Chia-Ping,
> >>>>
> >>>> Thanks for the reply.
> >>>>
> >>>> 1. In the motivation section, the KIP says "When a Kafka topic is
> expanded
> >>>> with new partitions, consumers using the latest auto offset reset
> policy
> >>>> will silently miss all records produced to those partitions before the
> >>>> consumer discovers them.". If a user sets
> >>>> auto.offset.reset=by_duration=1sec, the same record loss issue could
> also
> >>>> happen, right?
> >>>>
> >>>> 2. I was thinking auto.offset.reset.new.partitions will take the same
> >>>> values as auto.offset.reset. So a user could set it by_duration if
> needed.
> >>>>
> >>>> Jun
> >>>>
> >>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <ch...@apache.org>
> wrote:
> >>>>
> >>>>> hi Jun
> >>>>>
> >>>>> Thanks for the feedback. I might be missing something important from
> your
> >>>>> suggestion, so please bear with me as I try to clarify with a few
> questions:
> >>>>>
> >>>>> 1. Is there a strong use case for extending this logic to other reset
> >>>>> policies? Unlike latest, policies like earliest or by_duration don't
> seem
> >>>>> to suffer from the same silent data loss issue when a partition is
> expanded.
> >>>>>
> >>>>> 2. What values would we expect users to configure for
> >>>>> auto.offset.reset.new.partitions? If they set it to earliest or
> latest,
> >>>>> we might run into the exact same edge cases. For example, if a
> consumer is
> >>>>> offline for a while and a new partition is created during that
> downtime,
> >>>>> the user might actually want to skip to latest when resuming, rather
> than
> >>>>> reading from earliest just because the partition is technically
> "new" to
> >>>>> the group.
> >>>>>
> >>>>> This is exactly why we opted for introducing a max.age threshold. It
> gives
> >>>>> users a time-bound way to define what is genuinely "hot/new" and
> what is
> >>>>> just an old partition they haven't seen yet.
> >>>>>
> >>>>> Best,
> >>>>> Chia-Ping
> >>>>>
> >>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>>>> Hi, Jiunn-Yang,
> >>>>>>
> >>>>>> Thanks for the KIP.
> >>>>>>
> >>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only
> applies when
> >>>>>> auto.offset.reset is latest. However, it seems that the motivation
> >>>>> equally
> >>>>>> applies when auto.offset.reset is set to other values like
> by_duration.
> >>>>> The
> >>>>>> intention is that we want to have a separate way to control newly
> created
> >>>>>> partitions vs existing partitions when the group starts. Have we
> >>>>> considered
> >>>>>> adding a new config like auto.offset.reset.new.partitions? If this
> new
> >>>>>> config is not set, the offset reset policy defaults to the policy
> used
> >>>>> for
> >>>>>> existing partitions. The user could set it explicitly to customize
> the
> >>>>>> behavior for new partitions.
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> I’d like to manually bump this thread.
> >>>>>>>
> >>>>>>> Best Regards,
> >>>>>>> Jiunn-Yang
> >>>>>>>
> >>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>>>>
> >>>>>>>> Hello all,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback.
> >>>>>>>>
> >>>>>>>> DJ01/DJ02:
> >>>>>>>>
> >>>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata
> struct
> >>>>>>> gains a new
> >>>>>>>> field PartitionAgeMs (int64, default -1), computed server-side by
> the
> >>>>>>> broker as
> >>>>>>>> broker_current_time - partition_creation_time.
> >>>>>>>>
> >>>>>>>> Also add the consumer heartbeat flow. when MembershipManager
> detects
> >>>>> a
> >>>>>>> newly assigned
> >>>>>>>> partition, it explicitly invalidates the metadata for the affected
> >>>>> topic
> >>>>>>> and forces a fresh MetadataRequest
> >>>>>>>> before making the offset reset decision, even if the topic ID is
> >>>>> already
> >>>>>>> in the cache.
> >>>>>>>>
> >>>>>>>> MB0:
> >>>>>>>>
> >>>>>>>> The consumer learns the broker's maximum supported
> MetadataResponse
> >>>>>>> version via the
> >>>>>>>> ApiVersions negotiation at connection time. If the negotiated
> >>>>> version is
> >>>>>>> unsupported, the consumer
> >>>>>>>> knows the broker does not support PartitionAgeMs at all and can
> >>>>> throw an
> >>>>>>> UnsupportedVersionException
> >>>>>>>> immediately, rather than silently falling back to latest and
> risking
> >>>>>>> data loss without any operator-visible signal.
> >>>>>>>>
> >>>>>>>> MB1/MB2/MB3:
> >>>>>>>>
> >>>>>>>> I have addressed these changes in the KIP.
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>> Jiunn-Yang
> >>>>>>>>
> >>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> >>>>>>>>>
> >>>>>>>>> hi David
> >>>>>>>>>
> >>>>>>>>> I agree with the direction of moving the 'age' resolution from
> the
> >>>>>>> Heartbeat API to the Metadata API to keep the control plane clean.
> The
> >>>>> main
> >>>>>>> trade-off, as we noted before, is introducing inter-broker clock
> skew.
> >>>>> The
> >>>>>>> Group Coordinator approach provided a single source of truth for
> time.
> >>>>>>>>>
> >>>>>>>>> However, realistically, this time skew should be negligible.
> Given
> >>>>> that
> >>>>>>> the max.age threshold will likely be configured in minutes or
> hours, a
> >>>>>>> typical NTP skew (in milliseconds) between brokers won't impact the
> >>>>>>> fallback decision.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Chia-Ping
> >>>>>>>>>
> >>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29
> 寫道:
> >>>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>
> >>>>>>>>>> Sorry, I haven't really followed the previous conversation but I
> >>>>> took a
> >>>>>>>>>> quick look at this one.
> >>>>>>>>>>
> >>>>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>>>> ConsumerGroupHeartbeat
> >>>>>>>>>> API after reading the KIP. There is a new boolean; the KIP
> states
> >>>>> that
> >>>>>>>>>> partition ages are returned only when this boolean is set.
> >>>>> Implicitly,
> >>>>>>> this
> >>>>>>>>>> means that when the consumer receives a new partition, it will
> >>>>> issue a
> >>>>>>> new
> >>>>>>>>>> HB request with the boolean set to receive the ages. Is my
> >>>>>>> understanding
> >>>>>>>>>> correct? We should perhaps clarify the flow and also explain
> how it
> >>>>>>> fits
> >>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
> >>>>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this given
> >>>>> that
> >>>>>>> a new
> >>>>>>>>>> round trip is done anyway. Alternatively, it could simply
> include
> >>>>> the
> >>>>>>>>>> metadata. Generally, we should be rather cautious about not
> >>>>> overloading
> >>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The API
> is
> >>>>> a
> >>>>>>>>>> control plane API for assigning or revoking partitions. The fact
> >>>>> that
> >>>>>>> we
> >>>>>>>>>> don't want to add it to the corresponding Streams API also
> suggests
> >>>>>>>>>> something is not quite right. What would we do if we want to
> >>>>> support
> >>>>>>>>>> Streams in the future?
> >>>>>>>>>>
> >>>>>>>>>> Best,
> >>>>>>>>>> David
> >>>>>>>>>>
> >>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
> >>>>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Jiunn,
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you for this great kip. Good to know about the gap.
> >>>>>>>>>>>
> >>>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges
> field.
> >>>>> Can a
> >>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
> >>>>> TopicPartitions)
> >>>>>>> be
> >>>>>>>>>>> used here and avoid version bump?
> >>>>>>>>>>>
> >>>>>>>>>>> mb-1 - For the new config, is there a recommended value or a
> >>>>> ConfigDef
> >>>>>>>>>>> validator? Probably it should based on the metadata.max.age.ms
> ?
> >>>>>>> Sizing
> >>>>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>>>>
> >>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams, would
> it
> >>>>> be
> >>>>>>> better
> >>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to the
> >>>>>>>>>>> StreamsConfig block list
> >>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>>>> for a
> >>>>>>>>>>> clear warning, incase users configure it? This is the most
> >>>>> familiar
> >>>>>>>>>>> consumer config and users might easily mistakenly configure
> it. Or
> >>>>>>> may be
> >>>>>>>>>>> it's not worth it to add.
> >>>>>>>>>>>
> >>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to
> earliest"
> >>>>>>> reads as
> >>>>>>>>>>> if the config were being changed per-partition which isn't
> >>>>> supported.
> >>>>>>> May
> >>>>>>>>>>> be rephrasing to something like "consumer resolves the initial
> >>>>>>> position to
> >>>>>>>>>>> start offset for that partition" as if earliest was applied to
> >>>>> that
> >>>>>>>>>>> partition only and auto.offset.reset config is unchanged.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Murali
> >>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi chia,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Best Regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> chia_0: Should we expose the partition creation time via the
> >>>>> Admin
> >>>>>>> API?
> >>>>>>>>>>>> I assume it would be valuable for users to diagnose and
> >>>>> troubleshoot
> >>>>>>> the
> >>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Best,
> >>>>>>>>>>>>> Chia-Ping
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot
> Data
> >>>>>>> Loss
> >>>>>>>>>>> on
> >>>>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>>>> <
> >>>>>>>>>>>
> >>>>>>>
> >>>>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This proposal aims to introduces
> >>>>> auto.offset.reset.latest.max.age,
> >>>>>>> a
> >>>>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot)
> partitions
> >>>>>>> from
> >>>>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>>>> younger than the configured threshold automatically fall
> back
> >>>>> to
> >>>>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>>>> during topic expansion without forcing a full historical
> >>>>> reprocess.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best regards,
> >>>>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello chia,

Thanks for the feedback,

> If the creation time exists, the returned value should always be greater than or equal to zero, right?
I have explicitly mentioned this in the KIP.

>> New	Old (MetadataResponse v0–13)	positive	any 	field absent	UnsupportedVersionException

The earliest point at which we can detect the version mismatch is during the 
first metadata fetch after assignment, which occurs inside poll(). Therefore, the 
user would encounter an UnsupportedVersionException from poll(). I’ll clarify this in the KIP.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月17日 下午4:50 寫道:
> 
> hi Jiunn
> 
>> PartitionAgeMs (int64, default -1): The age of this partition in milliseconds, computed server-side by the broker as broker_current_time - partition_creation_time. Returns -1 if the broker does not support this feature or the partition creation time is unknown.
> 
> If the creation time exists, the returned value should always be greater than or equal to zero, right?
> 
>> New	Old (MetadataResponse v0–13)	positive	any 	field absent	UnsupportedVersionException
> 
> Will user encounter UnsupportedVersionException when calling `poll()`?
> 
> Best,
> Chia-Ping
> 
> 
> On 2026/05/16 04:30:49 黃竣陽 wrote:
>> Hello Jun, chia,
>> 
>> I've updated KIP-1327 with a design change based on the discussion 
>> feedback. 
>> 
>> The updated design decouples the new-partition reset behavior from 
>> the base auto.offset.reset policy: 
>> 
>> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset values 
>> (latest, earliest, by_duration, none). 
>> - For new ("hot") partitions, the consumer resets to auto.offset.reset.new.partitions 
>> config setting
>> - For existing ("cold") partitions, the base auto.offset.reset policy continues 
>> to apply unchanged. 
>> - The new-partition reset behavior is represented by a separate internal config 
>> (auto.offset.reset.new.partitions, currently fixed to earliest). This decoupled design makes 
>> it straightforward to promote the behavior to a public user-facing configuration in a future KIP.
>> 
>> Best Regards,
>> Jiunn-Yang
>> 
>> 
>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
>>> 
>>> hi Jun
>>> 
>>> I see what you mean now. The proposal from me is listed below:
>>> 
>>> 1) Add auto.offset.reset.new.partitions with a default value of earliest. It fixes the data loss from both by_duration and latest, and it does not change the logic of auto.offset.reset=earliest.
>>> 2) Mark auto.offset.reset.new.partitions as an internal configuration. auto.offset.reset.new.partitions=earliest already addresses the issue, and we can discuss the use cases of other values in a separate KIP.
>>> 3) Both configs, auto.offset.reset.new.partitions and auto.offset.reset.latest.max.age.ms, will be applied to all for consistency.
>>> 
>>> WDYT?
>>> 
>>> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>>>> Hi, Chia-Ping,
>>>> 
>>>> Thanks for the reply.
>>>> 
>>>> 1. In the motivation section, the KIP says "When a Kafka topic is expanded
>>>> with new partitions, consumers using the latest auto offset reset policy
>>>> will silently miss all records produced to those partitions before the
>>>> consumer discovers them.". If a user sets
>>>> auto.offset.reset=by_duration=1sec, the same record loss issue could also
>>>> happen, right?
>>>> 
>>>> 2. I was thinking auto.offset.reset.new.partitions will take the same
>>>> values as auto.offset.reset. So a user could set it by_duration if needed.
>>>> 
>>>> Jun
>>>> 
>>>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <ch...@apache.org> wrote:
>>>> 
>>>>> hi Jun
>>>>> 
>>>>> Thanks for the feedback. I might be missing something important from your
>>>>> suggestion, so please bear with me as I try to clarify with a few questions:
>>>>> 
>>>>> 1. Is there a strong use case for extending this logic to other reset
>>>>> policies? Unlike latest, policies like earliest or by_duration don't seem
>>>>> to suffer from the same silent data loss issue when a partition is expanded.
>>>>> 
>>>>> 2. What values would we expect users to configure for
>>>>> auto.offset.reset.new.partitions? If they set it to earliest or latest,
>>>>> we might run into the exact same edge cases. For example, if a consumer is
>>>>> offline for a while and a new partition is created during that downtime,
>>>>> the user might actually want to skip to latest when resuming, rather than
>>>>> reading from earliest just because the partition is technically "new" to
>>>>> the group.
>>>>> 
>>>>> This is exactly why we opted for introducing a max.age threshold. It gives
>>>>> users a time-bound way to define what is genuinely "hot/new" and what is
>>>>> just an old partition they haven't seen yet.
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>>>> Hi, Jiunn-Yang,
>>>>>> 
>>>>>> Thanks for the KIP.
>>>>>> 
>>>>>> I find auto.offset.reset.latest.max.age a bit weird. It only applies when
>>>>>> auto.offset.reset is latest. However, it seems that the motivation
>>>>> equally
>>>>>> applies when auto.offset.reset is set to other values like by_duration.
>>>>> The
>>>>>> intention is that we want to have a separate way to control newly created
>>>>>> partitions vs existing partitions when the group starts. Have we
>>>>> considered
>>>>>> adding a new config like auto.offset.reset.new.partitions? If this new
>>>>>> config is not set, the offset reset policy defaults to the policy used
>>>>> for
>>>>>> existing partitions. The user could set it explicitly to customize the
>>>>>> behavior for new partitions.
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>> 
>>>>>>> Hi all,
>>>>>>> 
>>>>>>> I’d like to manually bump this thread.
>>>>>>> 
>>>>>>> Best Regards,
>>>>>>> Jiunn-Yang
>>>>>>> 
>>>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>>>> 
>>>>>>>> Hello all,
>>>>>>>> 
>>>>>>>> Thanks for the feedback.
>>>>>>>> 
>>>>>>>> DJ01/DJ02:
>>>>>>>> 
>>>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
>>>>>>> gains a new
>>>>>>>> field PartitionAgeMs (int64, default -1), computed server-side by the
>>>>>>> broker as
>>>>>>>> broker_current_time - partition_creation_time.
>>>>>>>> 
>>>>>>>> Also add the consumer heartbeat flow. when MembershipManager detects
>>>>> a
>>>>>>> newly assigned
>>>>>>>> partition, it explicitly invalidates the metadata for the affected
>>>>> topic
>>>>>>> and forces a fresh MetadataRequest
>>>>>>>> before making the offset reset decision, even if the topic ID is
>>>>> already
>>>>>>> in the cache.
>>>>>>>> 
>>>>>>>> MB0:
>>>>>>>> 
>>>>>>>> The consumer learns the broker's maximum supported MetadataResponse
>>>>>>> version via the
>>>>>>>> ApiVersions negotiation at connection time. If the negotiated
>>>>> version is
>>>>>>> unsupported, the consumer
>>>>>>>> knows the broker does not support PartitionAgeMs at all and can
>>>>> throw an
>>>>>>> UnsupportedVersionException
>>>>>>>> immediately, rather than silently falling back to latest and risking
>>>>>>> data loss without any operator-visible signal.
>>>>>>>> 
>>>>>>>> MB1/MB2/MB3:
>>>>>>>> 
>>>>>>>> I have addressed these changes in the KIP.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
>>>>>>>>> 
>>>>>>>>> hi David
>>>>>>>>> 
>>>>>>>>> I agree with the direction of moving the 'age' resolution from the
>>>>>>> Heartbeat API to the Metadata API to keep the control plane clean. The
>>>>> main
>>>>>>> trade-off, as we noted before, is introducing inter-broker clock skew.
>>>>> The
>>>>>>> Group Coordinator approach provided a single source of truth for time.
>>>>>>>>> 
>>>>>>>>> However, realistically, this time skew should be negligible. Given
>>>>> that
>>>>>>> the max.age threshold will likely be configured in minutes or hours, a
>>>>>>> typical NTP skew (in milliseconds) between brokers won't impact the
>>>>>>> fallback decision.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
>>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>> 
>>>>>>>>>> Sorry, I haven't really followed the previous conversation but I
>>>>> took a
>>>>>>>>>> quick look at this one.
>>>>>>>>>> 
>>>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>>>> ConsumerGroupHeartbeat
>>>>>>>>>> API after reading the KIP. There is a new boolean; the KIP states
>>>>> that
>>>>>>>>>> partition ages are returned only when this boolean is set.
>>>>> Implicitly,
>>>>>>> this
>>>>>>>>>> means that when the consumer receives a new partition, it will
>>>>> issue a
>>>>>>> new
>>>>>>>>>> HB request with the boolean set to receive the ages. Is my
>>>>>>> understanding
>>>>>>>>>> correct? We should perhaps clarify the flow and also explain how it
>>>>>>> fits
>>>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
>>>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this given
>>>>> that
>>>>>>> a new
>>>>>>>>>> round trip is done anyway. Alternatively, it could simply include
>>>>> the
>>>>>>>>>> metadata. Generally, we should be rather cautious about not
>>>>> overloading
>>>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is
>>>>> a
>>>>>>>>>> control plane API for assigning or revoking partitions. The fact
>>>>> that
>>>>>>> we
>>>>>>>>>> don't want to add it to the corresponding Streams API also suggests
>>>>>>>>>> something is not quite right. What would we do if we want to
>>>>> support
>>>>>>>>>> Streams in the future?
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> David
>>>>>>>>>> 
>>>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
>>>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> Hi Jiunn,
>>>>>>>>>>> 
>>>>>>>>>>> Thank you for this great kip. Good to know about the gap.
>>>>>>>>>>> 
>>>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges field.
>>>>> Can a
>>>>>>>>>>> tagged field (for ex: on response, PartitionAges on
>>>>> TopicPartitions)
>>>>>>> be
>>>>>>>>>>> used here and avoid version bump?
>>>>>>>>>>> 
>>>>>>>>>>> mb-1 - For the new config, is there a recommended value or a
>>>>> ConfigDef
>>>>>>>>>>> validator? Probably it should based on the metadata.max.age.ms ?
>>>>>>> Sizing
>>>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>>>> 
>>>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it
>>>>> be
>>>>>>> better
>>>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to the
>>>>>>>>>>> StreamsConfig block list
>>>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>>>> for a
>>>>>>>>>>> clear warning, incase users configure it? This is the most
>>>>> familiar
>>>>>>>>>>> consumer config and users might easily mistakenly configure it. Or
>>>>>>> may be
>>>>>>>>>>> it's not worth it to add.
>>>>>>>>>>> 
>>>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
>>>>>>> reads as
>>>>>>>>>>> if the config were being changed per-partition which isn't
>>>>> supported.
>>>>>>> May
>>>>>>>>>>> be rephrasing to something like "consumer resolves the initial
>>>>>>> position to
>>>>>>>>>>> start offset for that partition" as if earliest was applied to
>>>>> that
>>>>>>>>>>> partition only and auto.offset.reset config is unchanged.
>>>>>>>>>>> 
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Murali
>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi chia,
>>>>>>>>>>>> 
>>>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>>>> 
>>>>>>>>>>>>> chia_0: Should we expose the partition creation time via the
>>>>> Admin
>>>>>>> API?
>>>>>>>>>>>> I assume it would be valuable for users to diagnose and
>>>>> troubleshoot
>>>>>>> the
>>>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> Chia-Ping
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
>>>>>>> Loss
>>>>>>>>>>> on
>>>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>>>> <
>>>>>>>>>>> 
>>>>>>> 
>>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This proposal aims to introduces
>>>>> auto.offset.reset.latest.max.age,
>>>>>>> a
>>>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot) partitions
>>>>>>> from
>>>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>>>> younger than the configured threshold automatically fall back
>>>>> to
>>>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>>>> during topic expansion without forcing a full historical
>>>>> reprocess.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Jiunn

> PartitionAgeMs (int64, default -1): The age of this partition in milliseconds, computed server-side by the broker as broker_current_time - partition_creation_time. Returns -1 if the broker does not support this feature or the partition creation time is unknown.

If the creation time exists, the returned value should always be greater than or equal to zero, right?

> New	Old (MetadataResponse v0–13)	positive	any 	field absent	UnsupportedVersionException

Will user encounter UnsupportedVersionException when calling `poll()`?

Best,
Chia-Ping
 

On 2026/05/16 04:30:49 黃竣陽 wrote:
> Hello Jun, chia,
> 
> I've updated KIP-1327 with a design change based on the discussion 
> feedback. 
> 
> The updated design decouples the new-partition reset behavior from 
> the base auto.offset.reset policy: 
> 
> - auto.offset.reset.max.age.ms now applies to all auto.offset.reset values 
> (latest, earliest, by_duration, none). 
> - For new ("hot") partitions, the consumer resets to auto.offset.reset.new.partitions 
> config setting
> - For existing ("cold") partitions, the base auto.offset.reset policy continues 
> to apply unchanged. 
> - The new-partition reset behavior is represented by a separate internal config 
> (auto.offset.reset.new.partitions, currently fixed to earliest). This decoupled design makes 
> it straightforward to promote the behavior to a public user-facing configuration in a future KIP.
> 
> Best Regards,
> Jiunn-Yang
> 
> 
> > Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
> > 
> > hi Jun
> > 
> > I see what you mean now. The proposal from me is listed below:
> > 
> > 1) Add auto.offset.reset.new.partitions with a default value of earliest. It fixes the data loss from both by_duration and latest, and it does not change the logic of auto.offset.reset=earliest.
> > 2) Mark auto.offset.reset.new.partitions as an internal configuration. auto.offset.reset.new.partitions=earliest already addresses the issue, and we can discuss the use cases of other values in a separate KIP.
> > 3) Both configs, auto.offset.reset.new.partitions and auto.offset.reset.latest.max.age.ms, will be applied to all for consistency.
> > 
> > WDYT?
> > 
> > On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> >> Hi, Chia-Ping,
> >> 
> >> Thanks for the reply.
> >> 
> >> 1. In the motivation section, the KIP says "When a Kafka topic is expanded
> >> with new partitions, consumers using the latest auto offset reset policy
> >> will silently miss all records produced to those partitions before the
> >> consumer discovers them.". If a user sets
> >> auto.offset.reset=by_duration=1sec, the same record loss issue could also
> >> happen, right?
> >> 
> >> 2. I was thinking auto.offset.reset.new.partitions will take the same
> >> values as auto.offset.reset. So a user could set it by_duration if needed.
> >> 
> >> Jun
> >> 
> >> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <ch...@apache.org> wrote:
> >> 
> >>> hi Jun
> >>> 
> >>> Thanks for the feedback. I might be missing something important from your
> >>> suggestion, so please bear with me as I try to clarify with a few questions:
> >>> 
> >>> 1. Is there a strong use case for extending this logic to other reset
> >>> policies? Unlike latest, policies like earliest or by_duration don't seem
> >>> to suffer from the same silent data loss issue when a partition is expanded.
> >>> 
> >>> 2. What values would we expect users to configure for
> >>> auto.offset.reset.new.partitions? If they set it to earliest or latest,
> >>> we might run into the exact same edge cases. For example, if a consumer is
> >>> offline for a while and a new partition is created during that downtime,
> >>> the user might actually want to skip to latest when resuming, rather than
> >>> reading from earliest just because the partition is technically "new" to
> >>> the group.
> >>> 
> >>> This is exactly why we opted for introducing a max.age threshold. It gives
> >>> users a time-bound way to define what is genuinely "hot/new" and what is
> >>> just an old partition they haven't seen yet.
> >>> 
> >>> Best,
> >>> Chia-Ping
> >>> 
> >>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> >>>> Hi, Jiunn-Yang,
> >>>> 
> >>>> Thanks for the KIP.
> >>>> 
> >>>> I find auto.offset.reset.latest.max.age a bit weird. It only applies when
> >>>> auto.offset.reset is latest. However, it seems that the motivation
> >>> equally
> >>>> applies when auto.offset.reset is set to other values like by_duration.
> >>> The
> >>>> intention is that we want to have a separate way to control newly created
> >>>> partitions vs existing partitions when the group starts. Have we
> >>> considered
> >>>> adding a new config like auto.offset.reset.new.partitions? If this new
> >>>> config is not set, the offset reset policy defaults to the policy used
> >>> for
> >>>> existing partitions. The user could set it explicitly to customize the
> >>>> behavior for new partitions.
> >>>> 
> >>>> Jun
> >>>> 
> >>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
> >>>> 
> >>>>> Hi all,
> >>>>> 
> >>>>> I’d like to manually bump this thread.
> >>>>> 
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>> 
> >>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >>>>>> 
> >>>>>> Hello all,
> >>>>>> 
> >>>>>> Thanks for the feedback.
> >>>>>> 
> >>>>>> DJ01/DJ02:
> >>>>>> 
> >>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
> >>>>> gains a new
> >>>>>> field PartitionAgeMs (int64, default -1), computed server-side by the
> >>>>> broker as
> >>>>>> broker_current_time - partition_creation_time.
> >>>>>> 
> >>>>>> Also add the consumer heartbeat flow. when MembershipManager detects
> >>> a
> >>>>> newly assigned
> >>>>>> partition, it explicitly invalidates the metadata for the affected
> >>> topic
> >>>>> and forces a fresh MetadataRequest
> >>>>>> before making the offset reset decision, even if the topic ID is
> >>> already
> >>>>> in the cache.
> >>>>>> 
> >>>>>> MB0:
> >>>>>> 
> >>>>>> The consumer learns the broker's maximum supported MetadataResponse
> >>>>> version via the
> >>>>>> ApiVersions negotiation at connection time. If the negotiated
> >>> version is
> >>>>> unsupported, the consumer
> >>>>>> knows the broker does not support PartitionAgeMs at all and can
> >>> throw an
> >>>>> UnsupportedVersionException
> >>>>>> immediately, rather than silently falling back to latest and risking
> >>>>> data loss without any operator-visible signal.
> >>>>>> 
> >>>>>> MB1/MB2/MB3:
> >>>>>> 
> >>>>>> I have addressed these changes in the KIP.
> >>>>>> 
> >>>>>> Best Regards,
> >>>>>> Jiunn-Yang
> >>>>>> 
> >>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> >>>>>>> 
> >>>>>>> hi David
> >>>>>>> 
> >>>>>>> I agree with the direction of moving the 'age' resolution from the
> >>>>> Heartbeat API to the Metadata API to keep the control plane clean. The
> >>> main
> >>>>> trade-off, as we noted before, is introducing inter-broker clock skew.
> >>> The
> >>>>> Group Coordinator approach provided a single source of truth for time.
> >>>>>>> 
> >>>>>>> However, realistically, this time skew should be negligible. Given
> >>> that
> >>>>> the max.age threshold will likely be configured in minutes or hours, a
> >>>>> typical NTP skew (in milliseconds) between brokers won't impact the
> >>>>> fallback decision.
> >>>>>>> 
> >>>>>>> Best,
> >>>>>>> Chia-Ping
> >>>>>>> 
> >>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
> >>>>>>>> 
> >>>>>>>> Hi all,
> >>>>>>>> 
> >>>>>>>> Thanks for the KIP!
> >>>>>>>> 
> >>>>>>>> Sorry, I haven't really followed the previous conversation but I
> >>> took a
> >>>>>>>> quick look at this one.
> >>>>>>>> 
> >>>>>>>> DJ01: I don't clearly understand the flow with the
> >>>>> ConsumerGroupHeartbeat
> >>>>>>>> API after reading the KIP. There is a new boolean; the KIP states
> >>> that
> >>>>>>>> partition ages are returned only when this boolean is set.
> >>> Implicitly,
> >>>>> this
> >>>>>>>> means that when the consumer receives a new partition, it will
> >>> issue a
> >>>>> new
> >>>>>>>> HB request with the boolean set to receive the ages. Is my
> >>>>> understanding
> >>>>>>>> correct? We should perhaps clarify the flow and also explain how it
> >>>>> fits
> >>>>>>>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
> >>>>>>>> DJ02: It my understanding is correct, I wonder if
> >>>>>>>> the ConsumerGroupHeartbeat API is the right place for this given
> >>> that
> >>>>> a new
> >>>>>>>> round trip is done anyway. Alternatively, it could simply include
> >>> the
> >>>>>>>> metadata. Generally, we should be rather cautious about not
> >>> overloading
> >>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is
> >>> a
> >>>>>>>> control plane API for assigning or revoking partitions. The fact
> >>> that
> >>>>> we
> >>>>>>>> don't want to add it to the corresponding Streams API also suggests
> >>>>>>>> something is not quite right. What would we do if we want to
> >>> support
> >>>>>>>> Streams in the future?
> >>>>>>>> 
> >>>>>>>> Best,
> >>>>>>>> David
> >>>>>>>> 
> >>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
> >>>>>>>>> dev@kafka.apache.org> wrote:
> >>>>>>>>> 
> >>>>>>>>> Hi Jiunn,
> >>>>>>>>> 
> >>>>>>>>> Thank you for this great kip. Good to know about the gap.
> >>>>>>>>> 
> >>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges field.
> >>> Can a
> >>>>>>>>> tagged field (for ex: on response, PartitionAges on
> >>> TopicPartitions)
> >>>>> be
> >>>>>>>>> used here and avoid version bump?
> >>>>>>>>> 
> >>>>>>>>> mb-1 - For the new config, is there a recommended value or a
> >>> ConfigDef
> >>>>>>>>> validator? Probably it should based on the metadata.max.age.ms ?
> >>>>> Sizing
> >>>>>>>>> instructions can be part of javadocs I guess.
> >>>>>>>>> 
> >>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it
> >>> be
> >>>>> better
> >>>>>>>>> to add this new config auto.offset.reset.latest.max.age to the
> >>>>>>>>> StreamsConfig block list
> >>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> >>>>> for a
> >>>>>>>>> clear warning, incase users configure it? This is the most
> >>> familiar
> >>>>>>>>> consumer config and users might easily mistakenly configure it. Or
> >>>>> may be
> >>>>>>>>> it's not worth it to add.
> >>>>>>>>> 
> >>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
> >>>>> reads as
> >>>>>>>>> if the config were being changed per-partition which isn't
> >>> supported.
> >>>>> May
> >>>>>>>>> be rephrasing to something like "consumer resolves the initial
> >>>>> position to
> >>>>>>>>> start offset for that partition" as if earliest was applied to
> >>> that
> >>>>>>>>> partition only and auto.offset.reset config is unchanged.
> >>>>>>>>> 
> >>>>>>>>> Thanks,
> >>>>>>>>> Murali
> >>>>>>>>> 
> >>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>>>>>> 
> >>>>>>>>>> Hi chia,
> >>>>>>>>>> 
> >>>>>>>>>> I have updated the KIP to include this change.
> >>>>>>>>>> 
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Jiunn-Yang
> >>>>>>>>>> 
> >>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
> >>>>>>>>>>> 
> >>>>>>>>>>> hi Jiunn-Yang
> >>>>>>>>>>> 
> >>>>>>>>>>> chia_0: Should we expose the partition creation time via the
> >>> Admin
> >>>>> API?
> >>>>>>>>>> I assume it would be valuable for users to diagnose and
> >>> troubleshoot
> >>>>> the
> >>>>>>>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>>>>>> 
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Chia-Ping
> >>>>>>>>>>> 
> >>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
> >>>>> Loss
> >>>>>>>>> on
> >>>>>>>>>> Partition Expansion for Latest Policy
> >>>>>>>>>>>> <
> >>>>>>>>> 
> >>>>> 
> >>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>>>> This proposal aims to introduces
> >>> auto.offset.reset.latest.max.age,
> >>>>> a
> >>>>>>>>>> consumer config that lets the
> >>>>>>>>>>>> latest reset policy distinguish newly expanded (hot) partitions
> >>>>> from
> >>>>>>>>>> long-existing (cold) ones. Partitions
> >>>>>>>>>>>> younger than the configured threshold automatically fall back
> >>> to
> >>>>>>>>>> earliest, preventing silent data loss
> >>>>>>>>>>>> during topic expansion without forcing a full historical
> >>> reprocess.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Best regards,
> >>>>>>>>>>>> Jiunn-Yang
> >>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>> 
> >>>>>> 
> >>>>> 
> >>>>> 
> >>>> 
> >>> 
> >> 
> 
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hello Jun, chia,

I've updated KIP-1327 with a design change based on the discussion 
feedback. 

The updated design decouples the new-partition reset behavior from 
the base auto.offset.reset policy: 

- auto.offset.reset.max.age.ms now applies to all auto.offset.reset values 
(latest, earliest, by_duration, none). 
- For new ("hot") partitions, the consumer resets to auto.offset.reset.new.partitions 
config setting
- For existing ("cold") partitions, the base auto.offset.reset policy continues 
to apply unchanged. 
- The new-partition reset behavior is represented by a separate internal config 
(auto.offset.reset.new.partitions, currently fixed to earliest). This decoupled design makes 
it straightforward to promote the behavior to a public user-facing configuration in a future KIP.

Best Regards,
Jiunn-Yang


> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月16日 清晨7:46 寫道:
> 
> hi Jun
> 
> I see what you mean now. The proposal from me is listed below:
> 
> 1) Add auto.offset.reset.new.partitions with a default value of earliest. It fixes the data loss from both by_duration and latest, and it does not change the logic of auto.offset.reset=earliest.
> 2) Mark auto.offset.reset.new.partitions as an internal configuration. auto.offset.reset.new.partitions=earliest already addresses the issue, and we can discuss the use cases of other values in a separate KIP.
> 3) Both configs, auto.offset.reset.new.partitions and auto.offset.reset.latest.max.age.ms, will be applied to all for consistency.
> 
> WDYT?
> 
> On 2026/05/15 20:53:20 Jun Rao via dev wrote:
>> Hi, Chia-Ping,
>> 
>> Thanks for the reply.
>> 
>> 1. In the motivation section, the KIP says "When a Kafka topic is expanded
>> with new partitions, consumers using the latest auto offset reset policy
>> will silently miss all records produced to those partitions before the
>> consumer discovers them.". If a user sets
>> auto.offset.reset=by_duration=1sec, the same record loss issue could also
>> happen, right?
>> 
>> 2. I was thinking auto.offset.reset.new.partitions will take the same
>> values as auto.offset.reset. So a user could set it by_duration if needed.
>> 
>> Jun
>> 
>> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <ch...@apache.org> wrote:
>> 
>>> hi Jun
>>> 
>>> Thanks for the feedback. I might be missing something important from your
>>> suggestion, so please bear with me as I try to clarify with a few questions:
>>> 
>>> 1. Is there a strong use case for extending this logic to other reset
>>> policies? Unlike latest, policies like earliest or by_duration don't seem
>>> to suffer from the same silent data loss issue when a partition is expanded.
>>> 
>>> 2. What values would we expect users to configure for
>>> auto.offset.reset.new.partitions? If they set it to earliest or latest,
>>> we might run into the exact same edge cases. For example, if a consumer is
>>> offline for a while and a new partition is created during that downtime,
>>> the user might actually want to skip to latest when resuming, rather than
>>> reading from earliest just because the partition is technically "new" to
>>> the group.
>>> 
>>> This is exactly why we opted for introducing a max.age threshold. It gives
>>> users a time-bound way to define what is genuinely "hot/new" and what is
>>> just an old partition they haven't seen yet.
>>> 
>>> Best,
>>> Chia-Ping
>>> 
>>> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>>>> Hi, Jiunn-Yang,
>>>> 
>>>> Thanks for the KIP.
>>>> 
>>>> I find auto.offset.reset.latest.max.age a bit weird. It only applies when
>>>> auto.offset.reset is latest. However, it seems that the motivation
>>> equally
>>>> applies when auto.offset.reset is set to other values like by_duration.
>>> The
>>>> intention is that we want to have a separate way to control newly created
>>>> partitions vs existing partitions when the group starts. Have we
>>> considered
>>>> adding a new config like auto.offset.reset.new.partitions? If this new
>>>> config is not set, the offset reset policy defaults to the policy used
>>> for
>>>> existing partitions. The user could set it explicitly to customize the
>>>> behavior for new partitions.
>>>> 
>>>> Jun
>>>> 
>>>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> I’d like to manually bump this thread.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>>>> 
>>>>>> Hello all,
>>>>>> 
>>>>>> Thanks for the feedback.
>>>>>> 
>>>>>> DJ01/DJ02:
>>>>>> 
>>>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
>>>>> gains a new
>>>>>> field PartitionAgeMs (int64, default -1), computed server-side by the
>>>>> broker as
>>>>>> broker_current_time - partition_creation_time.
>>>>>> 
>>>>>> Also add the consumer heartbeat flow. when MembershipManager detects
>>> a
>>>>> newly assigned
>>>>>> partition, it explicitly invalidates the metadata for the affected
>>> topic
>>>>> and forces a fresh MetadataRequest
>>>>>> before making the offset reset decision, even if the topic ID is
>>> already
>>>>> in the cache.
>>>>>> 
>>>>>> MB0:
>>>>>> 
>>>>>> The consumer learns the broker's maximum supported MetadataResponse
>>>>> version via the
>>>>>> ApiVersions negotiation at connection time. If the negotiated
>>> version is
>>>>> unsupported, the consumer
>>>>>> knows the broker does not support PartitionAgeMs at all and can
>>> throw an
>>>>> UnsupportedVersionException
>>>>>> immediately, rather than silently falling back to latest and risking
>>>>> data loss without any operator-visible signal.
>>>>>> 
>>>>>> MB1/MB2/MB3:
>>>>>> 
>>>>>> I have addressed these changes in the KIP.
>>>>>> 
>>>>>> Best Regards,
>>>>>> Jiunn-Yang
>>>>>> 
>>>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
>>>>>>> 
>>>>>>> hi David
>>>>>>> 
>>>>>>> I agree with the direction of moving the 'age' resolution from the
>>>>> Heartbeat API to the Metadata API to keep the control plane clean. The
>>> main
>>>>> trade-off, as we noted before, is introducing inter-broker clock skew.
>>> The
>>>>> Group Coordinator approach provided a single source of truth for time.
>>>>>>> 
>>>>>>> However, realistically, this time skew should be negligible. Given
>>> that
>>>>> the max.age threshold will likely be configured in minutes or hours, a
>>>>> typical NTP skew (in milliseconds) between brokers won't impact the
>>>>> fallback decision.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Chia-Ping
>>>>>>> 
>>>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Thanks for the KIP!
>>>>>>>> 
>>>>>>>> Sorry, I haven't really followed the previous conversation but I
>>> took a
>>>>>>>> quick look at this one.
>>>>>>>> 
>>>>>>>> DJ01: I don't clearly understand the flow with the
>>>>> ConsumerGroupHeartbeat
>>>>>>>> API after reading the KIP. There is a new boolean; the KIP states
>>> that
>>>>>>>> partition ages are returned only when this boolean is set.
>>> Implicitly,
>>>>> this
>>>>>>>> means that when the consumer receives a new partition, it will
>>> issue a
>>>>> new
>>>>>>>> HB request with the boolean set to receive the ages. Is my
>>>>> understanding
>>>>>>>> correct? We should perhaps clarify the flow and also explain how it
>>>>> fits
>>>>>>>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
>>>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>>>> the ConsumerGroupHeartbeat API is the right place for this given
>>> that
>>>>> a new
>>>>>>>> round trip is done anyway. Alternatively, it could simply include
>>> the
>>>>>>>> metadata. Generally, we should be rather cautious about not
>>> overloading
>>>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is
>>> a
>>>>>>>> control plane API for assigning or revoking partitions. The fact
>>> that
>>>>> we
>>>>>>>> don't want to add it to the corresponding Streams API also suggests
>>>>>>>> something is not quite right. What would we do if we want to
>>> support
>>>>>>>> Streams in the future?
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> David
>>>>>>>> 
>>>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
>>>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Jiunn,
>>>>>>>>> 
>>>>>>>>> Thank you for this great kip. Good to know about the gap.
>>>>>>>>> 
>>>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges field.
>>> Can a
>>>>>>>>> tagged field (for ex: on response, PartitionAges on
>>> TopicPartitions)
>>>>> be
>>>>>>>>> used here and avoid version bump?
>>>>>>>>> 
>>>>>>>>> mb-1 - For the new config, is there a recommended value or a
>>> ConfigDef
>>>>>>>>> validator? Probably it should based on the metadata.max.age.ms ?
>>>>> Sizing
>>>>>>>>> instructions can be part of javadocs I guess.
>>>>>>>>> 
>>>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it
>>> be
>>>>> better
>>>>>>>>> to add this new config auto.offset.reset.latest.max.age to the
>>>>>>>>> StreamsConfig block list
>>> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>>>> for a
>>>>>>>>> clear warning, incase users configure it? This is the most
>>> familiar
>>>>>>>>> consumer config and users might easily mistakenly configure it. Or
>>>>> may be
>>>>>>>>> it's not worth it to add.
>>>>>>>>> 
>>>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
>>>>> reads as
>>>>>>>>> if the config were being changed per-partition which isn't
>>> supported.
>>>>> May
>>>>>>>>> be rephrasing to something like "consumer resolves the initial
>>>>> position to
>>>>>>>>> start offset for that partition" as if earliest was applied to
>>> that
>>>>>>>>> partition only and auto.offset.reset config is unchanged.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Murali
>>>>>>>>> 
>>>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi chia,
>>>>>>>>>> 
>>>>>>>>>> I have updated the KIP to include this change.
>>>>>>>>>> 
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
>>>>>>>>>>> 
>>>>>>>>>>> hi Jiunn-Yang
>>>>>>>>>>> 
>>>>>>>>>>> chia_0: Should we expose the partition creation time via the
>>> Admin
>>>>> API?
>>>>>>>>>> I assume it would be valuable for users to diagnose and
>>> troubleshoot
>>>>> the
>>>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>>>> 
>>>>>>>>>>> Best,
>>>>>>>>>>> Chia-Ping
>>>>>>>>>>> 
>>>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>> 
>>>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
>>>>> Loss
>>>>>>>>> on
>>>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>>>> <
>>>>>>>>> 
>>>>> 
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> This proposal aims to introduces
>>> auto.offset.reset.latest.max.age,
>>>>> a
>>>>>>>>>> consumer config that lets the
>>>>>>>>>>>> latest reset policy distinguish newly expanded (hot) partitions
>>>>> from
>>>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>>>> younger than the configured threshold automatically fall back
>>> to
>>>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>>>> during topic expansion without forcing a full historical
>>> reprocess.
>>>>>>>>>>>> 
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Jiunn-Yang
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Jun

I see what you mean now. The proposal from me is listed below:

1) Add auto.offset.reset.new.partitions with a default value of earliest. It fixes the data loss from both by_duration and latest, and it does not change the logic of auto.offset.reset=earliest.
2) Mark auto.offset.reset.new.partitions as an internal configuration. auto.offset.reset.new.partitions=earliest already addresses the issue, and we can discuss the use cases of other values in a separate KIP.
3) Both configs, auto.offset.reset.new.partitions and auto.offset.reset.latest.max.age.ms, will be applied to all for consistency.

WDYT?

On 2026/05/15 20:53:20 Jun Rao via dev wrote:
> Hi, Chia-Ping,
> 
> Thanks for the reply.
> 
> 1. In the motivation section, the KIP says "When a Kafka topic is expanded
> with new partitions, consumers using the latest auto offset reset policy
> will silently miss all records produced to those partitions before the
> consumer discovers them.". If a user sets
> auto.offset.reset=by_duration=1sec, the same record loss issue could also
> happen, right?
> 
> 2. I was thinking auto.offset.reset.new.partitions will take the same
> values as auto.offset.reset. So a user could set it by_duration if needed.
> 
> Jun
> 
> On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <ch...@apache.org> wrote:
> 
> > hi Jun
> >
> > Thanks for the feedback. I might be missing something important from your
> > suggestion, so please bear with me as I try to clarify with a few questions:
> >
> > 1. Is there a strong use case for extending this logic to other reset
> > policies? Unlike latest, policies like earliest or by_duration don't seem
> > to suffer from the same silent data loss issue when a partition is expanded.
> >
> > 2. What values would we expect users to configure for
> > auto.offset.reset.new.partitions? If they set it to earliest or latest,
> > we might run into the exact same edge cases. For example, if a consumer is
> > offline for a while and a new partition is created during that downtime,
> > the user might actually want to skip to latest when resuming, rather than
> > reading from earliest just because the partition is technically "new" to
> > the group.
> >
> > This is exactly why we opted for introducing a max.age threshold. It gives
> > users a time-bound way to define what is genuinely "hot/new" and what is
> > just an old partition they haven't seen yet.
> >
> > Best,
> > Chia-Ping
> >
> > On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > > Hi, Jiunn-Yang,
> > >
> > > Thanks for the KIP.
> > >
> > > I find auto.offset.reset.latest.max.age a bit weird. It only applies when
> > > auto.offset.reset is latest. However, it seems that the motivation
> > equally
> > > applies when auto.offset.reset is set to other values like by_duration.
> > The
> > > intention is that we want to have a separate way to control newly created
> > > partitions vs existing partitions when the group starts. Have we
> > considered
> > > adding a new config like auto.offset.reset.new.partitions? If this new
> > > config is not set, the offset reset policy defaults to the policy used
> > for
> > > existing partitions. The user could set it explicitly to customize the
> > > behavior for new partitions.
> > >
> > > Jun
> > >
> > > On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I’d like to manually bump this thread.
> > > >
> > > > Best Regards,
> > > > Jiunn-Yang
> > > >
> > > > > 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > Thanks for the feedback.
> > > > >
> > > > > DJ01/DJ02:
> > > > >
> > > > > MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
> > > > gains a new
> > > > > field PartitionAgeMs (int64, default -1), computed server-side by the
> > > > broker as
> > > > > broker_current_time - partition_creation_time.
> > > > >
> > > > > Also add the consumer heartbeat flow. when MembershipManager detects
> > a
> > > > newly assigned
> > > > > partition, it explicitly invalidates the metadata for the affected
> > topic
> > > > and forces a fresh MetadataRequest
> > > > > before making the offset reset decision, even if the topic ID is
> > already
> > > > in the cache.
> > > > >
> > > > > MB0:
> > > > >
> > > > > The consumer learns the broker's maximum supported MetadataResponse
> > > > version via the
> > > > > ApiVersions negotiation at connection time. If the negotiated
> > version is
> > > > unsupported, the consumer
> > > > > knows the broker does not support PartitionAgeMs at all and can
> > throw an
> > > > UnsupportedVersionException
> > > > > immediately, rather than silently falling back to latest and risking
> > > > data loss without any operator-visible signal.
> > > > >
> > > > > MB1/MB2/MB3:
> > > > >
> > > > > I have addressed these changes in the KIP.
> > > > >
> > > > > Best Regards,
> > > > > Jiunn-Yang
> > > > >
> > > > >> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> > > > >>
> > > > >> hi David
> > > > >>
> > > > >> I agree with the direction of moving the 'age' resolution from the
> > > > Heartbeat API to the Metadata API to keep the control plane clean. The
> > main
> > > > trade-off, as we noted before, is introducing inter-broker clock skew.
> > The
> > > > Group Coordinator approach provided a single source of truth for time.
> > > > >>
> > > > >> However, realistically, this time skew should be negligible. Given
> > that
> > > > the max.age threshold will likely be configured in minutes or hours, a
> > > > typical NTP skew (in milliseconds) between brokers won't impact the
> > > > fallback decision.
> > > > >>
> > > > >> Best,
> > > > >> Chia-Ping
> > > > >>
> > > > >>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
> > > > >>>
> > > > >>> Hi all,
> > > > >>>
> > > > >>> Thanks for the KIP!
> > > > >>>
> > > > >>> Sorry, I haven't really followed the previous conversation but I
> > took a
> > > > >>> quick look at this one.
> > > > >>>
> > > > >>> DJ01: I don't clearly understand the flow with the
> > > > ConsumerGroupHeartbeat
> > > > >>> API after reading the KIP. There is a new boolean; the KIP states
> > that
> > > > >>> partition ages are returned only when this boolean is set.
> > Implicitly,
> > > > this
> > > > >>> means that when the consumer receives a new partition, it will
> > issue a
> > > > new
> > > > >>> HB request with the boolean set to receive the ages. Is my
> > > > understanding
> > > > >>> correct? We should perhaps clarify the flow and also explain how it
> > > > fits
> > > > >>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
> > > > >>> DJ02: It my understanding is correct, I wonder if
> > > > >>> the ConsumerGroupHeartbeat API is the right place for this given
> > that
> > > > a new
> > > > >>> round trip is done anyway. Alternatively, it could simply include
> > the
> > > > >>> metadata. Generally, we should be rather cautious about not
> > overloading
> > > > >>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is
> > a
> > > > >>> control plane API for assigning or revoking partitions. The fact
> > that
> > > > we
> > > > >>> don't want to add it to the corresponding Streams API also suggests
> > > > >>> something is not quite right. What would we do if we want to
> > support
> > > > >>> Streams in the future?
> > > > >>>
> > > > >>> Best,
> > > > >>> David
> > > > >>>
> > > > >>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
> > > > >>>> dev@kafka.apache.org> wrote:
> > > > >>>>
> > > > >>>> Hi Jiunn,
> > > > >>>>
> > > > >>>> Thank you for this great kip. Good to know about the gap.
> > > > >>>>
> > > > >>>> mb-0 - why a new v2 version bump for RequestPartitionAges field.
> > Can a
> > > > >>>> tagged field (for ex: on response, PartitionAges on
> > TopicPartitions)
> > > > be
> > > > >>>> used here and avoid version bump?
> > > > >>>>
> > > > >>>> mb-1 - For the new config, is there a recommended value or a
> > ConfigDef
> > > > >>>> validator? Probably it should based on the metadata.max.age.ms ?
> > > > Sizing
> > > > >>>> instructions can be part of javadocs I guess.
> > > > >>>>
> > > > >>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it
> > be
> > > > better
> > > > >>>> to add this new config auto.offset.reset.latest.max.age to the
> > > > >>>> StreamsConfig block list
> > (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > > for a
> > > > >>>> clear warning, incase users configure it? This is the most
> > familiar
> > > > >>>> consumer config and users might easily mistakenly configure it. Or
> > > > may be
> > > > >>>> it's not worth it to add.
> > > > >>>>
> > > > >>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
> > > > reads as
> > > > >>>> if the config were being changed per-partition which isn't
> > supported.
> > > > May
> > > > >>>> be rephrasing to something like "consumer resolves the initial
> > > > position to
> > > > >>>> start offset for that partition" as if earliest was applied to
> > that
> > > > >>>> partition only and auto.offset.reset config is unchanged.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Murali
> > > > >>>>
> > > > >>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
> > > > >>>>>
> > > > >>>>> Hi chia,
> > > > >>>>>
> > > > >>>>> I have updated the KIP to include this change.
> > > > >>>>>
> > > > >>>>> Best Regards,
> > > > >>>>> Jiunn-Yang
> > > > >>>>>
> > > > >>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
> > > > >>>>>>
> > > > >>>>>> hi Jiunn-Yang
> > > > >>>>>>
> > > > >>>>>> chia_0: Should we expose the partition creation time via the
> > Admin
> > > > API?
> > > > >>>>> I assume it would be valuable for users to diagnose and
> > troubleshoot
> > > > the
> > > > >>>>> behavior of auto.offset.reset.latest.max.age
> > > > >>>>>>
> > > > >>>>>> Best,
> > > > >>>>>> Chia-Ping
> > > > >>>>>>
> > > > >>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > > >>>>>>> Hello everyone,
> > > > >>>>>>>
> > > > >>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
> > > > Loss
> > > > >>>> on
> > > > >>>>> Partition Expansion for Latest Policy
> > > > >>>>>>> <
> > > > >>>>
> > > >
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > > >>>>>
> > > > >>>>>>>
> > > > >>>>>>> This proposal aims to introduces
> > auto.offset.reset.latest.max.age,
> > > > a
> > > > >>>>> consumer config that lets the
> > > > >>>>>>> latest reset policy distinguish newly expanded (hot) partitions
> > > > from
> > > > >>>>> long-existing (cold) ones. Partitions
> > > > >>>>>>> younger than the configured threshold automatically fall back
> > to
> > > > >>>>> earliest, preventing silent data loss
> > > > >>>>>>> during topic expansion without forcing a full historical
> > reprocess.
> > > > >>>>>>>
> > > > >>>>>>> Best regards,
> > > > >>>>>>> Jiunn-Yang
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >
> > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Chia-Ping,

Thanks for the reply.

1. In the motivation section, the KIP says "When a Kafka topic is expanded
with new partitions, consumers using the latest auto offset reset policy
will silently miss all records produced to those partitions before the
consumer discovers them.". If a user sets
auto.offset.reset=by_duration=1sec, the same record loss issue could also
happen, right?

2. I was thinking auto.offset.reset.new.partitions will take the same
values as auto.offset.reset. So a user could set it by_duration if needed.

Jun

On Thu, May 14, 2026 at 4:06 PM Chia-Ping Tsai <ch...@apache.org> wrote:

> hi Jun
>
> Thanks for the feedback. I might be missing something important from your
> suggestion, so please bear with me as I try to clarify with a few questions:
>
> 1. Is there a strong use case for extending this logic to other reset
> policies? Unlike latest, policies like earliest or by_duration don't seem
> to suffer from the same silent data loss issue when a partition is expanded.
>
> 2. What values would we expect users to configure for
> auto.offset.reset.new.partitions? If they set it to earliest or latest,
> we might run into the exact same edge cases. For example, if a consumer is
> offline for a while and a new partition is created during that downtime,
> the user might actually want to skip to latest when resuming, rather than
> reading from earliest just because the partition is technically "new" to
> the group.
>
> This is exactly why we opted for introducing a max.age threshold. It gives
> users a time-bound way to define what is genuinely "hot/new" and what is
> just an old partition they haven't seen yet.
>
> Best,
> Chia-Ping
>
> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> > Hi, Jiunn-Yang,
> >
> > Thanks for the KIP.
> >
> > I find auto.offset.reset.latest.max.age a bit weird. It only applies when
> > auto.offset.reset is latest. However, it seems that the motivation
> equally
> > applies when auto.offset.reset is set to other values like by_duration.
> The
> > intention is that we want to have a separate way to control newly created
> > partitions vs existing partitions when the group starts. Have we
> considered
> > adding a new config like auto.offset.reset.new.partitions? If this new
> > config is not set, the offset reset policy defaults to the policy used
> for
> > existing partitions. The user could set it explicitly to customize the
> > behavior for new partitions.
> >
> > Jun
> >
> > On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I’d like to manually bump this thread.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > > > 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> > > >
> > > > Hello all,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > > DJ01/DJ02:
> > > >
> > > > MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
> > > gains a new
> > > > field PartitionAgeMs (int64, default -1), computed server-side by the
> > > broker as
> > > > broker_current_time - partition_creation_time.
> > > >
> > > > Also add the consumer heartbeat flow. when MembershipManager detects
> a
> > > newly assigned
> > > > partition, it explicitly invalidates the metadata for the affected
> topic
> > > and forces a fresh MetadataRequest
> > > > before making the offset reset decision, even if the topic ID is
> already
> > > in the cache.
> > > >
> > > > MB0:
> > > >
> > > > The consumer learns the broker's maximum supported MetadataResponse
> > > version via the
> > > > ApiVersions negotiation at connection time. If the negotiated
> version is
> > > unsupported, the consumer
> > > > knows the broker does not support PartitionAgeMs at all and can
> throw an
> > > UnsupportedVersionException
> > > > immediately, rather than silently falling back to latest and risking
> > > data loss without any operator-visible signal.
> > > >
> > > > MB1/MB2/MB3:
> > > >
> > > > I have addressed these changes in the KIP.
> > > >
> > > > Best Regards,
> > > > Jiunn-Yang
> > > >
> > > >> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> > > >>
> > > >> hi David
> > > >>
> > > >> I agree with the direction of moving the 'age' resolution from the
> > > Heartbeat API to the Metadata API to keep the control plane clean. The
> main
> > > trade-off, as we noted before, is introducing inter-broker clock skew.
> The
> > > Group Coordinator approach provided a single source of truth for time.
> > > >>
> > > >> However, realistically, this time skew should be negligible. Given
> that
> > > the max.age threshold will likely be configured in minutes or hours, a
> > > typical NTP skew (in milliseconds) between brokers won't impact the
> > > fallback decision.
> > > >>
> > > >> Best,
> > > >> Chia-Ping
> > > >>
> > > >>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
> > > >>>
> > > >>> Hi all,
> > > >>>
> > > >>> Thanks for the KIP!
> > > >>>
> > > >>> Sorry, I haven't really followed the previous conversation but I
> took a
> > > >>> quick look at this one.
> > > >>>
> > > >>> DJ01: I don't clearly understand the flow with the
> > > ConsumerGroupHeartbeat
> > > >>> API after reading the KIP. There is a new boolean; the KIP states
> that
> > > >>> partition ages are returned only when this boolean is set.
> Implicitly,
> > > this
> > > >>> means that when the consumer receives a new partition, it will
> issue a
> > > new
> > > >>> HB request with the boolean set to receive the ages. Is my
> > > understanding
> > > >>> correct? We should perhaps clarify the flow and also explain how it
> > > fits
> > > >>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
> > > >>> DJ02: It my understanding is correct, I wonder if
> > > >>> the ConsumerGroupHeartbeat API is the right place for this given
> that
> > > a new
> > > >>> round trip is done anyway. Alternatively, it could simply include
> the
> > > >>> metadata. Generally, we should be rather cautious about not
> overloading
> > > >>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is
> a
> > > >>> control plane API for assigning or revoking partitions. The fact
> that
> > > we
> > > >>> don't want to add it to the corresponding Streams API also suggests
> > > >>> something is not quite right. What would we do if we want to
> support
> > > >>> Streams in the future?
> > > >>>
> > > >>> Best,
> > > >>> David
> > > >>>
> > > >>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
> > > >>>> dev@kafka.apache.org> wrote:
> > > >>>>
> > > >>>> Hi Jiunn,
> > > >>>>
> > > >>>> Thank you for this great kip. Good to know about the gap.
> > > >>>>
> > > >>>> mb-0 - why a new v2 version bump for RequestPartitionAges field.
> Can a
> > > >>>> tagged field (for ex: on response, PartitionAges on
> TopicPartitions)
> > > be
> > > >>>> used here and avoid version bump?
> > > >>>>
> > > >>>> mb-1 - For the new config, is there a recommended value or a
> ConfigDef
> > > >>>> validator? Probably it should based on the metadata.max.age.ms ?
> > > Sizing
> > > >>>> instructions can be part of javadocs I guess.
> > > >>>>
> > > >>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it
> be
> > > better
> > > >>>> to add this new config auto.offset.reset.latest.max.age to the
> > > >>>> StreamsConfig block list
> (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > > for a
> > > >>>> clear warning, incase users configure it? This is the most
> familiar
> > > >>>> consumer config and users might easily mistakenly configure it. Or
> > > may be
> > > >>>> it's not worth it to add.
> > > >>>>
> > > >>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
> > > reads as
> > > >>>> if the config were being changed per-partition which isn't
> supported.
> > > May
> > > >>>> be rephrasing to something like "consumer resolves the initial
> > > position to
> > > >>>> start offset for that partition" as if earliest was applied to
> that
> > > >>>> partition only and auto.offset.reset config is unchanged.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Murali
> > > >>>>
> > > >>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
> > > >>>>>
> > > >>>>> Hi chia,
> > > >>>>>
> > > >>>>> I have updated the KIP to include this change.
> > > >>>>>
> > > >>>>> Best Regards,
> > > >>>>> Jiunn-Yang
> > > >>>>>
> > > >>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
> > > >>>>>>
> > > >>>>>> hi Jiunn-Yang
> > > >>>>>>
> > > >>>>>> chia_0: Should we expose the partition creation time via the
> Admin
> > > API?
> > > >>>>> I assume it would be valuable for users to diagnose and
> troubleshoot
> > > the
> > > >>>>> behavior of auto.offset.reset.latest.max.age
> > > >>>>>>
> > > >>>>>> Best,
> > > >>>>>> Chia-Ping
> > > >>>>>>
> > > >>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > > >>>>>>> Hello everyone,
> > > >>>>>>>
> > > >>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
> > > Loss
> > > >>>> on
> > > >>>>> Partition Expansion for Latest Policy
> > > >>>>>>> <
> > > >>>>
> > >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > > >>>>>
> > > >>>>>>>
> > > >>>>>>> This proposal aims to introduces
> auto.offset.reset.latest.max.age,
> > > a
> > > >>>>> consumer config that lets the
> > > >>>>>>> latest reset policy distinguish newly expanded (hot) partitions
> > > from
> > > >>>>> long-existing (cold) ones. Partitions
> > > >>>>>>> younger than the configured threshold automatically fall back
> to
> > > >>>>> earliest, preventing silent data loss
> > > >>>>>>> during topic expansion without forcing a full historical
> reprocess.
> > > >>>>>>>
> > > >>>>>>> Best regards,
> > > >>>>>>> Jiunn-Yang
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >
> > >
> > >
> >
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hi Ryan,

Thanks for the feedback! I agree that adding the .ms suffix is the right call. I’ll update the 
config name to auto.offset.reset.latest.max.age.ms. 

Regarding min.age vs max.age, I think both are technically valid since they describe the 
same threshold from different perspectives: 

- min.age: the minimum age a partition must reach before latest applies 
- max.age: the maximum age a partition can have while still being considered hot 

I prefer max.age because the primary goal of this KIP is protecting newly expanded partitions 
from silent data loss. The key question for users is “how new does a partition need to be to 
receive special treatment?”, and max.age answers that directly. In contrast, min.age frames 
the config around latest, which is already the default behavior and does not need additional 
emphasis.

Best Regards,
Jiunn-Yang

> Chia-Ping Tsai <ch...@apache.org> 於 2026年5月15日 清晨7:04 寫道:
> 
> hi Jun
> 
> Thanks for the feedback. I might be missing something important from your suggestion, so please bear with me as I try to clarify with a few questions:
> 
> 1. Is there a strong use case for extending this logic to other reset policies? Unlike latest, policies like earliest or by_duration don't seem to suffer from the same silent data loss issue when a partition is expanded.
> 
> 2. What values would we expect users to configure for auto.offset.reset.new.partitions? If they set it to earliest or latest, we might run into the exact same edge cases. For example, if a consumer is offline for a while and a new partition is created during that downtime, the user might actually want to skip to latest when resuming, rather than reading from earliest just because the partition is technically "new" to the group.
> 
> This is exactly why we opted for introducing a max.age threshold. It gives users a time-bound way to define what is genuinely "hot/new" and what is just an old partition they haven't seen yet.
> 
> Best,
> Chia-Ping
> 
> On 2026/05/14 20:48:09 Jun Rao via dev wrote:
>> Hi, Jiunn-Yang,
>> 
>> Thanks for the KIP.
>> 
>> I find auto.offset.reset.latest.max.age a bit weird. It only applies when
>> auto.offset.reset is latest. However, it seems that the motivation equally
>> applies when auto.offset.reset is set to other values like by_duration. The
>> intention is that we want to have a separate way to control newly created
>> partitions vs existing partitions when the group starts. Have we considered
>> adding a new config like auto.offset.reset.new.partitions? If this new
>> config is not set, the offset reset policy defaults to the policy used for
>> existing partitions. The user could set it explicitly to customize the
>> behavior for new partitions.
>> 
>> Jun
>> 
>> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
>> 
>>> Hi all,
>>> 
>>> I’d like to manually bump this thread.
>>> 
>>> Best Regards,
>>> Jiunn-Yang
>>> 
>>>> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
>>>> 
>>>> Hello all,
>>>> 
>>>> Thanks for the feedback.
>>>> 
>>>> DJ01/DJ02:
>>>> 
>>>> MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
>>> gains a new
>>>> field PartitionAgeMs (int64, default -1), computed server-side by the
>>> broker as
>>>> broker_current_time - partition_creation_time.
>>>> 
>>>> Also add the consumer heartbeat flow. when MembershipManager detects a
>>> newly assigned
>>>> partition, it explicitly invalidates the metadata for the affected topic
>>> and forces a fresh MetadataRequest
>>>> before making the offset reset decision, even if the topic ID is already
>>> in the cache.
>>>> 
>>>> MB0:
>>>> 
>>>> The consumer learns the broker's maximum supported MetadataResponse
>>> version via the
>>>> ApiVersions negotiation at connection time. If the negotiated version is
>>> unsupported, the consumer
>>>> knows the broker does not support PartitionAgeMs at all and can throw an
>>> UnsupportedVersionException
>>>> immediately, rather than silently falling back to latest and risking
>>> data loss without any operator-visible signal.
>>>> 
>>>> MB1/MB2/MB3:
>>>> 
>>>> I have addressed these changes in the KIP.
>>>> 
>>>> Best Regards,
>>>> Jiunn-Yang
>>>> 
>>>>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
>>>>> 
>>>>> hi David
>>>>> 
>>>>> I agree with the direction of moving the 'age' resolution from the
>>> Heartbeat API to the Metadata API to keep the control plane clean. The main
>>> trade-off, as we noted before, is introducing inter-broker clock skew. The
>>> Group Coordinator approach provided a single source of truth for time.
>>>>> 
>>>>> However, realistically, this time skew should be negligible. Given that
>>> the max.age threshold will likely be configured in minutes or hours, a
>>> typical NTP skew (in milliseconds) between brokers won't impact the
>>> fallback decision.
>>>>> 
>>>>> Best,
>>>>> Chia-Ping
>>>>> 
>>>>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
>>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> Thanks for the KIP!
>>>>>> 
>>>>>> Sorry, I haven't really followed the previous conversation but I took a
>>>>>> quick look at this one.
>>>>>> 
>>>>>> DJ01: I don't clearly understand the flow with the
>>> ConsumerGroupHeartbeat
>>>>>> API after reading the KIP. There is a new boolean; the KIP states that
>>>>>> partition ages are returned only when this boolean is set. Implicitly,
>>> this
>>>>>> means that when the consumer receives a new partition, it will issue a
>>> new
>>>>>> HB request with the boolean set to receive the ages. Is my
>>> understanding
>>>>>> correct? We should perhaps clarify the flow and also explain how it
>>> fits
>>>>>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
>>>>>> DJ02: It my understanding is correct, I wonder if
>>>>>> the ConsumerGroupHeartbeat API is the right place for this given that
>>> a new
>>>>>> round trip is done anyway. Alternatively, it could simply include the
>>>>>> metadata. Generally, we should be rather cautious about not overloading
>>>>>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is a
>>>>>> control plane API for assigning or revoking partitions. The fact that
>>> we
>>>>>> don't want to add it to the corresponding Streams API also suggests
>>>>>> something is not quite right. What would we do if we want to support
>>>>>> Streams in the future?
>>>>>> 
>>>>>> Best,
>>>>>> David
>>>>>> 
>>>>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
>>>>>>> dev@kafka.apache.org> wrote:
>>>>>>> 
>>>>>>> Hi Jiunn,
>>>>>>> 
>>>>>>> Thank you for this great kip. Good to know about the gap.
>>>>>>> 
>>>>>>> mb-0 - why a new v2 version bump for RequestPartitionAges field. Can a
>>>>>>> tagged field (for ex: on response, PartitionAges on TopicPartitions)
>>> be
>>>>>>> used here and avoid version bump?
>>>>>>> 
>>>>>>> mb-1 - For the new config, is there a recommended value or a ConfigDef
>>>>>>> validator? Probably it should based on the metadata.max.age.ms ?
>>> Sizing
>>>>>>> instructions can be part of javadocs I guess.
>>>>>>> 
>>>>>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it be
>>> better
>>>>>>> to add this new config auto.offset.reset.latest.max.age to the
>>>>>>> StreamsConfig block list (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
>>> for a
>>>>>>> clear warning, incase users configure it? This is the most familiar
>>>>>>> consumer config and users might easily mistakenly configure it. Or
>>> may be
>>>>>>> it's not worth it to add.
>>>>>>> 
>>>>>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
>>> reads as
>>>>>>> if the config were being changed per-partition which isn't supported.
>>> May
>>>>>>> be rephrasing to something like "consumer resolves the initial
>>> position to
>>>>>>> start offset for that partition" as if earliest was applied to that
>>>>>>> partition only and auto.offset.reset config is unchanged.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Murali
>>>>>>> 
>>>>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>>>>> 
>>>>>>>> Hi chia,
>>>>>>>> 
>>>>>>>> I have updated the KIP to include this change.
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
>>>>>>>>> 
>>>>>>>>> hi Jiunn-Yang
>>>>>>>>> 
>>>>>>>>> chia_0: Should we expose the partition creation time via the Admin
>>> API?
>>>>>>>> I assume it would be valuable for users to diagnose and troubleshoot
>>> the
>>>>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Chia-Ping
>>>>>>>>> 
>>>>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>>>>> Hello everyone,
>>>>>>>>>> 
>>>>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
>>> Loss
>>>>>>> on
>>>>>>>> Partition Expansion for Latest Policy
>>>>>>>>>> <
>>>>>>> 
>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> This proposal aims to introduces auto.offset.reset.latest.max.age,
>>> a
>>>>>>>> consumer config that lets the
>>>>>>>>>> latest reset policy distinguish newly expanded (hot) partitions
>>> from
>>>>>>>> long-existing (cold) ones. Partitions
>>>>>>>>>> younger than the configured threshold automatically fall back to
>>>>>>>> earliest, preventing silent data loss
>>>>>>>>>> during topic expansion without forcing a full historical reprocess.
>>>>>>>>>> 
>>>>>>>>>> Best regards,
>>>>>>>>>> Jiunn-Yang
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>>> 
>> 


Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Chia-Ping Tsai <ch...@apache.org>.
hi Jun

Thanks for the feedback. I might be missing something important from your suggestion, so please bear with me as I try to clarify with a few questions:

1. Is there a strong use case for extending this logic to other reset policies? Unlike latest, policies like earliest or by_duration don't seem to suffer from the same silent data loss issue when a partition is expanded.

2. What values would we expect users to configure for auto.offset.reset.new.partitions? If they set it to earliest or latest, we might run into the exact same edge cases. For example, if a consumer is offline for a while and a new partition is created during that downtime, the user might actually want to skip to latest when resuming, rather than reading from earliest just because the partition is technically "new" to the group.

This is exactly why we opted for introducing a max.age threshold. It gives users a time-bound way to define what is genuinely "hot/new" and what is just an old partition they haven't seen yet.

Best,
Chia-Ping

On 2026/05/14 20:48:09 Jun Rao via dev wrote:
> Hi, Jiunn-Yang,
> 
> Thanks for the KIP.
> 
> I find auto.offset.reset.latest.max.age a bit weird. It only applies when
> auto.offset.reset is latest. However, it seems that the motivation equally
> applies when auto.offset.reset is set to other values like by_duration. The
> intention is that we want to have a separate way to control newly created
> partitions vs existing partitions when the group starts. Have we considered
> adding a new config like auto.offset.reset.new.partitions? If this new
> config is not set, the offset reset policy defaults to the policy used for
> existing partitions. The user could set it explicitly to customize the
> behavior for new partitions.
> 
> Jun
> 
> On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:
> 
> > Hi all,
> >
> > I’d like to manually bump this thread.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> > > 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> > >
> > > Hello all,
> > >
> > > Thanks for the feedback.
> > >
> > > DJ01/DJ02:
> > >
> > > MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
> > gains a new
> > > field PartitionAgeMs (int64, default -1), computed server-side by the
> > broker as
> > > broker_current_time - partition_creation_time.
> > >
> > > Also add the consumer heartbeat flow. when MembershipManager detects a
> > newly assigned
> > > partition, it explicitly invalidates the metadata for the affected topic
> > and forces a fresh MetadataRequest
> > > before making the offset reset decision, even if the topic ID is already
> > in the cache.
> > >
> > > MB0:
> > >
> > > The consumer learns the broker's maximum supported MetadataResponse
> > version via the
> > > ApiVersions negotiation at connection time. If the negotiated version is
> > unsupported, the consumer
> > > knows the broker does not support PartitionAgeMs at all and can throw an
> > UnsupportedVersionException
> > > immediately, rather than silently falling back to latest and risking
> > data loss without any operator-visible signal.
> > >
> > > MB1/MB2/MB3:
> > >
> > > I have addressed these changes in the KIP.
> > >
> > > Best Regards,
> > > Jiunn-Yang
> > >
> > >> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> > >>
> > >> hi David
> > >>
> > >> I agree with the direction of moving the 'age' resolution from the
> > Heartbeat API to the Metadata API to keep the control plane clean. The main
> > trade-off, as we noted before, is introducing inter-broker clock skew. The
> > Group Coordinator approach provided a single source of truth for time.
> > >>
> > >> However, realistically, this time skew should be negligible. Given that
> > the max.age threshold will likely be configured in minutes or hours, a
> > typical NTP skew (in milliseconds) between brokers won't impact the
> > fallback decision.
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
> > >>>
> > >>> Hi all,
> > >>>
> > >>> Thanks for the KIP!
> > >>>
> > >>> Sorry, I haven't really followed the previous conversation but I took a
> > >>> quick look at this one.
> > >>>
> > >>> DJ01: I don't clearly understand the flow with the
> > ConsumerGroupHeartbeat
> > >>> API after reading the KIP. There is a new boolean; the KIP states that
> > >>> partition ages are returned only when this boolean is set. Implicitly,
> > this
> > >>> means that when the consumer receives a new partition, it will issue a
> > new
> > >>> HB request with the boolean set to receive the ages. Is my
> > understanding
> > >>> correct? We should perhaps clarify the flow and also explain how it
> > fits
> > >>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
> > >>> DJ02: It my understanding is correct, I wonder if
> > >>> the ConsumerGroupHeartbeat API is the right place for this given that
> > a new
> > >>> round trip is done anyway. Alternatively, it could simply include the
> > >>> metadata. Generally, we should be rather cautious about not overloading
> > >>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is a
> > >>> control plane API for assigning or revoking partitions. The fact that
> > we
> > >>> don't want to add it to the corresponding Streams API also suggests
> > >>> something is not quite right. What would we do if we want to support
> > >>> Streams in the future?
> > >>>
> > >>> Best,
> > >>> David
> > >>>
> > >>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
> > >>>> dev@kafka.apache.org> wrote:
> > >>>>
> > >>>> Hi Jiunn,
> > >>>>
> > >>>> Thank you for this great kip. Good to know about the gap.
> > >>>>
> > >>>> mb-0 - why a new v2 version bump for RequestPartitionAges field. Can a
> > >>>> tagged field (for ex: on response, PartitionAges on TopicPartitions)
> > be
> > >>>> used here and avoid version bump?
> > >>>>
> > >>>> mb-1 - For the new config, is there a recommended value or a ConfigDef
> > >>>> validator? Probably it should based on the metadata.max.age.ms ?
> > Sizing
> > >>>> instructions can be part of javadocs I guess.
> > >>>>
> > >>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it be
> > better
> > >>>> to add this new config auto.offset.reset.latest.max.age to the
> > >>>> StreamsConfig block list (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> > for a
> > >>>> clear warning, incase users configure it? This is the most familiar
> > >>>> consumer config and users might easily mistakenly configure it. Or
> > may be
> > >>>> it's not worth it to add.
> > >>>>
> > >>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
> > reads as
> > >>>> if the config were being changed per-partition which isn't supported.
> > May
> > >>>> be rephrasing to something like "consumer resolves the initial
> > position to
> > >>>> start offset for that partition" as if earliest was applied to that
> > >>>> partition only and auto.offset.reset config is unchanged.
> > >>>>
> > >>>> Thanks,
> > >>>> Murali
> > >>>>
> > >>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
> > >>>>>
> > >>>>> Hi chia,
> > >>>>>
> > >>>>> I have updated the KIP to include this change.
> > >>>>>
> > >>>>> Best Regards,
> > >>>>> Jiunn-Yang
> > >>>>>
> > >>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
> > >>>>>>
> > >>>>>> hi Jiunn-Yang
> > >>>>>>
> > >>>>>> chia_0: Should we expose the partition creation time via the Admin
> > API?
> > >>>>> I assume it would be valuable for users to diagnose and troubleshoot
> > the
> > >>>>> behavior of auto.offset.reset.latest.max.age
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Chia-Ping
> > >>>>>>
> > >>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> > >>>>>>> Hello everyone,
> > >>>>>>>
> > >>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
> > Loss
> > >>>> on
> > >>>>> Partition Expansion for Latest Policy
> > >>>>>>> <
> > >>>>
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> > >>>>>
> > >>>>>>>
> > >>>>>>> This proposal aims to introduces auto.offset.reset.latest.max.age,
> > a
> > >>>>> consumer config that lets the
> > >>>>>>> latest reset policy distinguish newly expanded (hot) partitions
> > from
> > >>>>> long-existing (cold) ones. Partitions
> > >>>>>>> younger than the configured threshold automatically fall back to
> > >>>>> earliest, preventing silent data loss
> > >>>>>>> during topic expansion without forcing a full historical reprocess.
> > >>>>>>>
> > >>>>>>> Best regards,
> > >>>>>>> Jiunn-Yang
> > >>>>>
> > >>>>>
> > >>>>
> > >
> >
> >
> 

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by Jun Rao via dev <de...@kafka.apache.org>.
Hi, Jiunn-Yang,

Thanks for the KIP.

I find auto.offset.reset.latest.max.age a bit weird. It only applies when
auto.offset.reset is latest. However, it seems that the motivation equally
applies when auto.offset.reset is set to other values like by_duration. The
intention is that we want to have a separate way to control newly created
partitions vs existing partitions when the group starts. Have we considered
adding a new config like auto.offset.reset.new.partitions? If this new
config is not set, the offset reset policy defaults to the policy used for
existing partitions. The user could set it explicitly to customize the
behavior for new partitions.

Jun

On Thu, May 7, 2026 at 5:07 AM 黃竣陽 <s7...@gmail.com> wrote:

> Hi all,
>
> I’d like to manually bump this thread.
>
> Best Regards,
> Jiunn-Yang
>
> > 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> >
> > Hello all,
> >
> > Thanks for the feedback.
> >
> > DJ01/DJ02:
> >
> > MetadataResponse bumps from v13 to v14. The PartitionMetadata struct
> gains a new
> > field PartitionAgeMs (int64, default -1), computed server-side by the
> broker as
> > broker_current_time - partition_creation_time.
> >
> > Also add the consumer heartbeat flow. when MembershipManager detects a
> newly assigned
> > partition, it explicitly invalidates the metadata for the affected topic
> and forces a fresh MetadataRequest
> > before making the offset reset decision, even if the topic ID is already
> in the cache.
> >
> > MB0:
> >
> > The consumer learns the broker's maximum supported MetadataResponse
> version via the
> > ApiVersions negotiation at connection time. If the negotiated version is
> unsupported, the consumer
> > knows the broker does not support PartitionAgeMs at all and can throw an
> UnsupportedVersionException
> > immediately, rather than silently falling back to latest and risking
> data loss without any operator-visible signal.
> >
> > MB1/MB2/MB3:
> >
> > I have addressed these changes in the KIP.
> >
> > Best Regards,
> > Jiunn-Yang
> >
> >> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
> >>
> >> hi David
> >>
> >> I agree with the direction of moving the 'age' resolution from the
> Heartbeat API to the Metadata API to keep the control plane clean. The main
> trade-off, as we noted before, is introducing inter-broker clock skew. The
> Group Coordinator approach provided a single source of truth for time.
> >>
> >> However, realistically, this time skew should be negligible. Given that
> the max.age threshold will likely be configured in minutes or hours, a
> typical NTP skew (in milliseconds) between brokers won't impact the
> fallback decision.
> >>
> >> Best,
> >> Chia-Ping
> >>
> >>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
> >>>
> >>> Hi all,
> >>>
> >>> Thanks for the KIP!
> >>>
> >>> Sorry, I haven't really followed the previous conversation but I took a
> >>> quick look at this one.
> >>>
> >>> DJ01: I don't clearly understand the flow with the
> ConsumerGroupHeartbeat
> >>> API after reading the KIP. There is a new boolean; the KIP states that
> >>> partition ages are returned only when this boolean is set. Implicitly,
> this
> >>> means that when the consumer receives a new partition, it will issue a
> new
> >>> HB request with the boolean set to receive the ages. Is my
> understanding
> >>> correct? We should perhaps clarify the flow and also explain how it
> fits
> >>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
> >>> DJ02: It my understanding is correct, I wonder if
> >>> the ConsumerGroupHeartbeat API is the right place for this given that
> a new
> >>> round trip is done anyway. Alternatively, it could simply include the
> >>> metadata. Generally, we should be rather cautious about not overloading
> >>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is a
> >>> control plane API for assigning or revoking partitions. The fact that
> we
> >>> don't want to add it to the corresponding Streams API also suggests
> >>> something is not quite right. What would we do if we want to support
> >>> Streams in the future?
> >>>
> >>> Best,
> >>> David
> >>>
> >>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
> >>>> dev@kafka.apache.org> wrote:
> >>>>
> >>>> Hi Jiunn,
> >>>>
> >>>> Thank you for this great kip. Good to know about the gap.
> >>>>
> >>>> mb-0 - why a new v2 version bump for RequestPartitionAges field. Can a
> >>>> tagged field (for ex: on response, PartitionAges on TopicPartitions)
> be
> >>>> used here and avoid version bump?
> >>>>
> >>>> mb-1 - For the new config, is there a recommended value or a ConfigDef
> >>>> validator? Probably it should based on the metadata.max.age.ms ?
> Sizing
> >>>> instructions can be part of javadocs I guess.
> >>>>
> >>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it be
> better
> >>>> to add this new config auto.offset.reset.latest.max.age to the
> >>>> StreamsConfig block list (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS)
> for a
> >>>> clear warning, incase users configure it? This is the most familiar
> >>>> consumer config and users might easily mistakenly configure it. Or
> may be
> >>>> it's not worth it to add.
> >>>>
> >>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest"
> reads as
> >>>> if the config were being changed per-partition which isn't supported.
> May
> >>>> be rephrasing to something like "consumer resolves the initial
> position to
> >>>> start offset for that partition" as if earliest was applied to that
> >>>> partition only and auto.offset.reset config is unchanged.
> >>>>
> >>>> Thanks,
> >>>> Murali
> >>>>
> >>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
> >>>>>
> >>>>> Hi chia,
> >>>>>
> >>>>> I have updated the KIP to include this change.
> >>>>>
> >>>>> Best Regards,
> >>>>> Jiunn-Yang
> >>>>>
> >>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
> >>>>>>
> >>>>>> hi Jiunn-Yang
> >>>>>>
> >>>>>> chia_0: Should we expose the partition creation time via the Admin
> API?
> >>>>> I assume it would be valuable for users to diagnose and troubleshoot
> the
> >>>>> behavior of auto.offset.reset.latest.max.age
> >>>>>>
> >>>>>> Best,
> >>>>>> Chia-Ping
> >>>>>>
> >>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
> >>>>>>> Hello everyone,
> >>>>>>>
> >>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data
> Loss
> >>>> on
> >>>>> Partition Expansion for Latest Policy
> >>>>>>> <
> >>>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
> >>>>>
> >>>>>>>
> >>>>>>> This proposal aims to introduces auto.offset.reset.latest.max.age,
> a
> >>>>> consumer config that lets the
> >>>>>>> latest reset policy distinguish newly expanded (hot) partitions
> from
> >>>>> long-existing (cold) ones. Partitions
> >>>>>>> younger than the configured threshold automatically fall back to
> >>>>> earliest, preventing silent data loss
> >>>>>>> during topic expansion without forcing a full historical reprocess.
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>> Jiunn-Yang
> >>>>>
> >>>>>
> >>>>
> >
>
>

Re: [DISCUSS] KIP-1327 Prevent Hot Data Loss on Partition Expansion for Latest Policy

Posted by 黃竣陽 <s7...@gmail.com>.
Hi all,

I’d like to manually bump this thread.

Best Regards,
Jiunn-Yang

> 黃竣陽 <s7...@gmail.com> 於 2026年5月1日 晚上10:37 寫道:
> 
> Hello all,
> 
> Thanks for the feedback.
> 
> DJ01/DJ02:
> 
> MetadataResponse bumps from v13 to v14. The PartitionMetadata struct gains a new 
> field PartitionAgeMs (int64, default -1), computed server-side by the broker as 
> broker_current_time - partition_creation_time.
> 
> Also add the consumer heartbeat flow. when MembershipManager detects a newly assigned 
> partition, it explicitly invalidates the metadata for the affected topic and forces a fresh MetadataRequest
> before making the offset reset decision, even if the topic ID is already in the cache.
> 
> MB0:
> 
> The consumer learns the broker's maximum supported MetadataResponse version via the
> ApiVersions negotiation at connection time. If the negotiated version is unsupported, the consumer 
> knows the broker does not support PartitionAgeMs at all and can throw an UnsupportedVersionException 
> immediately, rather than silently falling back to latest and risking data loss without any operator-visible signal.
> 
> MB1/MB2/MB3:
> 
> I have addressed these changes in the KIP.
> 
> Best Regards,
> Jiunn-Yang
> 
>> Chia-Ping Tsai <ch...@gmail.com> 於 2026年4月29日 下午4:04 寫道:
>> 
>> hi David
>> 
>> I agree with the direction of moving the 'age' resolution from the Heartbeat API to the Metadata API to keep the control plane clean. The main trade-off, as we noted before, is introducing inter-broker clock skew. The Group Coordinator approach provided a single source of truth for time.
>> 
>> However, realistically, this time skew should be negligible. Given that the max.age threshold will likely be configured in minutes or hours, a typical NTP skew (in milliseconds) between brokers won't impact the fallback decision.
>> 
>> Best,
>> Chia-Ping
>> 
>>> David Jacot via dev <de...@kafka.apache.org> 於 2026年4月29日 下午3:29 寫道:
>>> 
>>> Hi all,
>>> 
>>> Thanks for the KIP!
>>> 
>>> Sorry, I haven't really followed the previous conversation but I took a
>>> quick look at this one.
>>> 
>>> DJ01: I don't clearly understand the flow with the ConsumerGroupHeartbeat
>>> API after reading the KIP. There is a new boolean; the KIP states that
>>> partition ages are returned only when this boolean is set. Implicitly, this
>>> means that when the consumer receives a new partition, it will issue a new
>>> HB request with the boolean set to receive the ages. Is my understanding
>>> correct? We should perhaps clarify the flow and also explain how it fits
>>> into the existing flow (e.g. list offsets, fetch offsets, etc.).
>>> DJ02: It my understanding is correct, I wonder if
>>> the ConsumerGroupHeartbeat API is the right place for this given that a new
>>> round trip is done anyway. Alternatively, it could simply include the
>>> metadata. Generally, we should be rather cautious about not overloading
>>> the ConsumerGroupHeartbeat API with unrelated concepts. The API is a
>>> control plane API for assigning or revoking partitions. The fact that we
>>> don't want to add it to the corresponding Streams API also suggests
>>> something is not quite right. What would we do if we want to support
>>> Streams in the future?
>>> 
>>> Best,
>>> David
>>> 
>>>> On Wed, Apr 29, 2026 at 12:28 AM Muralidhar Basani via dev <
>>>> dev@kafka.apache.org> wrote:
>>>> 
>>>> Hi Jiunn,
>>>> 
>>>> Thank you for this great kip. Good to know about the gap.
>>>> 
>>>> mb-0 - why a new v2 version bump for RequestPartitionAges field. Can a
>>>> tagged field (for ex: on response, PartitionAges on TopicPartitions) be
>>>> used here and avoid version bump?
>>>> 
>>>> mb-1 - For the new config, is there a recommended value or a ConfigDef
>>>> validator? Probably it should based on the metadata.max.age.ms ? Sizing
>>>> instructions can be part of javadocs I guess.
>>>> 
>>>> mb-2 - (minor) As there are no changes to Kafka Streams, would it be better
>>>> to add this new config auto.offset.reset.latest.max.age to the
>>>> StreamsConfig block list (NON_CONFIGURABLE_CONSUMER_DEFAULT_CONFIGS) for a
>>>> clear warning, incase users configure it? This is the most familiar
>>>> consumer config and users might easily mistakenly configure it. Or may be
>>>> it's not worth it to add.
>>>> 
>>>> mb-3 - (minor) The phrasing "the consumer falls back to earliest" reads as
>>>> if the config were being changed per-partition which isn't supported. May
>>>> be rephrasing to something like "consumer resolves the initial position to
>>>> start offset for that partition" as if earliest was applied to that
>>>> partition only and auto.offset.reset config is unchanged.
>>>> 
>>>> Thanks,
>>>> Murali
>>>> 
>>>>> On Tue, Apr 28, 2026 at 2:48 PM 黃竣陽 <s7...@gmail.com> wrote:
>>>>> 
>>>>> Hi chia,
>>>>> 
>>>>> I have updated the KIP to include this change.
>>>>> 
>>>>> Best Regards,
>>>>> Jiunn-Yang
>>>>> 
>>>>>> Chia-Ping Tsai <ch...@apache.org> 於 2026年4月28日 晚上8:03 寫道:
>>>>>> 
>>>>>> hi Jiunn-Yang
>>>>>> 
>>>>>> chia_0: Should we expose the partition creation time via the Admin API?
>>>>> I assume it would be valuable for users to diagnose and troubleshoot the
>>>>> behavior of auto.offset.reset.latest.max.age
>>>>>> 
>>>>>> Best,
>>>>>> Chia-Ping
>>>>>> 
>>>>>> On 2026/04/28 10:47:58 黃竣陽 wrote:
>>>>>>> Hello everyone,
>>>>>>> 
>>>>>>> I would like to start a discussion on KIP-1327 Prevent Hot Data Loss
>>>> on
>>>>> Partition Expansion for Latest Policy
>>>>>>> <
>>>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/KY4mGQ__;!!Ayb5sqE7!qF4q1QzF1RRgP61D7A2xuEai1ky7fepKDKFFvpNBuePikH-ULmT87TvuuZzy5kau5E4y5zMZAmfQQiwZomM$
>>>>> 
>>>>>>> 
>>>>>>> This proposal aims to introduces auto.offset.reset.latest.max.age, a
>>>>> consumer config that lets the
>>>>>>> latest reset policy distinguish newly expanded (hot) partitions from
>>>>> long-existing (cold) ones. Partitions
>>>>>>> younger than the configured threshold automatically fall back to
>>>>> earliest, preventing silent data loss
>>>>>>> during topic expansion without forcing a full historical reprocess.
>>>>>>> 
>>>>>>> Best regards,
>>>>>>> Jiunn-Yang
>>>>> 
>>>>> 
>>>> 
>