You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Matthias J. Sax" <mj...@apache.org> on 2026/06/03 21:55:04 UTC
Re: [DISCUSS] KIP-1321: Headers aware StreamPartitioner
Thanks for the KIP Uladzislau, and sorry for late reply... Mailing list
is way too busy.
Overall the KIP make sense to me. I am just wondering if we should
deprecate the exiting `partitions(...)` methods, allowing us to remove
it with AK 5.0 (including to change the new `partitions(... Headers)`
one to _not_ have a default implementation?
-Matthias
On 5/3/26 3:28 AM, Uladzislau Blok wrote:
> Fixed link:
> KIP-1321: https://cwiki.apache.org/confluence/x/pYwmGQ
>
>> On Sun, May 3, 2026 at 12:23 PM Uladzislau Blok <bl...@gmail.com> wrote:
>>
>> Hello All,
>>
>> I would like to start a discussion on KIP-1321: Headers Aware StreamPartitioner. This need was identified while implementing headers support for state stores (KIP-1271).
>>
>> The proposal involves adding a default method to the public interface and propagating headers to the underlying serde. This change will enable users to build custom partitioners based on header information.
>>
>> KIP-1321: https://cwiki.apache.org/confluence/x/phttps://cwiki.apache.org/confluence/x/pYwmGQYwmGQ
>>
>> Best regards,
>> Uladzislau Blok
>>
>>
>> On Sun, May 3, 2026 at 12:19 PM Uladzislau Blok <bl...@gmail.com> wrote:
>>>
>>> Hello All,
>>>
>>> I'd like to start discussion on headers aware streams partitioner, which we found out while implementing headers support for state stores
>>>
>>> KIP:
>>> https://cwiki.apache.org/confluence/x/pYwmGQ
Re: [DISCUSS] KIP-1321: Headers aware StreamPartitioner
Posted by Uladzislau Blok <bl...@gmail.com>.
Hello Alieh,
Thanks for your comments.
AS1: Yes, that makes sense. KIP was updated. It's just a bit more
documentation so I don't think a new voting thread is needed (but I
will mention this in follow-up, though).
AS2: We allow headers to be empty, but we don't allow them to be null.
I will mention that explicitly in the documentation. Regarding
mutation, IMO I don't see a case where a user would even try to do it,
but I can mention this case as well.
KIP was updated, and the update was mentioned in the voting thread.
Let me know if you have more questions
On Tue, Jun 9, 2026 at 10:45 PM Alieh Saeedi via dev
<de...@kafka.apache.org> wrote:
>
> Thanks Uladzislau for the KIP.
>
> Good that the KIP is already accepted and sorry for being so late. I just
> have a few minor suggestions that might improve the KIP, but of course it’s
> up to you whether you want to update it.
>
> AS1: Could we add a note to the Compatibility/Migration section covering
> lambda partitioners: (a) they will bind to the deprecated method and emit
> deprecation warnings, (b) they cannot access headers without being
> rewritten as a class, and (c) they will require a signature change at 5.0
> when the old method is removed.
>
> AS2: Can headers ever be null? Also Headers is mutable, but in in
> StreamPartitioner they must be immutable? Just document it? (headers is
> read-only within partitions(...) and mutating it has undefined behavior).
> This ensures nobody relies on partition-time header edits leaking into the
> produced record.
> Thanks,
> Alieh
>
>
>
>
>
>
>
>
> On Fri, Jun 5, 2026 at 2:14 PM Uladzislau Blok <bl...@gmail.com> wrote:
>
> > Hello Matthias,
> >
> > I think it makes sense to deprecate the non-header version. If users
> > have their own implementation of a partitioner, they will have enough
> > time to update it. KIP was updated.
> >
> > Since there are no major questions for KIP, I will start the voting
> > thread. Thanks for the review.
> >
> > Regards,
> > Uladzislau Blok.
> >
> > On Wed, Jun 3, 2026 at 11:57 PM Matthias J. Sax <mj...@apache.org> wrote:
> > >
> > > Thanks for the KIP Uladzislau, and sorry for late reply... Mailing list
> > > is way too busy.
> > >
> > >
> > > Overall the KIP make sense to me. I am just wondering if we should
> > > deprecate the exiting `partitions(...)` methods, allowing us to remove
> > > it with AK 5.0 (including to change the new `partitions(... Headers)`
> > > one to _not_ have a default implementation?
> > >
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 5/3/26 3:28 AM, Uladzislau Blok wrote:
> > > > Fixed link:
> > > > KIP-1321:
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/pYwmGQ__;!!Ayb5sqE7!r4jw6IsZsgsr1vql8HqfzT6lAExaOPrse43LTGD0f26UzLLJAKX3DzKwVwdC8C_ITynzeP7c6HPUg3kCDA$
> > > >
> > > >> On Sun, May 3, 2026 at 12:23 PM Uladzislau Blok <bl...@gmail.com>
> > wrote:
> > > >>
> > > >> Hello All,
> > > >>
> > > >> I would like to start a discussion on KIP-1321: Headers Aware
> > StreamPartitioner. This need was identified while implementing headers
> > support for state stores (KIP-1271).
> > > >>
> > > >> The proposal involves adding a default method to the public interface
> > and propagating headers to the underlying serde. This change will enable
> > users to build custom partitioners based on header information.
> > > >>
> > > >> KIP-1321:
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/phttps:/*cwiki.apache.org/confluence/x/pYwmGQYwmGQ__;Lw!!Ayb5sqE7!r4jw6IsZsgsr1vql8HqfzT6lAExaOPrse43LTGD0f26UzLLJAKX3DzKwVwdC8C_ITynzeP7c6HPPhwWP5Q$
> > > >>
> > > >> Best regards,
> > > >> Uladzislau Blok
> > > >>
> > > >>
> > > >> On Sun, May 3, 2026 at 12:19 PM Uladzislau Blok <bl...@gmail.com>
> > wrote:
> > > >>>
> > > >>> Hello All,
> > > >>>
> > > >>> I'd like to start discussion on headers aware streams partitioner,
> > which we found out while implementing headers support for state stores
> > > >>>
> > > >>> KIP:
> > > >>>
> > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/pYwmGQ__;!!Ayb5sqE7!r4jw6IsZsgsr1vql8HqfzT6lAExaOPrse43LTGD0f26UzLLJAKX3DzKwVwdC8C_ITynzeP7c6HPUg3kCDA$
> > >
> >
Re: [DISCUSS] KIP-1321: Headers aware StreamPartitioner
Posted by Alieh Saeedi via dev <de...@kafka.apache.org>.
Thanks Uladzislau for the KIP.
Good that the KIP is already accepted and sorry for being so late. I just
have a few minor suggestions that might improve the KIP, but of course it’s
up to you whether you want to update it.
AS1: Could we add a note to the Compatibility/Migration section covering
lambda partitioners: (a) they will bind to the deprecated method and emit
deprecation warnings, (b) they cannot access headers without being
rewritten as a class, and (c) they will require a signature change at 5.0
when the old method is removed.
AS2: Can headers ever be null? Also Headers is mutable, but in in
StreamPartitioner they must be immutable? Just document it? (headers is
read-only within partitions(...) and mutating it has undefined behavior).
This ensures nobody relies on partition-time header edits leaking into the
produced record.
Thanks,
Alieh
On Fri, Jun 5, 2026 at 2:14 PM Uladzislau Blok <bl...@gmail.com> wrote:
> Hello Matthias,
>
> I think it makes sense to deprecate the non-header version. If users
> have their own implementation of a partitioner, they will have enough
> time to update it. KIP was updated.
>
> Since there are no major questions for KIP, I will start the voting
> thread. Thanks for the review.
>
> Regards,
> Uladzislau Blok.
>
> On Wed, Jun 3, 2026 at 11:57 PM Matthias J. Sax <mj...@apache.org> wrote:
> >
> > Thanks for the KIP Uladzislau, and sorry for late reply... Mailing list
> > is way too busy.
> >
> >
> > Overall the KIP make sense to me. I am just wondering if we should
> > deprecate the exiting `partitions(...)` methods, allowing us to remove
> > it with AK 5.0 (including to change the new `partitions(... Headers)`
> > one to _not_ have a default implementation?
> >
> >
> >
> > -Matthias
> >
> >
> > On 5/3/26 3:28 AM, Uladzislau Blok wrote:
> > > Fixed link:
> > > KIP-1321:
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/pYwmGQ__;!!Ayb5sqE7!r4jw6IsZsgsr1vql8HqfzT6lAExaOPrse43LTGD0f26UzLLJAKX3DzKwVwdC8C_ITynzeP7c6HPUg3kCDA$
> > >
> > >> On Sun, May 3, 2026 at 12:23 PM Uladzislau Blok <bl...@gmail.com>
> wrote:
> > >>
> > >> Hello All,
> > >>
> > >> I would like to start a discussion on KIP-1321: Headers Aware
> StreamPartitioner. This need was identified while implementing headers
> support for state stores (KIP-1271).
> > >>
> > >> The proposal involves adding a default method to the public interface
> and propagating headers to the underlying serde. This change will enable
> users to build custom partitioners based on header information.
> > >>
> > >> KIP-1321:
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/phttps:/*cwiki.apache.org/confluence/x/pYwmGQYwmGQ__;Lw!!Ayb5sqE7!r4jw6IsZsgsr1vql8HqfzT6lAExaOPrse43LTGD0f26UzLLJAKX3DzKwVwdC8C_ITynzeP7c6HPPhwWP5Q$
> > >>
> > >> Best regards,
> > >> Uladzislau Blok
> > >>
> > >>
> > >> On Sun, May 3, 2026 at 12:19 PM Uladzislau Blok <bl...@gmail.com>
> wrote:
> > >>>
> > >>> Hello All,
> > >>>
> > >>> I'd like to start discussion on headers aware streams partitioner,
> which we found out while implementing headers support for state stores
> > >>>
> > >>> KIP:
> > >>>
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/x/pYwmGQ__;!!Ayb5sqE7!r4jw6IsZsgsr1vql8HqfzT6lAExaOPrse43LTGD0f26UzLLJAKX3DzKwVwdC8C_ITynzeP7c6HPUg3kCDA$
> >
>
Re: [DISCUSS] KIP-1321: Headers aware StreamPartitioner
Posted by Uladzislau Blok <bl...@gmail.com>.
Hello Matthias,
I think it makes sense to deprecate the non-header version. If users
have their own implementation of a partitioner, they will have enough
time to update it. KIP was updated.
Since there are no major questions for KIP, I will start the voting
thread. Thanks for the review.
Regards,
Uladzislau Blok.
On Wed, Jun 3, 2026 at 11:57 PM Matthias J. Sax <mj...@apache.org> wrote:
>
> Thanks for the KIP Uladzislau, and sorry for late reply... Mailing list
> is way too busy.
>
>
> Overall the KIP make sense to me. I am just wondering if we should
> deprecate the exiting `partitions(...)` methods, allowing us to remove
> it with AK 5.0 (including to change the new `partitions(... Headers)`
> one to _not_ have a default implementation?
>
>
>
> -Matthias
>
>
> On 5/3/26 3:28 AM, Uladzislau Blok wrote:
> > Fixed link:
> > KIP-1321: https://cwiki.apache.org/confluence/x/pYwmGQ
> >
> >> On Sun, May 3, 2026 at 12:23 PM Uladzislau Blok <bl...@gmail.com> wrote:
> >>
> >> Hello All,
> >>
> >> I would like to start a discussion on KIP-1321: Headers Aware StreamPartitioner. This need was identified while implementing headers support for state stores (KIP-1271).
> >>
> >> The proposal involves adding a default method to the public interface and propagating headers to the underlying serde. This change will enable users to build custom partitioners based on header information.
> >>
> >> KIP-1321: https://cwiki.apache.org/confluence/x/phttps://cwiki.apache.org/confluence/x/pYwmGQYwmGQ
> >>
> >> Best regards,
> >> Uladzislau Blok
> >>
> >>
> >> On Sun, May 3, 2026 at 12:19 PM Uladzislau Blok <bl...@gmail.com> wrote:
> >>>
> >>> Hello All,
> >>>
> >>> I'd like to start discussion on headers aware streams partitioner, which we found out while implementing headers support for state stores
> >>>
> >>> KIP:
> >>> https://cwiki.apache.org/confluence/x/pYwmGQ
>