[jira] [Comment Edited] (KAFKA-20518) Allow configuring a default StreamPartitioner

["Muralidhar Basani (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-20518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18076260#comment-18076260 ] 

Muralidhar Basani edited comment on KAFKA-20518 at 4/25/26 7:56 PM:
--------------------------------------------------------------------

[~mjsax]  to me it looks like there are some limitations using partitioner.class.

Partitioner class gets serialized keyBytes, but StreamPartitioner needs to partition on objects (typed key obj) for ex. For this, biggest challenge is, it has to be deserialized to get the field of a composite key if users want to. Lot of re-processing. 
And Partitioner returns int (partitionid) and StreamPartitioner returns OPtional<Set<Integer>> (multiple partitionids). This means it can send this one record to multiple partitions at once. We loose this flexibility.

Currently DefaultStreamPartitioner is the default if users don't configure any StreamsPartitioner and default.stream.partitioner.class config will allow user override the hardcoded DefaultStreamPartitioner

Also, partitioner.class applies to every topic the embedded Producer writes to, including internal changelog and repartition topics. (May be this is not preferred always ?)

If the default value for default.stream.partitioner.class config is null, there wouldn't be any breaking changes I guess.


was (Author: muralibasani):
[~mjsax]  to me it looks like there are some limitations using partitioner.class.

Partitioner class gets serialized keyBytes, but StreamPartitioner needs to partition on objects (typed key obj) for ex. For this, biggest challenge is, it has to be deserialized to get the field of a composite key if users want to. Lot of re-processing. 
And Partitioner returns int (partitionid) and StreamPartitioner returns OPtional<Set<Integer>> (multiple partitionids). This means it can send this one record to multiple partitions at once. We loose this flexibility.

Currently DefaultStreamPartitioner is the default if users don't configure any StreamsPartitioner and default.stream.partitioner.class config will allow user override the hardcoded DefaultStreamPartitioner

Also, partitioner.class applies to every topic the embedded Producer writes to, including internal changelog and repartition topics.

If the default value for default.stream.partitioner.class config is null, there wouldn't be any breaking changes I guess.

> Allow configuring a default StreamPartitioner
> ---------------------------------------------
>
>                 Key: KAFKA-20518
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20518
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Stef Noten
>            Priority: Minor
>
> Similar to configuring the default {{{}Serde{}}}s, it would be useful to be able to configure a default {{{}StreamPartitioner{}}}.
> Often, you want to treat certain keys always in the same manner. Currently such a decision translates into configuration of the partitioner on every possible operation, e.g. with foreign-key joins or groupings. Moreover, many automatic repartitions are subtly triggered downstream by making a small change upstream (e.g. changing {{processValues}} to a {{{}process{}}}). It's easy to make mistakes against this, especially as it's also not possible to verify partitioning with {{TopologyTestDriver}} nor with validations on the {{{}TopologyDescription{}}}.
> An example: in our application we have many composite record keys, which should always be partitioned by the first part. With a default partitioner, it would be possible to do automatic correct partitioning by e.g. key class type, annotation, implementing an interface, ...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)