KIP-794 inhibits Partitioner delegation with no fallback option

[James Olsen <ja...@inaseq.com>]

Prior to KIP-794 it was possible to create a custom Partitioner that could delegate to the DefaultPartitioner.  This has been deprecated so we can now only delegate to BuiltInPartitioner.partitionForKey which does not handle a non-keyed message.  Hence there is now no mechanism for a custom Partitioner to fallback to default partitioning, e.g. for the non-keyed sticky case.

Why was Partitioner delegation useful?
1. A single Producer can be used to write to multiple Topics and each Topic may have different partitioning requirements.  The Producer can only have a single Partitioner so the Partitioner needs to be able to switch behaviour based on the Topic, including the need to fallback to default behaviour if a given Topic does not have a custom requirement.
2. Multiple services may need to produce to the same Topic and these services may be authored by different teams.  A single custom Partitioner that encapsulates the Topic specific partitioning logic can be used by all teams at all times for all Topics ensuring that mistakes are not made.

It looks like there would be an easy solution which does not rely on Partitioner delegation.  This would be to allow the Partitioner to return -1 (RecordMetadata.UNKNOWN_PARTITION) and have the Producer fallback to default behaviour.  This option was rejected as the solution to KIP-794 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner#KIP794:StrictlyUniformStickyPartitioner-Partitioner.partitiontoreturn-1<https://cwiki.apache.org/confluence/display/KAFKA/KIP-794:+Strictly+Uniform+Sticky+Partitioner#KIP794:StrictlyUniformStickyPartitioner-Partitioner.partitiontoreturn-1>) and dismissed as not useful to Partitioners in general.  So although not necessary for KIP-794 which could remain implemented as it currently is, I think this fallback scenario is something that should be supported - it is useful to custom Partitioners for the reasons outlined above.

So I would like to propose that KafkaProducer.partition(...) not throw IllegalArgumentException if the Partitioner returns RecordMetadata.UNKNOWN_PARTITION and instead continue with the default behaviour.  Maybe with a configuration flag to enable this behaviour so as not to break existing expectations?

Fwd: KIP-794 inhibits Partitioner delegation with no fallback option

[James Olsen <ja...@inaseq.com>]

Begin forwarded message:

From: James Olsen <ja...@inaseq.com>>
Subject: KIP-794 inhibits Partitioner delegation with no fallback option
Date: 16 December 2022 at 08:28:07 NZDT
To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>
Reply-To: users@kafka.apache.org<ma...@kafka.apache.org>

Prior to KIP-794 it was possible to create a custom Partitioner that could delegate to the DefaultPartitioner.  This has been deprecated so we can now only delegate to BuiltInPartitioner.partitionForKey which does not handle a non-keyed message.  Hence there is now no mechanism for a custom Partitioner to fallback to default partitioning, e.g. for the non-keyed sticky case.

Why was Partitioner delegation useful?
1. A single Producer can be used to write to multiple Topics and each Topic may have different partitioning requirements.  The Producer can only have a single Partitioner so the Partitioner needs to be able to switch behaviour based on the Topic, including the need to fallback to default behaviour if a given Topic does not have a custom requirement.
2. Multiple services may need to produce to the same Topic and these services may be authored by different teams.  A single custom Partitioner that encapsulates the Topic specific partitioning logic can be used by all teams at all times for all Topics ensuring that mistakes are not made.

It looks like there would be an easy solution which does not rely on Partitioner delegation.  This would be to allow the Partitioner to return -1 (RecordMetadata.UNKNOWN_PARTITION) and have the Producer fallback to default behaviour.  This option was rejected as the solution to KIP-794 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner#KIP794:StrictlyUniformStickyPartitioner-Partitioner.partitiontoreturn-1<https://cwiki.apache.org/confluence/display/KAFKA/KIP-794:+Strictly+Uniform+Sticky+Partitioner#KIP794:StrictlyUniformStickyPartitioner-Partitioner.partitiontoreturn-1>) and dismissed as not useful to Partitioners in general.  So although not necessary for KIP-794 which could remain implemented as it currently is, I think this fallback scenario is something that should be supported - it is useful to custom Partitioners for the reasons outlined above.

So I would like to propose that KafkaProducer.partition(...) not throw IllegalArgumentException if the Partitioner returns RecordMetadata.UNKNOWN_PARTITION and instead continue with the default behaviour.  Maybe with a configuration flag to enable this behaviour so as not to break existing expectations?