[jira] [Comment Edited] (KAFKA-14546) Allow Partitioner to return -1 to indicate default partitioning

["James Olsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774202#comment-17774202 ] 

James Olsen edited comment on KAFKA-14546 at 10/11/23 7:09 PM:
---------------------------------------------------------------

I'm still hoping for a solution to this issue.  I can understand the reluctance to allow a `Partitioner` to return `RecordMetadata.UNKNOWN_PARTITION` however there is another solution that would work.  As my requirement discussed above is to ensure consistent partitioning for each Topic using a single Partitioner, it would be suitable to have the `Partitioner` indicate to the `KafkaProducer` if it handles the Topic.  This could be implemented as follows:

 
{code:java}
public interface Partitioner ... {
...
    default boolean partitionsTopic(String topic) {
        return true;
    }
...
}
{code}
 

`KafkaProducer` would call this method first and fallback to default partitioning if it returns `false`.  This would be backwards compatible with existing code and have minimal overhead.


was (Author: jim_b_o):
I'm still hoping for a solution to this issue.  I can understand the reluctance to allow a `Partitioner` to return `RecordMetadata.UNKNOWN_PARTITION` however there is another solution that would work.  As my requirement discussed above is to ensure consistent partitioning for each Topic using a single Partitioner, it would be suitable to have the `Partitioner` indicate to the `KafkaProducer` if it handles the Topic.  This could be implemented as follows:

```

public interface Partitioner ... {

...
    default boolean partitionsTopic(String topic) {

        return true;
    }

...
}

```

`KafkaProducer` would call this method first and fallback to default partitioning if it returns `false`.  This would be backwards compatible with existing code and have minimal overhead.

> Allow Partitioner to return -1 to indicate default partitioning
> ---------------------------------------------------------------
>
>                 Key: KAFKA-14546
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14546
>             Project: Kafka
>          Issue Type: Improvement
>          Components: producer 
>    Affects Versions: 3.3.1
>            Reporter: James Olsen
>            Priority: Major
>
> Prior to KIP-794 it was possible to create a custom Partitioner that could delegate to the DefaultPartitioner.  DefaultPartitioner has been deprecated so we can now only delegate to BuiltInPartitioner.partitionForKey which does not handle a non-keyed message.  Hence there is now no mechanism for a custom Partitioner to fallback to default partitioning, e.g. for the non-keyed sticky case.
> I would like to propose that KafkaProducer.partition(...) not throw IllegalArgumentException if the Partitioner returns RecordMetadata.UNKNOWN_PARTITION and instead continue with the default behaviour.  Maybe with a configuration flag to enable this behaviour so as not to break existing expectations?
> Why was Partitioner delegation with default fallback useful?
>  # A single Producer can be used to write to multiple Topics where each Topic may have different partitioning requirements.  The Producer can only have a single Partitioner so the Partitioner needs to be able to switch behaviour based on the Topic, including the need to fallback to default behaviour if a given Topic does not have a custom requirement.
>  # Multiple services may need to produce to the same Topic and these services may be authored by different teams.  A single custom Partitioner that encapsulates all Topic specific partitioning logic can be used by all teams at all times for all Topics ensuring that mistakes are not made.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)