[jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer

["Ismael Juma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15547518#comment-15547518 ] 

Ismael Juma commented on KAFKA-4254:
------------------------------------

Do we really want to have more cases where we fail slow on the off chance that some very rare operation is taking place? Adding partitions to a topic is pretty rare, so I think the second option may be better.

> Questionable handling of unknown partitions in KafkaProducer
> ------------------------------------------------------------
>
>                 Key: KAFKA-4254
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4254
>             Project: Kafka
>          Issue Type: Bug
>          Components: producer 
>            Reporter: Jason Gustafson
>            Assignee: Jason Gustafson
>             Fix For: 0.10.1.1
>
>
> Currently the producer will raise an {{IllegalArgumentException}} if the user attempts to write to a partition which has just been created. This is caused by the fact that the producer does not attempt to refetch topic metadata in this case, which means that its check for partition validity is based on stale metadata.
> If the topic for the partition did not already exist, it works fine because the producer will block until it has metadata for the topic, so this case is primarily hit when the number of partitions is dynamically increased. 
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the partition doesn't exist, we refetch metadata and try again (timing out when max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle the error. Raising {{IllegalArgumentException}} is not helpful in practice because it can also be caused by other error.s
> My inclination is to do the first one since the producer seems incorrect to tell the user that the partition is invalid.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)