[jira] [Comment Edited] (KAFKA-6833) KafkaProducer throws "Invalid partition given with record" exception

["Chris Egerton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16705465#comment-16705465 ] 

Chris Egerton edited comment on KAFKA-6833 at 11/30/18 11:43 PM:
-----------------------------------------------------------------

[~bob-barrett] another possible approach could be to alter the behavior of the producer while it waits for metadata on a topic-partition.

Currently, when a record is given to the producer to send, it checks its metadata cache and, if the topic-partition exists in the cache, [the cached data is used|https://github.com/apache/kafka/blob/9b476bc5f4a2fdbd62ad84e50e65331c21b321d0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L967-L968].

If the topic doesn't exist or that partition doesn't exist in the topic, the producer requests a metadata update and then waits until either {{max.block.ms}} has been exceeded or until an update has occurred that guarantees that the [topic exists|https://github.com/apache/kafka/blob/9b476bc5f4a2fdbd62ad84e50e65331c21b321d0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L998].

We could tweak the logic so that the producer waits until either {{max.block.ms}} has been exceeded or until the topic-partition exists.

This would relieve users of the KafkaProducer from having to handle retry logic for the suggested {{PartitionOutOfRangeException}}, although it could also potentially lead to slower failures when trying to send a record to an invalid partition.


was (Author: chrisegerton):
[~bob-barrett] another possible approach could be to alter the behavior of the producer while it waits for metadata on a topic-partition.

Currently, when a record is given to the producer to send, it checks its metadata cache and, if the topic-partition exists in the cache, [the cached data is used|https://github.com/apache/kafka/blob/9b476bc5f4a2fdbd62ad84e50e65331c21b321d0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L967-L968].

If the topic doesn't exist or that partition doesn't exist in the topic, the producer requests a metadata update and then waits until either {{max.block.ms}} has been exceeded or until an update has occurred that results in the [topic being created|https://github.com/apache/kafka/blob/9b476bc5f4a2fdbd62ad84e50e65331c21b321d0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L998].

We could tweak the logic so that the producer waits until either {{max.block.ms}} has been exceeded or until the topic-partition exists.

This would relieve users of the KafkaProducer from having to handle retry logic for the suggested {{PartitionOutOfRangeException}}, although it could also potentially lead to slower failures when trying to send a record to an invalid partition.

> KafkaProducer throws "Invalid partition given with record" exception
> --------------------------------------------------------------------
>
>                 Key: KAFKA-6833
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6833
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Arjun Satish
>            Assignee: Bob Barrett
>            Priority: Minor
>
> Currently, when creating topics via ZooKeeper, there is a small but definite delay between creating the nodes in ZK, and having the topics created in the brokers. the KafkaProducer maintains a metadata cache about topics which get updated after the broker metadata is updated. If an application adds partitions to a topic, and immediately tries to produce records to a new partition, a KafkaException is throw with a message similar to the following:
> {code:java}
> Caused by: org.apache.kafka.common.KafkaException: Invalid partition given with record: 12 is not in the range [0...1).
> {code}
> In this case, since the application has context that it created the topics, it might be worthwhile to consider if a more specific exception can be thrown instead of KafkaException. For example:
> {code:java}
> public class PartitionNotFoundException extends KafkaException {...}{code}
> This could allow the application to be able to interpret such an error, and act accordingly.
> EDIT: Correct "create topics" to "adds partitions to a topic".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)