[jira] [Commented] (KAFKA-1863) Exception categories / hierarchy in clients

["Navina Ramesh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277814#comment-14277814 ] 

Navina Ramesh commented on KAFKA-1863:
--------------------------------------

Summarizing the issue we face in Samza:

In Samza, we want to be able to retry producer.send when certain exceptions (mostly, RetriableException) occur, that can be mitigated over time with retries. This way, we can leverage the pipelining feature in the new API, while also making sure that a single send failure does not bring down the Samza container.

The main point of concern is that the defined callback can get invoked in either the main thread or the IO thread from Kafka. This means we have to catch exceptions in more than one point in Samza producer code to ensure that we retry on the right exceptions and also, have to share states across the main thread and the IO thread (which executes the callback). 

A clearly defined hierarchy of exceptions will be a good starting point to ensure that our implementation logic is correct and deterministic.

The patch for KafkaSystemProducer in Samza with the new producer API is available in this [patch|https://reviews.apache.org/r/29899/diff/#]

> Exception categories / hierarchy in clients
> -------------------------------------------
>
>                 Key: KAFKA-1863
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1863
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Guozhang Wang
>            Assignee: Guozhang Wang
>             Fix For: 0.9.0
>
>
> In the new clients package we introduces a new set of exceptions, but its hierarchy is not very clear as of today:
> {code}
> RuntimeException -> KafkaException -> BufferExhastedException
>                                                            -> ConfigException
>                                                            -> SerializationException
>                                                            -> QuotaViolationException
>                                                            -> SchemaException
>                                                            -> ApiException
> ApiException -> InvalidTopicException
>                      -> OffsetMetadataTooLarge (probabaly need to be renamed)
>                      -> RecordBatchTooLargeException
>                      -> RecordTooLargeException
>                      -> UnknownServerException
>                      -> RetriableException
> RetriableException -> CorruptRecordException
>                                -> InvalidMetadataException
>                                -> NotEnoughtReplicasAfterAppendException
>                                -> NotEnoughReplicasException
>                                -> OffsetOutOfRangeException
>                                -> TimeoutException
>                                -> UnknownTopicOrPartitionException
> {code}
> KafkaProducer.send() may throw KafkaExceptions that are not ApiExceptions; other exceptions will be set in the returned future metadata.
> We need better to
> 1. Re-examine the hierarchy. For example, for producers only exceptions that are thrown directly from the caller thread before it is appended to the batch buffer should be ApiExceptions; some exceptions could be renamed / merged.
> 2. Clearly document the exception category / hierarchy as part of the release.
> [~criccomini] may have some more feedbacks for this issue from Samza's usage experience. [~jkreps]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)