Error handling in async Producer

[Yury Ruchin <yu...@gmail.com>]

Hi,

Having Kafka 0.8, I send messages using Producer in async mode. I wonder
what will happen if a message cannot be sent (e. g. all brokers get down).

In sync mode, error handling is straightforward: after
"message.send.max.retries" the send() method will throw
FailedToSendMessageException. However, in async mode send() does not
actually check whether message has been successfully sent. It just enqueues
messages to be processed by ProducerSendThread in background. As I
currently understand, possible errors will not be exposed to the sender
then. Am I missing something? Is there any way to set error handler for
async Producer?

Re: Error handling in async Producer

[Jun Rao <ju...@gmail.com>]

If all brokers are down, the async send will fail. In trunk, we have a new
producer written in java, which allows the caller to register a callback
during async send.

Thanks,

Jun


On Wed, Mar 12, 2014 at 6:52 AM, Yury Ruchin <yu...@gmail.com> wrote:

> Hi,
>
> Having Kafka 0.8, I send messages using Producer in async mode. I wonder
> what will happen if a message cannot be sent (e. g. all brokers get down).
>
> In sync mode, error handling is straightforward: after
> "message.send.max.retries" the send() method will throw
> FailedToSendMessageException. However, in async mode send() does not
> actually check whether message has been successfully sent. It just enqueues
> messages to be processed by ProducerSendThread in background. As I
> currently understand, possible errors will not be exposed to the sender
> then. Am I missing something? Is there any way to set error handler for
> async Producer?
>

Re: Error handling in async Producer

[Viktor Kolodrevskiy <vi...@gmail.com>]

I had similar questions some time ago.

Here's answer I've got:

> In Kafka, we detect failures using ZK. So, if the network connectivity btw
> the producer and the broker is down, but the one btw the broker and ZK is
> up, we assume the broker is still alive and will continue to send the data
> to it. Within the the same data center, we assume this is extremely rare.
> If the network connectivity btw the broker and ZK is also down, the
> producer will be able to automatically detect the failure and send data to
> other brokers.

> In 0.7, there is no way to check whether a message is really sent or not,
> since the producer requests don't receive any acknowledgement. This is
> changed in 0.8, where a producer can choose when to receive an ack (see
> request.required.acks in
> http://kafka.apache.org/documentation.html#producerconfigs).
>
> Thanks,
>
> Jun

2014-03-12 15:52 GMT+02:00 Yury Ruchin <yu...@gmail.com>:
> Hi,
>
> Having Kafka 0.8, I send messages using Producer in async mode. I wonder
> what will happen if a message cannot be sent (e. g. all brokers get down).
>
> In sync mode, error handling is straightforward: after
> "message.send.max.retries" the send() method will throw
> FailedToSendMessageException. However, in async mode send() does not
> actually check whether message has been successfully sent. It just enqueues
> messages to be processed by ProducerSendThread in background. As I
> currently understand, possible errors will not be exposed to the sender
> then. Am I missing something? Is there any way to set error handler for
> async Producer?



-- 
Thanks,
Viktor