New producer very slow to call Callback on error

[Christofer Hedbrandh <ch...@knewton.com>]

I think this question might relate to the very recently posted "callback
handler is not getting called if cluster is down" topic from "ankit tyagi".

I am using the 0.8.2.1 new producer send(ProducerRecord<K,V> record,
Callback callback) with a Callback and never calling .get() on the
Future<RecordMetadata>. I have noticed that it takes ~10-15 minutes before
Callback.onCompletion() is called with an Exception when I take down the
Kafka cluster or block connections.

>From the documentation it is unclear what the expected producer behavior
should be. I had assumed that the configuration "retries" and "timeout.ms"
should control how long it would take for the Callback to be called with an
Exception, but that does not seem to be the case.

How do I know how long it will take for the Callback to be called with an
Exception. And can I somehow configure this to be much shorter than ~10-15
minutes.

relevant broker configs:
zookeeper.session.timeout.ms: 6000
zookeeper.connection.timeout.m: 1000000

relevant producer configs:
retries: 0
linger.ms: 0
timeout.ms: 30000
metadata.fetch.timeout.ms: 10000
metadata.max.age.ms: 10000
reconnect.backoff.ms: 10
retry.backoff.ms: 100


Thanks,
Christofer

Re: New producer very slow to call Callback on error

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

KIP-19 should address this issue.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+ti
meout+to+NetworkClient

Thanks,

Jiangjie (Becket) Qin

On 6/8/15, 10:44 AM, "Christofer Hedbrandh" <ch...@knewton.com> wrote:

>I think this question might relate to the very recently posted "callback
>handler is not getting called if cluster is down" topic from "ankit
>tyagi".
>
>I am using the 0.8.2.1 new producer send(ProducerRecord<K,V> record,
>Callback callback) with a Callback and never calling .get() on the
>Future<RecordMetadata>. I have noticed that it takes ~10-15 minutes before
>Callback.onCompletion() is called with an Exception when I take down the
>Kafka cluster or block connections.
>
>From the documentation it is unclear what the expected producer behavior
>should be. I had assumed that the configuration "retries" and "timeout.ms"
>should control how long it would take for the Callback to be called with
>an
>Exception, but that does not seem to be the case.
>
>How do I know how long it will take for the Callback to be called with an
>Exception. And can I somehow configure this to be much shorter than ~10-15
>minutes.
>
>relevant broker configs:
>zookeeper.session.timeout.ms: 6000
>zookeeper.connection.timeout.m: 1000000
>
>relevant producer configs:
>retries: 0
>linger.ms: 0
>timeout.ms: 30000
>metadata.fetch.timeout.ms: 10000
>metadata.max.age.ms: 10000
>reconnect.backoff.ms: 10
>retry.backoff.ms: 100
>
>
>Thanks,
>Christofer