KafkaProducer stuck in send() call

[Oleg Zhurakousky <oz...@hortonworks.com>]

So we have the following scenario
Kafka broker (Kerberized) goes into the state of unresponsiveness while KafkaProducer issued a send() call.  Regardless of the reasons why the broker ends up in this state my question is really about KafkaProducer. It continues to retry (see stack trace below) even after close() call was issued.

2016-04-21 12:55:45,668 WARN [kafka-producer-network-thread | NiFi-88acbefb-3d89-4be6-bb9e-d02e776ff459] org.apache.kafka.common.network.Selector Error in I/O with connection to my-cluster-3-1.foobar/132.221.14.63
java.io.IOException: Unable to authenticate using SASL javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]) occurred when evaluating Kafka Brokers  received SASL token. Kafka Client will go to AUTH_FAILED state.
at org.apache.kafka.common.network.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:172) ~[kafka-clients-0.8.2.2.3.2.0-2950.jar:na]
at org.apache.kafka.common.network.Channel.connect(Channel.java:71) ~[kafka-clients-0.8.2.2.3.2.0-2950.jar:na]
at org.apache.kafka.common.network.Selector.poll(Selector.java:326) ~[kafka-clients-0.8.2.2.3.2.0-2950.jar:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:221) [kafka-clients-0.8.2.2.3.2.0-2950.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:209) [kafka-clients-0.8.2.2.3.2.0-2950.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:127) [kafka-clients-0.8.2.2.3.2.0-2950.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_71]


So, my question; Is there a way to stop KafkaProducer in such was that those re-try threads are interrupted. Basically it appears that aside from killing the JVM the producer runs in, there is no other way to do that or is there?


Thank you
Oleg

Re: KafkaProducer stuck in send() call

[Oleg Zhurakousky <oz...@hortonworks.com>]

Thank you Vinay

Will give it a shot and see if it happens again.

Oleg

> On Apr 21, 2016, at 12:26 PM, vinay sharma <vi...@gmail.com> wrote:
> 
> Hi Oleg,
> 
> There are a few configurations of kafka producer that are worth considering
> in your case.:- ProducerConfig.RETRIES_CONFIG
> 
> it's java doc says "Setting a value greater than zero will cause the client
> to resend any record whose send fails with a potentially transient error.
> Note that this retry is no different than if the client resent the record
> upon receiving the error. Allowing retries will potentially change the
> ordering of records because if two records are sent to a single partition,
> and the first fails and is retried but the second succeeds, then the second
> record may appear first."
> 
> You can set retry off by setting this value to zero or less and handle send
> failures and retries yourself in your code.
> 
> Regards,
> Vinay Sharma

Re: KafkaProducer stuck in send() call

[vinay sharma <vi...@gmail.com>]

Hi Oleg,

There are a few configurations of kafka producer that are worth considering
in your case.:- ProducerConfig.RETRIES_CONFIG

it's java doc says "Setting a value greater than zero will cause the client
to resend any record whose send fails with a potentially transient error.
Note that this retry is no different than if the client resent the record
upon receiving the error. Allowing retries will potentially change the
ordering of records because if two records are sent to a single partition,
and the first fails and is retried but the second succeeds, then the second
record may appear first."

You can set retry off by setting this value to zero or less and handle send
failures and retries yourself in your code.

Regards,
Vinay Sharma