You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Raghav <ra...@gmail.com> on 2017/08/10 04:51:16 UTC
How to debug - NETWORK_EXCEPTION
Hi
I am sending very small 32 byte message to Kafka broker in a tight loop
with 250ms sleep. I have one broker, 1 partition, and replication factor =
1.
After about 4200 messages, I get *following *error pasted below.
How can I debug this error ? Can you please throw some ideas for me to
debug ? Stuck on this for a while now. Need help here.
17/08/10 04:46:58 WARN internals.Sender:307 Got error produce response with
correlation id 4205 on topic-partition topic04-0, retrying (2 attempts
left). Error: NETWORK_EXCEPTI
ON
17/08/10 04:47:29 WARN internals.Sender:307 Got error produce response with
correlation id 4207 on topic-partition topic04-0, retrying (1 attempts
left). Error: NETWORK_EXCEPT$ON
17/08/10 04:47:59 WARN internals.Sender:307 Got error produce response with
correlation id 4209 on topic-partition topic04-0, retrying (0 attempts
left). Error: NETWORK_EXCEPT$ON
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.NetworkException: The server disconnected
before a response was received.
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65)
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52)
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
--
Raghav
Re: How to debug - NETWORK_EXCEPTION
Posted by Raghav <ra...@gmail.com>.
Hey Martin
I am using default setting for queue.enqueue.timeout.ms. since I have not
set it my Java client. Network doesn't seem to time out either.
Could I be missing something else ?
On Sat, Aug 12, 2017 at 5:18 AM, Martin Gainty <mg...@hotmail.com> wrote:
>
>
> ________________________________
> From: Raghav <ra...@gmail.com>
> Sent: Thursday, August 10, 2017 12:51 AM
> To: Users; confluent-platform@googlegroups.com
> Subject: How to debug - NETWORK_EXCEPTION
>
> Hi
>
> I am sending very small 32 byte message to Kafka broker in a tight loop
> with 250ms sleep. I have one broker, 1 partition, and replication factor =
> 1.
>
> After about 4200 messages, I get *following *error pasted below.
>
> How can I debug this error ? Can you please throw some ideas for me to
> debug ? Stuck on this for a while now. Need help here.
>
> 17/08/10 04:46:58 WARN internals.Sender:307 Got error produce response with
> correlation id 4205 on topic-partition topic04-0, retrying (2 attempts
> left). Error: NETWORK_EXCEPTI
> ON
> 17/08/10 04:47:29 WARN internals.Sender:307 Got error produce response with
> correlation id 4207 on topic-partition topic04-0, retrying (1 attempts
> left). Error: NETWORK_EXCEPT$ON
> 17/08/10 04:47:59 WARN internals.Sender:307 Got error produce response with
> correlation id 4209 on topic-partition topic04-0, retrying (0 attempts
> left). Error: NETWORK_EXCEPT$ON
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.NetworkException: The server disconnected
> before a response was received.
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.
> valueOrError(FutureRecordMetadata.java:65)
> MG>i would check one of 2 possibilities :
> MG>A)your network is timing out
> MG>B)your message is queue is full
> MG>i would check timeout first so
> MG>in producer.properties check queue.enqueue.timeout.ms=
> MG>https://kafka.apache.org/08/documentation.html
> Apache Kafka<https://kafka.apache.org/08/documentation.html>
> kafka.apache.org
> Each partition is an ordered, immutable sequence of messages that is
> continually appended to—a commit log. The messages in the partitions are
> each assigned a ...
>
> MG>if you have zookeeper enabled check zookeeper.connection.timeout.ms in
> both server.properties and consumer.properties
>
> MG>check consumer.timeout.ms in consumer.properties
>
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(
> FutureRecordMetadata.java:52)
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(
> FutureRecordMetadata.java:25)
>
> --
> Raghav
>
--
Raghav
Re: How to debug - NETWORK_EXCEPTION
Posted by Martin Gainty <mg...@hotmail.com>.
________________________________
From: Raghav <ra...@gmail.com>
Sent: Thursday, August 10, 2017 12:51 AM
To: Users; confluent-platform@googlegroups.com
Subject: How to debug - NETWORK_EXCEPTION
Hi
I am sending very small 32 byte message to Kafka broker in a tight loop
with 250ms sleep. I have one broker, 1 partition, and replication factor =
1.
After about 4200 messages, I get *following *error pasted below.
How can I debug this error ? Can you please throw some ideas for me to
debug ? Stuck on this for a while now. Need help here.
17/08/10 04:46:58 WARN internals.Sender:307 Got error produce response with
correlation id 4205 on topic-partition topic04-0, retrying (2 attempts
left). Error: NETWORK_EXCEPTI
ON
17/08/10 04:47:29 WARN internals.Sender:307 Got error produce response with
correlation id 4207 on topic-partition topic04-0, retrying (1 attempts
left). Error: NETWORK_EXCEPT$ON
17/08/10 04:47:59 WARN internals.Sender:307 Got error produce response with
correlation id 4209 on topic-partition topic04-0, retrying (0 attempts
left). Error: NETWORK_EXCEPT$ON
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.NetworkException: The server disconnected
before a response was received.
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65)
MG>i would check one of 2 possibilities :
MG>A)your network is timing out
MG>B)your message is queue is full
MG>i would check timeout first so
MG>in producer.properties check queue.enqueue.timeout.ms=
MG>https://kafka.apache.org/08/documentation.html
Apache Kafka<https://kafka.apache.org/08/documentation.html>
kafka.apache.org
Each partition is an ordered, immutable sequence of messages that is continually appended to—a commit log. The messages in the partitions are each assigned a ...
MG>if you have zookeeper enabled check zookeeper.connection.timeout.ms in both server.properties and consumer.properties
MG>check consumer.timeout.ms in consumer.properties
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52)
at
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
--
Raghav
Re: How to debug - NETWORK_EXCEPTION
Posted by Raghav <ra...@gmail.com>.
Kafka Brokers only. Clients were Java client that used the same client
version as the broker.
On Thu, Aug 31, 2017 at 5:43 AM, Saravanan Tirugnanum <vt...@gmail.com>
wrote:
> Thank you Raghav. Was it like you upgraded Kafka Broker or Clients or both.
>
> Regards
> Saravanan
>
> On Wednesday, August 30, 2017 at 6:31:34 PM UTC-5, Raghav wrote:
>>
>> I was never able to debug this exception. I, unfortunately, moved to
>> Apache Kafka 10.2.1 from Confluent 3.2.1 and this issue went away.
>>
>> On Wed, Aug 30, 2017 at 11:46 AM, Saravanan Tirugnanum <vtsa...@gmail.com
>> > wrote:
>>
>>> Hi
>>>
>>> Were you able to figure out this issue? Any clue ?
>>>
>>> Regards
>>> Saravanan
>>>
>>>
>>> On Wednesday, August 9, 2017 at 11:51:19 PM UTC-5, Raghav wrote:
>>>>
>>>> Hi
>>>>
>>>> I am sending very small 32 byte message to Kafka broker in a tight loop
>>>> with 250ms sleep. I have one broker, 1 partition, and replication factor =
>>>> 1.
>>>>
>>>> After about 4200 messages, I get *following *error pasted below.
>>>>
>>>> How can I debug this error ? Can you please throw some ideas for me to
>>>> debug ? Stuck on this for a while now. Need help here.
>>>>
>>>> 17/08/10 04:46:58 WARN internals.Sender:307 Got error produce response
>>>> with correlation id 4205 on topic-partition topic04-0, retrying (2 attempts
>>>> left). Error: NETWORK_EXCEPTI
>>>> ON
>>>> 17/08/10 04:47:29 WARN internals.Sender:307 Got error produce response
>>>> with correlation id 4207 on topic-partition topic04-0, retrying (1 attempts
>>>> left). Error: NETWORK_EXCEPT$ON
>>>> 17/08/10 04:47:59 WARN internals.Sender:307 Got error produce response
>>>> with correlation id 4209 on topic-partition topic04-0, retrying (0 attempts
>>>> left). Error: NETWORK_EXCEPT$ON
>>>> java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException:
>>>> The server disconnected before a response was received.
>>>> at org.apache.kafka.clients.producer.internals.FutureRecordMeta
>>>> data.valueOrError(FutureRecordMetadata.java:65)
>>>> at org.apache.kafka.clients.producer.internals.FutureRecordMeta
>>>> data.get(FutureRecordMetadata.java:52)
>>>> at org.apache.kafka.clients.producer.internals.FutureRecordMeta
>>>> data.get(FutureRecordMetadata.java:25)
>>>>
>>>> --
>>>> Raghav
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Confluent Platform" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to confluent-platform+unsubscribe@googlegroups.com.
>>> To post to this group, send email to confluent...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/confluent-platform/0d1e67c3-7b5e-4b96-bc2d-bd9ba79d5dc7%
>>> 40googlegroups.com
>>> <https://groups.google.com/d/msgid/confluent-platform/0d1e67c3-7b5e-4b96-bc2d-bd9ba79d5dc7%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Raghav
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to confluent-platform+unsubscribe@googlegroups.com.
> To post to this group, send email to confluent-platform@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/confluent-platform/8f476b2e-c0a1-40bc-9968-
> f06f042fe780%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/8f476b2e-c0a1-40bc-9968-f06f042fe780%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>
--
Raghav
Re: How to debug - NETWORK_EXCEPTION
Posted by Saravanan Tirugnanum <vt...@gmail.com>.
Thank you Raghav. Was it like you upgraded Kafka Broker or Clients or both.
Regards
Saravanan
On Wednesday, August 30, 2017 at 6:31:34 PM UTC-5, Raghav wrote:
>
> I was never able to debug this exception. I, unfortunately, moved to
> Apache Kafka 10.2.1 from Confluent 3.2.1 and this issue went away.
>
> On Wed, Aug 30, 2017 at 11:46 AM, Saravanan Tirugnanum <vtsa...@gmail.com
> <javascript:>> wrote:
>
>> Hi
>>
>> Were you able to figure out this issue? Any clue ?
>>
>> Regards
>> Saravanan
>>
>>
>> On Wednesday, August 9, 2017 at 11:51:19 PM UTC-5, Raghav wrote:
>>>
>>> Hi
>>>
>>> I am sending very small 32 byte message to Kafka broker in a tight loop
>>> with 250ms sleep. I have one broker, 1 partition, and replication factor =
>>> 1.
>>>
>>> After about 4200 messages, I get *following *error pasted below.
>>>
>>> How can I debug this error ? Can you please throw some ideas for me to
>>> debug ? Stuck on this for a while now. Need help here.
>>>
>>> 17/08/10 04:46:58 WARN internals.Sender:307 Got error produce response
>>> with correlation id 4205 on topic-partition topic04-0, retrying (2 attempts
>>> left). Error: NETWORK_EXCEPTI
>>> ON
>>> 17/08/10 04:47:29 WARN internals.Sender:307 Got error produce response
>>> with correlation id 4207 on topic-partition topic04-0, retrying (1 attempts
>>> left). Error: NETWORK_EXCEPT$ON
>>> 17/08/10 04:47:59 WARN internals.Sender:307 Got error produce response
>>> with correlation id 4209 on topic-partition topic04-0, retrying (0 attempts
>>> left). Error: NETWORK_EXCEPT$ON
>>> java.util.concurrent.ExecutionException:
>>> org.apache.kafka.common.errors.NetworkException: The server disconnected
>>> before a response was received.
>>> at
>>> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65)
>>> at
>>> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52)
>>> at
>>> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
>>>
>>> --
>>> Raghav
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Confluent Platform" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to confluent-platform+unsubscribe@googlegroups.com <javascript:>.
>> To post to this group, send email to confluent...@googlegroups.com
>> <javascript:>.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/confluent-platform/0d1e67c3-7b5e-4b96-bc2d-bd9ba79d5dc7%40googlegroups.com
>> <https://groups.google.com/d/msgid/confluent-platform/0d1e67c3-7b5e-4b96-bc2d-bd9ba79d5dc7%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Raghav
>
Re: How to debug - NETWORK_EXCEPTION
Posted by Raghav <ra...@gmail.com>.
I was never able to debug this exception. I, unfortunately, moved to Apache
Kafka 10.2.1 from Confluent 3.2.1 and this issue went away.
On Wed, Aug 30, 2017 at 11:46 AM, Saravanan Tirugnanum <vt...@gmail.com>
wrote:
> Hi
>
> Were you able to figure out this issue? Any clue ?
>
> Regards
> Saravanan
>
>
> On Wednesday, August 9, 2017 at 11:51:19 PM UTC-5, Raghav wrote:
>>
>> Hi
>>
>> I am sending very small 32 byte message to Kafka broker in a tight loop
>> with 250ms sleep. I have one broker, 1 partition, and replication factor =
>> 1.
>>
>> After about 4200 messages, I get *following *error pasted below.
>>
>> How can I debug this error ? Can you please throw some ideas for me to
>> debug ? Stuck on this for a while now. Need help here.
>>
>> 17/08/10 04:46:58 WARN internals.Sender:307 Got error produce response
>> with correlation id 4205 on topic-partition topic04-0, retrying (2 attempts
>> left). Error: NETWORK_EXCEPTI
>> ON
>> 17/08/10 04:47:29 WARN internals.Sender:307 Got error produce response
>> with correlation id 4207 on topic-partition topic04-0, retrying (1 attempts
>> left). Error: NETWORK_EXCEPT$ON
>> 17/08/10 04:47:59 WARN internals.Sender:307 Got error produce response
>> with correlation id 4209 on topic-partition topic04-0, retrying (0 attempts
>> left). Error: NETWORK_EXCEPT$ON
>> java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NetworkException:
>> The server disconnected before a response was received.
>> at org.apache.kafka.clients.producer.internals.FutureRecordMeta
>> data.valueOrError(FutureRecordMetadata.java:65)
>> at org.apache.kafka.clients.producer.internals.FutureRecordMeta
>> data.get(FutureRecordMetadata.java:52)
>> at org.apache.kafka.clients.producer.internals.FutureRecordMeta
>> data.get(FutureRecordMetadata.java:25)
>>
>> --
>> Raghav
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Confluent Platform" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to confluent-platform+unsubscribe@googlegroups.com.
> To post to this group, send email to confluent-platform@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/confluent-platform/0d1e67c3-7b5e-4b96-bc2d-
> bd9ba79d5dc7%40googlegroups.com
> <https://groups.google.com/d/msgid/confluent-platform/0d1e67c3-7b5e-4b96-bc2d-bd9ba79d5dc7%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
Raghav
Re: How to debug - NETWORK_EXCEPTION
Posted by Saravanan Tirugnanum <vt...@gmail.com>.
Hi
Were you able to figure out this issue? Any clue ?
Regards
Saravanan
On Wednesday, August 9, 2017 at 11:51:19 PM UTC-5, Raghav wrote:
>
> Hi
>
> I am sending very small 32 byte message to Kafka broker in a tight loop
> with 250ms sleep. I have one broker, 1 partition, and replication factor =
> 1.
>
> After about 4200 messages, I get *following *error pasted below.
>
> How can I debug this error ? Can you please throw some ideas for me to
> debug ? Stuck on this for a while now. Need help here.
>
> 17/08/10 04:46:58 WARN internals.Sender:307 Got error produce response
> with correlation id 4205 on topic-partition topic04-0, retrying (2 attempts
> left). Error: NETWORK_EXCEPTI
> ON
> 17/08/10 04:47:29 WARN internals.Sender:307 Got error produce response
> with correlation id 4207 on topic-partition topic04-0, retrying (1 attempts
> left). Error: NETWORK_EXCEPT$ON
> 17/08/10 04:47:59 WARN internals.Sender:307 Got error produce response
> with correlation id 4209 on topic-partition topic04-0, retrying (0 attempts
> left). Error: NETWORK_EXCEPT$ON
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.NetworkException: The server disconnected
> before a response was received.
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65)
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52)
> at
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
>
> --
> Raghav
>