You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2014/09/17 23:14:08 UTC
[Java New Producer] CPU Usage Spike to 100% when network connection
is lost
Hi Kafka Dev team,
I see my CPU spike to 100% when network connection is lost for while. It
seems network IO thread are very busy logging following error message. Is
this expected behavior ?
2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR
org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
kafka producer I/O thread:
java.lang.IllegalStateException: No entry found for node -2
at org.apache.kafka.clients.ClusterConnectionStates.nodeState(
ClusterConnectionStates.java:110)
at org.apache.kafka.clients.ClusterConnectionStates.disconnected(
ClusterConnectionStates.java:99)
at org.apache.kafka.clients.NetworkClient.initiateConnect(
NetworkClient.java:394)
at org.apache.kafka.clients.NetworkClient.maybeUpdateMetadata(
NetworkClient.java:380)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:174)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:744)
Thanks,
Bhavesh
Re: [Java New Producer] CPU Usage Spike to 100% when network
connection is lost
Posted by Neha Narkhede <ne...@gmail.com>.
This seems like a problem. Please can you file a JIRA and attach the log4j
output there?
On Wed, Sep 17, 2014 at 2:14 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Dev team,
>
> I see my CPU spike to 100% when network connection is lost for while. It
> seems network IO thread are very busy logging following error message. Is
> this expected behavior ?
>
> 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR
> org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> kafka producer I/O thread:
>
> java.lang.IllegalStateException: No entry found for node -2
>
> at org.apache.kafka.clients.ClusterConnectionStates.nodeState(
> ClusterConnectionStates.java:110)
>
> at org.apache.kafka.clients.ClusterConnectionStates.disconnected(
> ClusterConnectionStates.java:99)
>
> at org.apache.kafka.clients.NetworkClient.initiateConnect(
> NetworkClient.java:394)
>
> at org.apache.kafka.clients.NetworkClient.maybeUpdateMetadata(
> NetworkClient.java:380)
>
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:174)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Thanks,
>
> Bhavesh
>
Re: [Java New Producer] CPU Usage Spike to 100% when network
connection is lost
Posted by Jay Kreps <ja...@gmail.com>.
Hmm, yes, the fix was prior to that, that sounds like a bug.
-Jay
On Thu, Sep 18, 2014 at 9:39 AM, Bhavesh Mistry
<mi...@gmail.com> wrote:
> HI Jay,
>
> I am running trunk producer based on following last commit with timestamp
> Mon Sep 15 20:34:14 2014 -0700. Please let me know if this timestamp
> contains the fix. Otherwise, I will file a bug with logs as Neha suggested.
>
> commit cf0f5750b39e675cf9a9c6d6394a665366db0f58
> Author: Alexis Midon <mi...@apache.org>
> Date: Mon Sep 15 20:34:14 2014 -0700
> KAFKA-1597 New metrics: ResponseQueueSize and BeingSentResponses;
> reviewed by Neha Narkhede and Jun Rao
>
>
> Thanks,
>
> Bhavesh
>
> On Wed, Sep 17, 2014 at 11:22 PM, Jay Kreps <ja...@gmail.com> wrote:
>
>> Also do you know what version you are running we did fix several bugs
>> similar to this against trunk.
>>
>> -Jay
>>
>> On Wed, Sep 17, 2014 at 2:14 PM, Bhavesh Mistry
>> <mi...@gmail.com> wrote:
>> > Hi Kafka Dev team,
>> >
>> > I see my CPU spike to 100% when network connection is lost for while. It
>> > seems network IO thread are very busy logging following error message.
>> Is
>> > this expected behavior ?
>> >
>> > 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR
>> > org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
>> > kafka producer I/O thread:
>> >
>> > java.lang.IllegalStateException: No entry found for node -2
>> >
>> > at org.apache.kafka.clients.ClusterConnectionStates.nodeState(
>> > ClusterConnectionStates.java:110)
>> >
>> > at org.apache.kafka.clients.ClusterConnectionStates.disconnected(
>> > ClusterConnectionStates.java:99)
>> >
>> > at org.apache.kafka.clients.NetworkClient.initiateConnect(
>> > NetworkClient.java:394)
>> >
>> > at org.apache.kafka.clients.NetworkClient.maybeUpdateMetadata(
>> > NetworkClient.java:380)
>> >
>> > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:174)
>> >
>> > at
>> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
>> >
>> > at
>> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
>> >
>> > at java.lang.Thread.run(Thread.java:744)
>> >
>> > Thanks,
>> >
>> > Bhavesh
>>
Re: [Java New Producer] CPU Usage Spike to 100% when network
connection is lost
Posted by Bhavesh Mistry <mi...@gmail.com>.
HI Jay,
I am running trunk producer based on following last commit with timestamp
Mon Sep 15 20:34:14 2014 -0700. Please let me know if this timestamp
contains the fix. Otherwise, I will file a bug with logs as Neha suggested.
commit cf0f5750b39e675cf9a9c6d6394a665366db0f58
Author: Alexis Midon <mi...@apache.org>
Date: Mon Sep 15 20:34:14 2014 -0700
KAFKA-1597 New metrics: ResponseQueueSize and BeingSentResponses;
reviewed by Neha Narkhede and Jun Rao
Thanks,
Bhavesh
On Wed, Sep 17, 2014 at 11:22 PM, Jay Kreps <ja...@gmail.com> wrote:
> Also do you know what version you are running we did fix several bugs
> similar to this against trunk.
>
> -Jay
>
> On Wed, Sep 17, 2014 at 2:14 PM, Bhavesh Mistry
> <mi...@gmail.com> wrote:
> > Hi Kafka Dev team,
> >
> > I see my CPU spike to 100% when network connection is lost for while. It
> > seems network IO thread are very busy logging following error message.
> Is
> > this expected behavior ?
> >
> > 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR
> > org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> > kafka producer I/O thread:
> >
> > java.lang.IllegalStateException: No entry found for node -2
> >
> > at org.apache.kafka.clients.ClusterConnectionStates.nodeState(
> > ClusterConnectionStates.java:110)
> >
> > at org.apache.kafka.clients.ClusterConnectionStates.disconnected(
> > ClusterConnectionStates.java:99)
> >
> > at org.apache.kafka.clients.NetworkClient.initiateConnect(
> > NetworkClient.java:394)
> >
> > at org.apache.kafka.clients.NetworkClient.maybeUpdateMetadata(
> > NetworkClient.java:380)
> >
> > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:174)
> >
> > at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
> >
> > at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> >
> > at java.lang.Thread.run(Thread.java:744)
> >
> > Thanks,
> >
> > Bhavesh
>
Re: [Java New Producer] CPU Usage Spike to 100% when network
connection is lost
Posted by Jay Kreps <ja...@gmail.com>.
Also do you know what version you are running we did fix several bugs
similar to this against trunk.
-Jay
On Wed, Sep 17, 2014 at 2:14 PM, Bhavesh Mistry
<mi...@gmail.com> wrote:
> Hi Kafka Dev team,
>
> I see my CPU spike to 100% when network connection is lost for while. It
> seems network IO thread are very busy logging following error message. Is
> this expected behavior ?
>
> 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR
> org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> kafka producer I/O thread:
>
> java.lang.IllegalStateException: No entry found for node -2
>
> at org.apache.kafka.clients.ClusterConnectionStates.nodeState(
> ClusterConnectionStates.java:110)
>
> at org.apache.kafka.clients.ClusterConnectionStates.disconnected(
> ClusterConnectionStates.java:99)
>
> at org.apache.kafka.clients.NetworkClient.initiateConnect(
> NetworkClient.java:394)
>
> at org.apache.kafka.clients.NetworkClient.maybeUpdateMetadata(
> NetworkClient.java:380)
>
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:174)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Thanks,
>
> Bhavesh
Re: [Java New Producer] CPU Usage Spike to 100% when network
connection is lost
Posted by Jay Kreps <ja...@gmail.com>.
Also do you know what version you are running we did fix several bugs
similar to this against trunk.
-Jay
On Wed, Sep 17, 2014 at 2:14 PM, Bhavesh Mistry
<mi...@gmail.com> wrote:
> Hi Kafka Dev team,
>
> I see my CPU spike to 100% when network connection is lost for while. It
> seems network IO thread are very busy logging following error message. Is
> this expected behavior ?
>
> 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR
> org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> kafka producer I/O thread:
>
> java.lang.IllegalStateException: No entry found for node -2
>
> at org.apache.kafka.clients.ClusterConnectionStates.nodeState(
> ClusterConnectionStates.java:110)
>
> at org.apache.kafka.clients.ClusterConnectionStates.disconnected(
> ClusterConnectionStates.java:99)
>
> at org.apache.kafka.clients.NetworkClient.initiateConnect(
> NetworkClient.java:394)
>
> at org.apache.kafka.clients.NetworkClient.maybeUpdateMetadata(
> NetworkClient.java:380)
>
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:174)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Thanks,
>
> Bhavesh
Re: [Java New Producer] CPU Usage Spike to 100% when network
connection is lost
Posted by Neha Narkhede <ne...@gmail.com>.
This seems like a problem. Please can you file a JIRA and attach the log4j
output there?
On Wed, Sep 17, 2014 at 2:14 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Kafka Dev team,
>
> I see my CPU spike to 100% when network connection is lost for while. It
> seems network IO thread are very busy logging following error message. Is
> this expected behavior ?
>
> 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR
> org.apache.kafka.clients.producer.internals.Sender - Uncaught error in
> kafka producer I/O thread:
>
> java.lang.IllegalStateException: No entry found for node -2
>
> at org.apache.kafka.clients.ClusterConnectionStates.nodeState(
> ClusterConnectionStates.java:110)
>
> at org.apache.kafka.clients.ClusterConnectionStates.disconnected(
> ClusterConnectionStates.java:99)
>
> at org.apache.kafka.clients.NetworkClient.initiateConnect(
> NetworkClient.java:394)
>
> at org.apache.kafka.clients.NetworkClient.maybeUpdateMetadata(
> NetworkClient.java:380)
>
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:174)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:175)
>
> at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Thanks,
>
> Bhavesh
>