You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Ramanan, Buvana (Nokia - US)" <bu...@nokia.com> on 2016/05/09 23:14:41 UTC
ERROR Processor got uncaught exception. (kafka.network.Processor)
Hello,
We are trying to understand the following error in one of the brokers in our cluster. It's a 3 broker cluster (all 3 servers are identical - Intel Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), serving about 8000 topics, single partitioned & replication factor = 3. Each topic gets data at a low rate - 200 bytes per sec. Cluster is managed by 3 server ZK cluster (these servers are different from Kafka broker servers). All 6 servers are connected via 10G switch. Producers run from external servers.
Leaders are balanced across the topics.
This error message occurs repeatedly in one of the servers. Between the hours of 10:30am and 1:30pm today, there were about 10 Million such occurrences.
We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I can share the current settings if needed.
Please let me know what is causing this error and a way to overcome it.
Regards,
Buvana
[2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.IllegalArgumentException: Attempted to decrease connection count for address with no connections, address: /X.Y.Z.144 (actual network address masked)
at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at scala.collection.AbstractMap.getOrElse(Map.scala:59)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.network.Processor.run(SocketServer.scala:445)
at java.lang.Thread.run(Thread.java:745)
[2016-05-09 10:38:43,932] ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.IllegalArgumentException: Attempted to decrease connection count for address with no connections, address: /X.Y.Z.144
at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
at kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at scala.collection.AbstractMap.getOrElse(Map.scala:59)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
at kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
at scala.collection.Iterator$class.foreach(Iterator.scala:742)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.network.Processor.run(SocketServer.scala:445)
at java.lang.Thread.run(Thread.java:745)
Re: ERROR Processor got uncaught exception. (kafka.network.Processor)
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks. I had a quick look at the code and it's not obvious how this could
happen. Let's see how your testing goes. :)
Ismael
On Wed, May 11, 2016 at 2:04 AM, Ramanan, Buvana (Nokia - US) <
buvana.ramanan@nokia.com> wrote:
> Ismael,
>
> We are setting up a 0.10.0 test cluster now. I will report on whether this
> bug springs up in that cluster or not after a week or so.
>
> Glad to hear that SocketServer bugs are being taken care of in 0.10.0 and
> hope this issue is ironed out as a result.
>
> Regards,
> Buvana
>
> -----Original Message-----
> From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> Ismael Juma
> Sent: Tuesday, May 10, 2016 5:00 PM
> To: users@kafka.apache.org
> Subject: Re: ERROR Processor got uncaught exception.
> (kafka.network.Processor)
>
> Thanks Buvana. Is this happening in production only or can you also
> reproduce it in a test cluster? If the latter, would you be able to test
> the latest 0.10.0.0 release candidate? We fixed a few issues in the
> SocketServer.
>
> Ismael
>
> On Tue, May 10, 2016 at 8:27 PM, Ramanan, Buvana (Nokia - US) <
> buvana.ramanan@nokia.com> wrote:
>
> > Ismael,
> >
> > Created bug:
> > https://issues.apache.org/jira/browse/KAFKA-3689
> >
> > Hope to get a quick resolution.
> >
> > Thanks,
> > Buvana
> >
> > -----Original Message-----
> > From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> > Ismael Juma
> > Sent: Tuesday, May 10, 2016 10:54 AM
> > To: users@kafka.apache.org
> > Subject: Re: ERROR Processor got uncaught exception.
> > (kafka.network.Processor)
> >
> > OK, thanks. I suggest filing a bug in JIRA and please provide as much
> > information as possible (steps to reproduce would be ideal, but
> > sometimes that is hard to do). It does look like a Kafka bug.
> >
> > Ismael
> >
> > On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) <
> > buvana.ramanan@nokia.com> wrote:
> >
> > > Ismael,
> > >
> > > Version 0.9.0.1
> > >
> > > Do you have any idea how to prevent this from happening? Is it a
> > > Kafka issue?
> > >
> > > -Buvana
> > >
> > > -----Original Message-----
> > > From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> > > Ismael Juma
> > > Sent: Monday, May 09, 2016 8:02 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: ERROR Processor got uncaught exception.
> > > (kafka.network.Processor)
> > >
> > > Hi Buvana,
> > >
> > > What is the version of the broker? It looks like the state of
> > > connection counts for quotas became inconsistent, that is, a given
> > > client ip address has no connections according to the internal
> > > state, but a disconnection
> > for
> > > that ip happened.
> > >
> > > Ismael
> > >
> > > On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) <
> > > buvana.ramanan@nokia.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > We are trying to understand the following error in one of the
> > > > brokers
> > in
> > > > our cluster. It's a 3 broker cluster (all 3 servers are identical
> > > > -
> > > Intel
> > > > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk),
> > serving
> > > > about 8000 topics, single partitioned & replication factor = 3.
> > > > Each
> > > topic
> > > > gets data at a low rate - 200 bytes per sec. Cluster is managed
> > > > by 3 server ZK cluster (these servers are different from Kafka
> > > > broker
> > > servers).
> > > > All 6 servers are connected via 10G switch. Producers run from
> > > > external servers.
> > > >
> > > > Leaders are balanced across the topics.
> > > >
> > > > This error message occurs repeatedly in one of the servers.
> > > > Between the hours of 10:30am and 1:30pm today, there were about 10
> > > > Million such occurrences.
> > > >
> > > > We tuned Linux network params (somaxconn, rmem_max, wmem_max,
> > > > etc). I
> > can
> > > > share the current settings if needed.
> > > >
> > > > Please let me know what is causing this error and a way to overcome
> it.
> > > >
> > > > Regards,
> > > > Buvana
> > > >
> > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > > > (kafka.network.Processor)
> > > > java.lang.IllegalArgumentException: Attempted to decrease
> > > > connection
> > > count
> > > > for address with no connections, address: /X.Y.Z.144 (actual
> > > > network address masked)
> > > > at
> > > >
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > > at
> > > >
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > > at
> scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > > > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > > > at
> > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > > > at
> > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > > > at
> scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > > at
> > scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > > at
> > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > > at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > > at kafka.network.Processor.run(SocketServer.scala:445)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > > > (kafka.network.Processor)
> > > > java.lang.IllegalArgumentException: Attempted to decrease
> > > > connection
> > > count
> > > > for address with no connections, address: /X.Y.Z.144
> > > > at
> > > >
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > > at
> > > >
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > > at
> scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > > > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > > > at
> > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > > > at
> > > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > > > at
> scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > > at
> > scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > > at
> > > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > > at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > > at kafka.network.Processor.run(SocketServer.scala:445)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > >
> > >
> >
>
RE: ERROR Processor got uncaught exception.
(kafka.network.Processor)
Posted by "Ramanan, Buvana (Nokia - US)" <bu...@nokia.com>.
Ismael,
We are setting up a 0.10.0 test cluster now. I will report on whether this bug springs up in that cluster or not after a week or so.
Glad to hear that SocketServer bugs are being taken care of in 0.10.0 and hope this issue is ironed out as a result.
Regards,
Buvana
-----Original Message-----
From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT Ismael Juma
Sent: Tuesday, May 10, 2016 5:00 PM
To: users@kafka.apache.org
Subject: Re: ERROR Processor got uncaught exception. (kafka.network.Processor)
Thanks Buvana. Is this happening in production only or can you also reproduce it in a test cluster? If the latter, would you be able to test the latest 0.10.0.0 release candidate? We fixed a few issues in the SocketServer.
Ismael
On Tue, May 10, 2016 at 8:27 PM, Ramanan, Buvana (Nokia - US) < buvana.ramanan@nokia.com> wrote:
> Ismael,
>
> Created bug:
> https://issues.apache.org/jira/browse/KAFKA-3689
>
> Hope to get a quick resolution.
>
> Thanks,
> Buvana
>
> -----Original Message-----
> From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> Ismael Juma
> Sent: Tuesday, May 10, 2016 10:54 AM
> To: users@kafka.apache.org
> Subject: Re: ERROR Processor got uncaught exception.
> (kafka.network.Processor)
>
> OK, thanks. I suggest filing a bug in JIRA and please provide as much
> information as possible (steps to reproduce would be ideal, but
> sometimes that is hard to do). It does look like a Kafka bug.
>
> Ismael
>
> On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) <
> buvana.ramanan@nokia.com> wrote:
>
> > Ismael,
> >
> > Version 0.9.0.1
> >
> > Do you have any idea how to prevent this from happening? Is it a
> > Kafka issue?
> >
> > -Buvana
> >
> > -----Original Message-----
> > From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> > Ismael Juma
> > Sent: Monday, May 09, 2016 8:02 PM
> > To: users@kafka.apache.org
> > Subject: Re: ERROR Processor got uncaught exception.
> > (kafka.network.Processor)
> >
> > Hi Buvana,
> >
> > What is the version of the broker? It looks like the state of
> > connection counts for quotas became inconsistent, that is, a given
> > client ip address has no connections according to the internal
> > state, but a disconnection
> for
> > that ip happened.
> >
> > Ismael
> >
> > On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) <
> > buvana.ramanan@nokia.com> wrote:
> >
> > > Hello,
> > >
> > > We are trying to understand the following error in one of the
> > > brokers
> in
> > > our cluster. It's a 3 broker cluster (all 3 servers are identical
> > > -
> > Intel
> > > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk),
> serving
> > > about 8000 topics, single partitioned & replication factor = 3.
> > > Each
> > topic
> > > gets data at a low rate - 200 bytes per sec. Cluster is managed
> > > by 3 server ZK cluster (these servers are different from Kafka
> > > broker
> > servers).
> > > All 6 servers are connected via 10G switch. Producers run from
> > > external servers.
> > >
> > > Leaders are balanced across the topics.
> > >
> > > This error message occurs repeatedly in one of the servers.
> > > Between the hours of 10:30am and 1:30pm today, there were about 10
> > > Million such occurrences.
> > >
> > > We tuned Linux network params (somaxconn, rmem_max, wmem_max,
> > > etc). I
> can
> > > share the current settings if needed.
> > >
> > > Please let me know what is causing this error and a way to overcome it.
> > >
> > > Regards,
> > > Buvana
> > >
> > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > > (kafka.network.Processor)
> > > java.lang.IllegalArgumentException: Attempted to decrease
> > > connection
> > count
> > > for address with no connections, address: /X.Y.Z.144 (actual
> > > network address masked)
> > > at
> > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > at
> > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > > at
> > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > > at
> > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > at
> > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > at kafka.network.Processor.run(SocketServer.scala:445)
> > > at java.lang.Thread.run(Thread.java:745)
> > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > > (kafka.network.Processor)
> > > java.lang.IllegalArgumentException: Attempted to decrease
> > > connection
> > count
> > > for address with no connections, address: /X.Y.Z.144
> > > at
> > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > at
> > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > > at
> > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > > at
> > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > at
> > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > at kafka.network.Processor.run(SocketServer.scala:445)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> >
>
Re: ERROR Processor got uncaught exception. (kafka.network.Processor)
Posted by Ismael Juma <is...@juma.me.uk>.
Thanks Buvana. Is this happening in production only or can you also
reproduce it in a test cluster? If the latter, would you be able to test
the latest 0.10.0.0 release candidate? We fixed a few issues in the
SocketServer.
Ismael
On Tue, May 10, 2016 at 8:27 PM, Ramanan, Buvana (Nokia - US) <
buvana.ramanan@nokia.com> wrote:
> Ismael,
>
> Created bug:
> https://issues.apache.org/jira/browse/KAFKA-3689
>
> Hope to get a quick resolution.
>
> Thanks,
> Buvana
>
> -----Original Message-----
> From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> Ismael Juma
> Sent: Tuesday, May 10, 2016 10:54 AM
> To: users@kafka.apache.org
> Subject: Re: ERROR Processor got uncaught exception.
> (kafka.network.Processor)
>
> OK, thanks. I suggest filing a bug in JIRA and please provide as much
> information as possible (steps to reproduce would be ideal, but sometimes
> that is hard to do). It does look like a Kafka bug.
>
> Ismael
>
> On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) <
> buvana.ramanan@nokia.com> wrote:
>
> > Ismael,
> >
> > Version 0.9.0.1
> >
> > Do you have any idea how to prevent this from happening? Is it a Kafka
> > issue?
> >
> > -Buvana
> >
> > -----Original Message-----
> > From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> > Ismael Juma
> > Sent: Monday, May 09, 2016 8:02 PM
> > To: users@kafka.apache.org
> > Subject: Re: ERROR Processor got uncaught exception.
> > (kafka.network.Processor)
> >
> > Hi Buvana,
> >
> > What is the version of the broker? It looks like the state of connection
> > counts for quotas became inconsistent, that is, a given client ip address
> > has no connections according to the internal state, but a disconnection
> for
> > that ip happened.
> >
> > Ismael
> >
> > On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) <
> > buvana.ramanan@nokia.com> wrote:
> >
> > > Hello,
> > >
> > > We are trying to understand the following error in one of the brokers
> in
> > > our cluster. It's a 3 broker cluster (all 3 servers are identical -
> > Intel
> > > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk),
> serving
> > > about 8000 topics, single partitioned & replication factor = 3. Each
> > topic
> > > gets data at a low rate - 200 bytes per sec. Cluster is managed by 3
> > > server ZK cluster (these servers are different from Kafka broker
> > servers).
> > > All 6 servers are connected via 10G switch. Producers run from external
> > > servers.
> > >
> > > Leaders are balanced across the topics.
> > >
> > > This error message occurs repeatedly in one of the servers. Between the
> > > hours of 10:30am and 1:30pm today, there were about 10 Million such
> > > occurrences.
> > >
> > > We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I
> can
> > > share the current settings if needed.
> > >
> > > Please let me know what is causing this error and a way to overcome it.
> > >
> > > Regards,
> > > Buvana
> > >
> > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > > (kafka.network.Processor)
> > > java.lang.IllegalArgumentException: Attempted to decrease connection
> > count
> > > for address with no connections, address: /X.Y.Z.144 (actual network
> > > address masked)
> > > at
> > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > at
> > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > > at
> > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > > at
> > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > at
> > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > at kafka.network.Processor.run(SocketServer.scala:445)
> > > at java.lang.Thread.run(Thread.java:745)
> > > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > > (kafka.network.Processor)
> > > java.lang.IllegalArgumentException: Attempted to decrease connection
> > count
> > > for address with no connections, address: /X.Y.Z.144
> > > at
> > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > at
> > > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > > at
> > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > > at
> > > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > > at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > > at
> > > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > > at kafka.network.Processor.run(SocketServer.scala:445)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> >
>
RE: ERROR Processor got uncaught exception.
(kafka.network.Processor)
Posted by "Ramanan, Buvana (Nokia - US)" <bu...@nokia.com>.
Ismael,
Created bug:
https://issues.apache.org/jira/browse/KAFKA-3689
Hope to get a quick resolution.
Thanks,
Buvana
-----Original Message-----
From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT Ismael Juma
Sent: Tuesday, May 10, 2016 10:54 AM
To: users@kafka.apache.org
Subject: Re: ERROR Processor got uncaught exception. (kafka.network.Processor)
OK, thanks. I suggest filing a bug in JIRA and please provide as much
information as possible (steps to reproduce would be ideal, but sometimes
that is hard to do). It does look like a Kafka bug.
Ismael
On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) <
buvana.ramanan@nokia.com> wrote:
> Ismael,
>
> Version 0.9.0.1
>
> Do you have any idea how to prevent this from happening? Is it a Kafka
> issue?
>
> -Buvana
>
> -----Original Message-----
> From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> Ismael Juma
> Sent: Monday, May 09, 2016 8:02 PM
> To: users@kafka.apache.org
> Subject: Re: ERROR Processor got uncaught exception.
> (kafka.network.Processor)
>
> Hi Buvana,
>
> What is the version of the broker? It looks like the state of connection
> counts for quotas became inconsistent, that is, a given client ip address
> has no connections according to the internal state, but a disconnection for
> that ip happened.
>
> Ismael
>
> On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) <
> buvana.ramanan@nokia.com> wrote:
>
> > Hello,
> >
> > We are trying to understand the following error in one of the brokers in
> > our cluster. It's a 3 broker cluster (all 3 servers are identical -
> Intel
> > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), serving
> > about 8000 topics, single partitioned & replication factor = 3. Each
> topic
> > gets data at a low rate - 200 bytes per sec. Cluster is managed by 3
> > server ZK cluster (these servers are different from Kafka broker
> servers).
> > All 6 servers are connected via 10G switch. Producers run from external
> > servers.
> >
> > Leaders are balanced across the topics.
> >
> > This error message occurs repeatedly in one of the servers. Between the
> > hours of 10:30am and 1:30pm today, there were about 10 Million such
> > occurrences.
> >
> > We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I can
> > share the current settings if needed.
> >
> > Please let me know what is causing this error and a way to overcome it.
> >
> > Regards,
> > Buvana
> >
> > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > (kafka.network.Processor)
> > java.lang.IllegalArgumentException: Attempted to decrease connection
> count
> > for address with no connections, address: /X.Y.Z.144 (actual network
> > address masked)
> > at
> > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > at
> > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > at
> > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > at
> > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > at
> > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > at kafka.network.Processor.run(SocketServer.scala:445)
> > at java.lang.Thread.run(Thread.java:745)
> > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > (kafka.network.Processor)
> > java.lang.IllegalArgumentException: Attempted to decrease connection
> count
> > for address with no connections, address: /X.Y.Z.144
> > at
> > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > at
> > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > at
> > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > at
> > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > at
> > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > at kafka.network.Processor.run(SocketServer.scala:445)
> > at java.lang.Thread.run(Thread.java:745)
> >
>
Re: ERROR Processor got uncaught exception. (kafka.network.Processor)
Posted by Ismael Juma <is...@juma.me.uk>.
OK, thanks. I suggest filing a bug in JIRA and please provide as much
information as possible (steps to reproduce would be ideal, but sometimes
that is hard to do). It does look like a Kafka bug.
Ismael
On Tue, May 10, 2016 at 2:45 PM, Ramanan, Buvana (Nokia - US) <
buvana.ramanan@nokia.com> wrote:
> Ismael,
>
> Version 0.9.0.1
>
> Do you have any idea how to prevent this from happening? Is it a Kafka
> issue?
>
> -Buvana
>
> -----Original Message-----
> From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT
> Ismael Juma
> Sent: Monday, May 09, 2016 8:02 PM
> To: users@kafka.apache.org
> Subject: Re: ERROR Processor got uncaught exception.
> (kafka.network.Processor)
>
> Hi Buvana,
>
> What is the version of the broker? It looks like the state of connection
> counts for quotas became inconsistent, that is, a given client ip address
> has no connections according to the internal state, but a disconnection for
> that ip happened.
>
> Ismael
>
> On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) <
> buvana.ramanan@nokia.com> wrote:
>
> > Hello,
> >
> > We are trying to understand the following error in one of the brokers in
> > our cluster. It's a 3 broker cluster (all 3 servers are identical -
> Intel
> > Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), serving
> > about 8000 topics, single partitioned & replication factor = 3. Each
> topic
> > gets data at a low rate - 200 bytes per sec. Cluster is managed by 3
> > server ZK cluster (these servers are different from Kafka broker
> servers).
> > All 6 servers are connected via 10G switch. Producers run from external
> > servers.
> >
> > Leaders are balanced across the topics.
> >
> > This error message occurs repeatedly in one of the servers. Between the
> > hours of 10:30am and 1:30pm today, there were about 10 Million such
> > occurrences.
> >
> > We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I can
> > share the current settings if needed.
> >
> > Please let me know what is causing this error and a way to overcome it.
> >
> > Regards,
> > Buvana
> >
> > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > (kafka.network.Processor)
> > java.lang.IllegalArgumentException: Attempted to decrease connection
> count
> > for address with no connections, address: /X.Y.Z.144 (actual network
> > address masked)
> > at
> > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > at
> > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > at
> > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > at
> > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > at
> > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > at kafka.network.Processor.run(SocketServer.scala:445)
> > at java.lang.Thread.run(Thread.java:745)
> > [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> > (kafka.network.Processor)
> > java.lang.IllegalArgumentException: Attempted to decrease connection
> count
> > for address with no connections, address: /X.Y.Z.144
> > at
> > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > at
> > kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> > at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> > at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> > at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> > at
> > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> > at
> > kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> > at
> > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> > at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> > at kafka.network.Processor.run(SocketServer.scala:445)
> > at java.lang.Thread.run(Thread.java:745)
> >
>
RE: ERROR Processor got uncaught exception.
(kafka.network.Processor)
Posted by "Ramanan, Buvana (Nokia - US)" <bu...@nokia.com>.
Ismael,
Version 0.9.0.1
Do you have any idea how to prevent this from happening? Is it a Kafka issue?
-Buvana
-----Original Message-----
From: ismaelj@gmail.com [mailto:ismaelj@gmail.com] On Behalf Of EXT Ismael Juma
Sent: Monday, May 09, 2016 8:02 PM
To: users@kafka.apache.org
Subject: Re: ERROR Processor got uncaught exception. (kafka.network.Processor)
Hi Buvana,
What is the version of the broker? It looks like the state of connection
counts for quotas became inconsistent, that is, a given client ip address
has no connections according to the internal state, but a disconnection for
that ip happened.
Ismael
On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) <
buvana.ramanan@nokia.com> wrote:
> Hello,
>
> We are trying to understand the following error in one of the brokers in
> our cluster. It's a 3 broker cluster (all 3 servers are identical - Intel
> Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), serving
> about 8000 topics, single partitioned & replication factor = 3. Each topic
> gets data at a low rate - 200 bytes per sec. Cluster is managed by 3
> server ZK cluster (these servers are different from Kafka broker servers).
> All 6 servers are connected via 10G switch. Producers run from external
> servers.
>
> Leaders are balanced across the topics.
>
> This error message occurs repeatedly in one of the servers. Between the
> hours of 10:30am and 1:30pm today, there were about 10 Million such
> occurrences.
>
> We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I can
> share the current settings if needed.
>
> Please let me know what is causing this error and a way to overcome it.
>
> Regards,
> Buvana
>
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count
> for address with no connections, address: /X.Y.Z.144 (actual network
> address masked)
> at
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:445)
> at java.lang.Thread.run(Thread.java:745)
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count
> for address with no connections, address: /X.Y.Z.144
> at
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:445)
> at java.lang.Thread.run(Thread.java:745)
>
Re: ERROR Processor got uncaught exception. (kafka.network.Processor)
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Buvana,
What is the version of the broker? It looks like the state of connection
counts for quotas became inconsistent, that is, a given client ip address
has no connections according to the internal state, but a disconnection for
that ip happened.
Ismael
On Tue, May 10, 2016 at 12:14 AM, Ramanan, Buvana (Nokia - US) <
buvana.ramanan@nokia.com> wrote:
> Hello,
>
> We are trying to understand the following error in one of the brokers in
> our cluster. It's a 3 broker cluster (all 3 servers are identical - Intel
> Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), serving
> about 8000 topics, single partitioned & replication factor = 3. Each topic
> gets data at a low rate - 200 bytes per sec. Cluster is managed by 3
> server ZK cluster (these servers are different from Kafka broker servers).
> All 6 servers are connected via 10G switch. Producers run from external
> servers.
>
> Leaders are balanced across the topics.
>
> This error message occurs repeatedly in one of the servers. Between the
> hours of 10:30am and 1:30pm today, there were about 10 Million such
> occurrences.
>
> We tuned Linux network params (somaxconn, rmem_max, wmem_max, etc). I can
> share the current settings if needed.
>
> Please let me know what is causing this error and a way to overcome it.
>
> Regards,
> Buvana
>
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count
> for address with no connections, address: /X.Y.Z.144 (actual network
> address masked)
> at
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:445)
> at java.lang.Thread.run(Thread.java:745)
> [2016-05-09 10:38:43,932] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.IllegalArgumentException: Attempted to decrease connection count
> for address with no connections, address: /X.Y.Z.144
> at
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at
> kafka.network.ConnectionQuotas$$anonfun$9.apply(SocketServer.scala:565)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at kafka.network.ConnectionQuotas.dec(SocketServer.scala:564)
> at
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:450)
> at
> kafka.network.Processor$$anonfun$run$13.apply(SocketServer.scala:445)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:445)
> at java.lang.Thread.run(Thread.java:745)
>