You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sa Li <sa...@gmail.com> on 2015/01/07 22:49:01 UTC
connection error among nodes
Hi, Experts
Our cluster is a 3 nodes cluster, I simply test producer locally, see
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test-rep-three 1000000 3000 -1 acks=1 bootstrap.servers=10.100.98.100:9092
buffer.memory=67108864 batch.size=8196
But I got such error, I do think this is critical issue, it just temporally
lose the connection and get back, what is the reason for this?
[2015-01-07 21:44:14,180] WARN Error in I/O with voluminous-mass.master/
10.100.98.101 (org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:745)
[2015-01-07 21:44:14,190] WARN Error in I/O with voluminous-mass.master/
10.100.98.101 (org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:745)
[2015-01-07 21:44:14,200] WARN Error in I/O with voluminous-mass.master/
10.100.98.101 (org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:745)
[2015-01-07 21:44:14,210] WARN Error in I/O with voluminous-mass.master/
10.100.98.101 (org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:745)
[2015-01-07 21:44:14,220] WARN Error in I/O with voluminous-mass.master/
10.100.98.101 (org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:745)
[2015-01-07 21:44:14,230] WARN Error in I/O with voluminous-mass.master/
10.100.98.101 (org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:745)
[2015-01-07 21:44:14,240] WARN Error in I/O with voluminous-mass.master/
10.100.98.101 (org.apache.kafka.common.network.Selector)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
at java.lang.Thread.run(Thread.java:745)
80 records sent, 13.6 records/sec (0.04 MB/sec), 31113.8 ms avg latency,
46912.0 max latency.
116695 records sent, 23339.0 records/sec (66.77 MB/sec), 5157.1 ms avg
latency, 46946.0 max latency.
thanks
--
Alec Li
Re: connection error among nodes
Posted by Jun Rao <ju...@confluent.io>.
1. bootstrap.servers is only used for issuing the very first metadata
request. Once the leader of a topic/partition is found, the producer will
talk to the leader brokers directly.
2. Do you see any disconnects in the broker log?
Thanks,
Jun
On Mon, Jan 19, 2015 at 11:05 AM, Sa Li <sa...@gmail.com> wrote:
> Hello, Jun
>
> I run such command:
>
> bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
> test-rep-three 1000000 3000 -1 acks=-1 bootstrap.servers=
> 10.100.98.100:9092,
> 10.100.98.101:9092 <http://10.100.98.100:9092/>, 10.100.98.102:9092
> <http://10.100.98.100:9092/> buffer.memory=67108864 batch.size=8196
>
> This is the perf test to producing data to different hosts. I have such
> questions:
> 1. If I use bootstrap.servers=10.100.98.100:9092, 10.100.98.101:9092
> <http://10.100.98.100:9092/>, 10.100.98.102:9092
> <http://10.100.98.100:9092/>, does this mean I send data to each of
> servers
> through 9092? However, if I set bootstrap.servers=10.100.98.100:9092 only,
> the producer will send data to 100, while replicate the data through other
> ports?
>
> here is the netstate I saw:
> netstat -plantue | egrep -i '.98.101|.98.102'
> tcp 0 0 10.100.98.100:37512 10.100.98.102:22
> ESTABLISHED 1004 7516316 5862/ssh
> tcp6 0 0 10.100.98.100:9092 10.100.98.102:56819
> ESTABLISHED 0 371522 3852/java
> tcp6 0 0 10.100.98.100:3888 10.100.98.101:40052
> ESTABLISHED 0 27514 1793/java
> tcp6 0 202 10.100.98.100:53592 10.100.98.101:9092
> ESTABLISHED 0 497715 3852/java
> tcp6 0 0 10.100.98.100:3888 10.100.98.102:32837
> ESTABLISHED 0 21701 1793/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.101:51053
> ESTABLISHED 0 371526 3852/java
> tcp6 0 270 10.100.98.100:53591 10.100.98.101:9092
> ESTABLISHED 0 497713 3852/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.102:57554
> ESTABLISHED 0 7491796 3852/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.101:51055
> ESTABLISHED 0 601277 3852/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.102:56824
> ESTABLISHED 0 614795 3852/java
> tcp6 0 0 10.100.98.100:48226 10.100.98.102:9092
> ESTABLISHED 0 3659949 3852/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.101:51054
> ESTABLISHED 0 601275 3852/java
> tcp6 0 0 10.100.98.100:48225 10.100.98.102:9092
> ESTABLISHED 0 3803556 3852/java
> tcp6 0 0 10.100.98.100:53593 10.100.98.101:9092
> ESTABLISHED 0 638462 3852/java
> tcp6 0 236 10.100.98.100:48228 10.100.98.102:9092
> ESTABLISHED 0 3936260 3852/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.102:56827
> ESTABLISHED 0 601276 3852/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.101:51052
> ESTABLISHED 0 614796 3852/java
> tcp6 0 230 10.100.98.100:53594 10.100.98.101:9092
> ESTABLISHED 0 637547 3852/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.102:56826
> ESTABLISHED 0 601274 3852/java
> tcp6 0 0 10.100.98.100:2181 10.100.98.102:34162
> ESTABLISHED 0 7491795 1793/java
> tcp6 0 230 10.100.98.100:48227 10.100.98.102:9092
> ESTABLISHED 0 3121735 3852/java
> tcp6 0 0 10.100.98.100:9092 10.100.98.102:56825
> ESTABLISHED 0 497716 3852/java
>
> 2. I still see the error of "server disconnect" during the perf test, but
> back to normal and print the results like
>
> 11212 records sent, 2239.7 records/sec (6.41 MB/sec), 7287.1 ms avg
> latency, 15005.0 max latency.
> 11620 records sent, 2313.8 records/sec (6.62 MB/sec), 7248.9 ms avg
> latency, 14807.0 max latency.
> 11522 records sent, 2293.4 records/sec (6.56 MB/sec), 7110.9 ms avg
> latency, 14551.0 max latency.
> 11058 records sent, 2200.2 records/sec (6.29 MB/sec), 7176.8 ms avg
> latency, 14774.0 max latency.
>
> But by monitoring the connection between nodes, we didn't see any
> disconnections, but we are not sure if the kafka servers disconnection took
> place or not, how to check it up?
>
> thanks
>
> AL
>
>
> On Sun, Jan 18, 2015 at 10:21 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Any issue with the network?
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Jan 7, 2015 at 1:59 PM, Sa Li <sa...@gmail.com> wrote:
> >
> > > Things bother me, sometimes, the errors won't pop out, sometimes it
> > comes,
> > > why?
> > >
> > > On Wed, Jan 7, 2015 at 1:49 PM, Sa Li <sa...@gmail.com> wrote:
> > >
> > > >
> > > > Hi, Experts
> > > >
> > > > Our cluster is a 3 nodes cluster, I simply test producer locally, see
> > > >
> > > > bin/kafka-run-class.sh
> > org.apache.kafka.clients.tools.ProducerPerformance
> > > > test-rep-three 1000000 3000 -1 acks=1 bootstrap.servers=
> > > 10.100.98.100:9092
> > > > buffer.memory=67108864 batch.size=8196
> > > >
> > > > But I got such error, I do think this is critical issue, it just
> > > > temporally lose the connection and get back, what is the reason for
> > this?
> > > >
> > > > [2015-01-07 21:44:14,180] WARN Error in I/O with
> > voluminous-mass.master/
> > > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > > at
> > > >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > > at
> > > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > > at
> > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > [2015-01-07 21:44:14,190] WARN Error in I/O with
> > voluminous-mass.master/
> > > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > > at
> > > >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > > at
> > > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > > at
> > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > [2015-01-07 21:44:14,200] WARN Error in I/O with
> > voluminous-mass.master/
> > > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > > at
> > > >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > > at
> > > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > > at
> > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > [2015-01-07 21:44:14,210] WARN Error in I/O with
> > voluminous-mass.master/
> > > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > > at
> > > >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > > at
> > > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > > at
> > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > [2015-01-07 21:44:14,220] WARN Error in I/O with
> > voluminous-mass.master/
> > > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > > at
> > > >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > > at
> > > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > > at
> > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > [2015-01-07 21:44:14,230] WARN Error in I/O with
> > voluminous-mass.master/
> > > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > > at
> > > >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > > at
> > > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > > at
> > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > [2015-01-07 21:44:14,240] WARN Error in I/O with
> > voluminous-mass.master/
> > > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > > java.net.ConnectException: Connection refused
> > > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > > at
> > > >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > > at
> > > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > > at
> > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > > at
> > > >
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > > at java.lang.Thread.run(Thread.java:745)
> > > > 80 records sent, 13.6 records/sec (0.04 MB/sec), 31113.8 ms avg
> > latency,
> > > > 46912.0 max latency.
> > > > 116695 records sent, 23339.0 records/sec (66.77 MB/sec), 5157.1 ms
> avg
> > > > latency, 46946.0 max latency.
> > > >
> > > >
> > > > thanks
> > > >
> > > > --
> > > >
> > > > Alec Li
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Alec Li
> > >
> >
>
>
>
> --
>
> Alec Li
>
Re: connection error among nodes
Posted by Sa Li <sa...@gmail.com>.
Hello, Jun
I run such command:
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
test-rep-three 1000000 3000 -1 acks=-1 bootstrap.servers=10.100.98.100:9092,
10.100.98.101:9092 <http://10.100.98.100:9092/>, 10.100.98.102:9092
<http://10.100.98.100:9092/> buffer.memory=67108864 batch.size=8196
This is the perf test to producing data to different hosts. I have such
questions:
1. If I use bootstrap.servers=10.100.98.100:9092, 10.100.98.101:9092
<http://10.100.98.100:9092/>, 10.100.98.102:9092
<http://10.100.98.100:9092/>, does this mean I send data to each of servers
through 9092? However, if I set bootstrap.servers=10.100.98.100:9092 only,
the producer will send data to 100, while replicate the data through other
ports?
here is the netstate I saw:
netstat -plantue | egrep -i '.98.101|.98.102'
tcp 0 0 10.100.98.100:37512 10.100.98.102:22
ESTABLISHED 1004 7516316 5862/ssh
tcp6 0 0 10.100.98.100:9092 10.100.98.102:56819
ESTABLISHED 0 371522 3852/java
tcp6 0 0 10.100.98.100:3888 10.100.98.101:40052
ESTABLISHED 0 27514 1793/java
tcp6 0 202 10.100.98.100:53592 10.100.98.101:9092
ESTABLISHED 0 497715 3852/java
tcp6 0 0 10.100.98.100:3888 10.100.98.102:32837
ESTABLISHED 0 21701 1793/java
tcp6 0 0 10.100.98.100:9092 10.100.98.101:51053
ESTABLISHED 0 371526 3852/java
tcp6 0 270 10.100.98.100:53591 10.100.98.101:9092
ESTABLISHED 0 497713 3852/java
tcp6 0 0 10.100.98.100:9092 10.100.98.102:57554
ESTABLISHED 0 7491796 3852/java
tcp6 0 0 10.100.98.100:9092 10.100.98.101:51055
ESTABLISHED 0 601277 3852/java
tcp6 0 0 10.100.98.100:9092 10.100.98.102:56824
ESTABLISHED 0 614795 3852/java
tcp6 0 0 10.100.98.100:48226 10.100.98.102:9092
ESTABLISHED 0 3659949 3852/java
tcp6 0 0 10.100.98.100:9092 10.100.98.101:51054
ESTABLISHED 0 601275 3852/java
tcp6 0 0 10.100.98.100:48225 10.100.98.102:9092
ESTABLISHED 0 3803556 3852/java
tcp6 0 0 10.100.98.100:53593 10.100.98.101:9092
ESTABLISHED 0 638462 3852/java
tcp6 0 236 10.100.98.100:48228 10.100.98.102:9092
ESTABLISHED 0 3936260 3852/java
tcp6 0 0 10.100.98.100:9092 10.100.98.102:56827
ESTABLISHED 0 601276 3852/java
tcp6 0 0 10.100.98.100:9092 10.100.98.101:51052
ESTABLISHED 0 614796 3852/java
tcp6 0 230 10.100.98.100:53594 10.100.98.101:9092
ESTABLISHED 0 637547 3852/java
tcp6 0 0 10.100.98.100:9092 10.100.98.102:56826
ESTABLISHED 0 601274 3852/java
tcp6 0 0 10.100.98.100:2181 10.100.98.102:34162
ESTABLISHED 0 7491795 1793/java
tcp6 0 230 10.100.98.100:48227 10.100.98.102:9092
ESTABLISHED 0 3121735 3852/java
tcp6 0 0 10.100.98.100:9092 10.100.98.102:56825
ESTABLISHED 0 497716 3852/java
2. I still see the error of "server disconnect" during the perf test, but
back to normal and print the results like
11212 records sent, 2239.7 records/sec (6.41 MB/sec), 7287.1 ms avg
latency, 15005.0 max latency.
11620 records sent, 2313.8 records/sec (6.62 MB/sec), 7248.9 ms avg
latency, 14807.0 max latency.
11522 records sent, 2293.4 records/sec (6.56 MB/sec), 7110.9 ms avg
latency, 14551.0 max latency.
11058 records sent, 2200.2 records/sec (6.29 MB/sec), 7176.8 ms avg
latency, 14774.0 max latency.
But by monitoring the connection between nodes, we didn't see any
disconnections, but we are not sure if the kafka servers disconnection took
place or not, how to check it up?
thanks
AL
On Sun, Jan 18, 2015 at 10:21 AM, Jun Rao <ju...@confluent.io> wrote:
> Any issue with the network?
>
> Thanks,
>
> Jun
>
> On Wed, Jan 7, 2015 at 1:59 PM, Sa Li <sa...@gmail.com> wrote:
>
> > Things bother me, sometimes, the errors won't pop out, sometimes it
> comes,
> > why?
> >
> > On Wed, Jan 7, 2015 at 1:49 PM, Sa Li <sa...@gmail.com> wrote:
> >
> > >
> > > Hi, Experts
> > >
> > > Our cluster is a 3 nodes cluster, I simply test producer locally, see
> > >
> > > bin/kafka-run-class.sh
> org.apache.kafka.clients.tools.ProducerPerformance
> > > test-rep-three 1000000 3000 -1 acks=1 bootstrap.servers=
> > 10.100.98.100:9092
> > > buffer.memory=67108864 batch.size=8196
> > >
> > > But I got such error, I do think this is critical issue, it just
> > > temporally lose the connection and get back, what is the reason for
> this?
> > >
> > > [2015-01-07 21:44:14,180] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,190] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,200] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,210] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,220] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,230] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > at java.lang.Thread.run(Thread.java:745)
> > > [2015-01-07 21:44:14,240] WARN Error in I/O with
> voluminous-mass.master/
> > > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > > java.net.ConnectException: Connection refused
> > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > at
> > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > > at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > > at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > > at
> > > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > > at java.lang.Thread.run(Thread.java:745)
> > > 80 records sent, 13.6 records/sec (0.04 MB/sec), 31113.8 ms avg
> latency,
> > > 46912.0 max latency.
> > > 116695 records sent, 23339.0 records/sec (66.77 MB/sec), 5157.1 ms avg
> > > latency, 46946.0 max latency.
> > >
> > >
> > > thanks
> > >
> > > --
> > >
> > > Alec Li
> > >
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
--
Alec Li
Re: connection error among nodes
Posted by Jun Rao <ju...@confluent.io>.
Any issue with the network?
Thanks,
Jun
On Wed, Jan 7, 2015 at 1:59 PM, Sa Li <sa...@gmail.com> wrote:
> Things bother me, sometimes, the errors won't pop out, sometimes it comes,
> why?
>
> On Wed, Jan 7, 2015 at 1:49 PM, Sa Li <sa...@gmail.com> wrote:
>
> >
> > Hi, Experts
> >
> > Our cluster is a 3 nodes cluster, I simply test producer locally, see
> >
> > bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
> > test-rep-three 1000000 3000 -1 acks=1 bootstrap.servers=
> 10.100.98.100:9092
> > buffer.memory=67108864 batch.size=8196
> >
> > But I got such error, I do think this is critical issue, it just
> > temporally lose the connection and get back, what is the reason for this?
> >
> > [2015-01-07 21:44:14,180] WARN Error in I/O with voluminous-mass.master/
> > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > at
> org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > at java.lang.Thread.run(Thread.java:745)
> > [2015-01-07 21:44:14,190] WARN Error in I/O with voluminous-mass.master/
> > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > at
> org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > at java.lang.Thread.run(Thread.java:745)
> > [2015-01-07 21:44:14,200] WARN Error in I/O with voluminous-mass.master/
> > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > at
> org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > at java.lang.Thread.run(Thread.java:745)
> > [2015-01-07 21:44:14,210] WARN Error in I/O with voluminous-mass.master/
> > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > at
> org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > at java.lang.Thread.run(Thread.java:745)
> > [2015-01-07 21:44:14,220] WARN Error in I/O with voluminous-mass.master/
> > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > at
> org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > at java.lang.Thread.run(Thread.java:745)
> > [2015-01-07 21:44:14,230] WARN Error in I/O with voluminous-mass.master/
> > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > at
> org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > at java.lang.Thread.run(Thread.java:745)
> > [2015-01-07 21:44:14,240] WARN Error in I/O with voluminous-mass.master/
> > 10.100.98.101 (org.apache.kafka.common.network.Selector)
> > java.net.ConnectException: Connection refused
> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> > at
> org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> > at
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> > at
> > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> > at java.lang.Thread.run(Thread.java:745)
> > 80 records sent, 13.6 records/sec (0.04 MB/sec), 31113.8 ms avg latency,
> > 46912.0 max latency.
> > 116695 records sent, 23339.0 records/sec (66.77 MB/sec), 5157.1 ms avg
> > latency, 46946.0 max latency.
> >
> >
> > thanks
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
>
> Alec Li
>
Re: connection error among nodes
Posted by Sa Li <sa...@gmail.com>.
Things bother me, sometimes, the errors won't pop out, sometimes it comes,
why?
On Wed, Jan 7, 2015 at 1:49 PM, Sa Li <sa...@gmail.com> wrote:
>
> Hi, Experts
>
> Our cluster is a 3 nodes cluster, I simply test producer locally, see
>
> bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
> test-rep-three 1000000 3000 -1 acks=1 bootstrap.servers=10.100.98.100:9092
> buffer.memory=67108864 batch.size=8196
>
> But I got such error, I do think this is critical issue, it just
> temporally lose the connection and get back, what is the reason for this?
>
> [2015-01-07 21:44:14,180] WARN Error in I/O with voluminous-mass.master/
> 10.100.98.101 (org.apache.kafka.common.network.Selector)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> at java.lang.Thread.run(Thread.java:745)
> [2015-01-07 21:44:14,190] WARN Error in I/O with voluminous-mass.master/
> 10.100.98.101 (org.apache.kafka.common.network.Selector)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> at java.lang.Thread.run(Thread.java:745)
> [2015-01-07 21:44:14,200] WARN Error in I/O with voluminous-mass.master/
> 10.100.98.101 (org.apache.kafka.common.network.Selector)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> at java.lang.Thread.run(Thread.java:745)
> [2015-01-07 21:44:14,210] WARN Error in I/O with voluminous-mass.master/
> 10.100.98.101 (org.apache.kafka.common.network.Selector)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> at java.lang.Thread.run(Thread.java:745)
> [2015-01-07 21:44:14,220] WARN Error in I/O with voluminous-mass.master/
> 10.100.98.101 (org.apache.kafka.common.network.Selector)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> at java.lang.Thread.run(Thread.java:745)
> [2015-01-07 21:44:14,230] WARN Error in I/O with voluminous-mass.master/
> 10.100.98.101 (org.apache.kafka.common.network.Selector)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> at java.lang.Thread.run(Thread.java:745)
> [2015-01-07 21:44:14,240] WARN Error in I/O with voluminous-mass.master/
> 10.100.98.101 (org.apache.kafka.common.network.Selector)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:232)
> at
> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:191)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:184)
> at
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:115)
> at java.lang.Thread.run(Thread.java:745)
> 80 records sent, 13.6 records/sec (0.04 MB/sec), 31113.8 ms avg latency,
> 46912.0 max latency.
> 116695 records sent, 23339.0 records/sec (66.77 MB/sec), 5157.1 ms avg
> latency, 46946.0 max latency.
>
>
> thanks
>
> --
>
> Alec Li
>
--
Alec Li