You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Alexius Ludeman <le...@lexinator.com> on 2012/01/18 02:00:43 UTC
chatty error: Exception causing close of session 0x0 due to
java.io.IOException: Len error 1835955314
hi,
I am getting the error from all 5 zookeeperd, "Exception causing close of
session 0x0 due to java.io.IOException: Len error 1835955314" every
microsecond(?).
I have a 5 node cluster on ubuntu maverick in aws. I've backported
zookeeperd and now am running 3.3.4+dfsg1-1. My apps are
using zookeeper-3.3.3.jar, but within the next 3-4 weeks I will be
upgrading to zookeeper-3.3.4.jar. This problem also existed with
zookeeperd on 3.3.3, in hopes of upgrading would solve my problem.
I have stopped the whole world, rm -rf /var/lib/zookeeper and restarted
without any change to this log message. I have several other environments
without this error showing, same versions of client and server but only 3
zookeeperd, and much fewer clients.
Our use of zookeeper is: appName is a key and the value is a server:port.
After the app reads in its required appName(s), it puts watches on these
appNames looking for changes. So we have a bunch of apps with many
watches. The client count is below.
I would be grateful if anyone has some insight on how to debug further.
thanks
lex
zookeeper-0.p2:/var/log/zookeeper/zookeeper.log:
2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
connection from /127.0.0.1:38653
2012-01-17 22:23:59,097 - INFO [Thread-2309197:NIOServerCnxn@1435][] -
Closed socket connection for client /127.0.0.1:38652 (no session
established for client)
2012-01-17 22:23:59,097 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
session 0x0 due to java.io.IOException: Len error 1835955314
2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection for
client /127.0.0.1:38653 (no session established for client)
2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
connection from /127.0.0.1:38654
2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command from /
127.0.0.1:38654
2012-01-17 22:23:59,098 - INFO
[Thread-2309198:NIOServerCnxn$StatCommand@1153][] - Stat command output
2012-01-17 22:23:59,098 - INFO [Thread-2309198:NIOServerCnxn@1435][] -
Closed socket connection for client /127.0.0.1:38654 (no session
established for client)
2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
connection from /127.0.0.1:38655
2012-01-17 22:23:59,099 - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
session 0x0 due to java.io.IOException: Len error 1835955314
2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection for
client /127.0.0.1:38655 (no session established for client)
2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
connection from /127.0.0.1:38656
2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command from /
127.0.0.1:38656
Connected to zookeeper-0.p2
envi
Environment:
zookeeper.version=3.3.4--1, built on 12/09/2011 00:03 GMT
host.name=zookeeper-0.p2.use.color.internal
java.version=1.6.0_20
java.vendor=Sun Microsystems Inc.
java.home=/usr/lib/jvm/java-6-openjdk/jre
java.class.path=/etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar
java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
java.io.tmpdir=/tmp
java.compiler=<NA>
os.name=Linux
os.arch=amd64
os.version=2.6.35-24-virtual
user.name=zookeeper
user.home=/var/lib/zookeeper
user.dir=/
Trying 10.118.145.79...Connected to zookeeper-0.p2
stats
Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
Clients:
/10.40.50.163:45526[1](queued=0,recved=2819,sent=2820)
/10.124.143.97:46512[0](queued=0,recved=1,sent=0)
/10.40.37.34:33308[1](queued=0,recved=2795,sent=2796)
/10.205.49.18:35339[1](queued=0,recved=2671,sent=2672)
/10.205.49.18:42419[1](queued=0,recved=2718,sent=2719)
/10.80.82.184:43535[1](queued=0,recved=2819,sent=2820)
/10.84.133.213:52533[1](queued=0,recved=2817,sent=2844)
Latency min/avg/max: 0/0/200
Received: 7330810
Sent: 7331408
Outstanding: 0
Zxid: 0x100014e31
Mode: follower
Node count: 197
Trying 10.34.37.214...Connected to zookeeper-1.p2
stat
Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
Clients:
/10.79.6.245:38198[1](queued=0,recved=2680,sent=2681)
/10.98.119.91:37301[1](queued=0,recved=2689,sent=2690)
/10.79.35.125:59744[1](queued=0,recved=2681,sent=2682)
/10.39.81.28:38047[1](queued=0,recved=2666,sent=2667)
/10.124.143.97:48969[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/81
Received: 4984011
Sent: 4984267
Outstanding: 0
Zxid: 0x100014e31
Mode: follower
Node count: 197
Trying 10.38.85.114...Connected to zookeeper-2.p2
stat
Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
Clients:
/10.84.133.213:51499[1](queued=0,recved=2655,sent=2656)
/10.124.143.97:39412[0](queued=0,recved=1,sent=0)
/10.39.81.28:56431[1](queued=0,recved=2715,sent=2716)
/10.204.183.195:40180[1](queued=0,recved=2664,sent=2665)
/10.32.51.156:59390[1](queued=0,recved=2634,sent=2635)
/10.79.6.245:42161[1](queued=0,recved=2731,sent=2732)
/10.118.14.138:40355[1](queued=0,recved=2643,sent=2644)
Latency min/avg/max: 0/0/33
Received: 6650278
Sent: 6651369
Outstanding: 0
Zxid: 0x100014e31
Mode: leader
Node count: 197
Trying 10.101.42.61...Connected to zookeeper-3.p2
stat
Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
Clients:
/10.204.230.212:57523[1](queued=0,recved=2719,sent=2720)
/10.85.19.23:52916[1](queued=0,recved=2836,sent=2837)
/10.79.35.125:54742[1](queued=0,recved=2734,sent=2735)
/10.124.143.97:60909[0](queued=0,recved=1,sent=0)
/10.204.230.212:57918[1](queued=0,recved=2674,sent=2675)
/10.100.90.101:35977[1](queued=0,recved=2680,sent=2681)
/10.98.119.91:42147[1](queued=0,recved=2745,sent=2746)
Latency min/avg/max: 0/0/123
Received: 8430215
Sent: 8430724
Outstanding: 0
Zxid: 0x100014e31
Mode: follower
Node count: 197
Trying 10.204.59.55...Connected to zookeeper-4.p2
stat
Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
Clients:
/10.100.90.101:34437[1](queued=0,recved=2728,sent=2729)
/10.204.183.195:49961[1](queued=0,recved=2706,sent=2707)
/10.124.143.97:34797[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/19
Received: 46410
Sent: 46411
Outstanding: 0
Zxid: 0x100014e31
Mode: follower
Node count: 197
Re: chatty error: Exception causing close of session 0x0 due to
java.io.IOException: Len error 1835955314
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Jan 18, 2012 at 6:00 AM, Alexius Ludeman <le...@lexinator.com> wrote:
> woot! that's some impressive remote detective work.
>
Pat gets the credit. I just converted to hex and remembered my dark past
secrets from the days when we wrote assembler by hand.
Re: chatty error: Exception causing close of session 0x0 due to
java.io.IOException: Len error 1835955314
Posted by Alexius Ludeman <le...@lexinator.com>.
woot! that's some impressive remote detective work.
Ganglia is the culprit here. It blindly tries "mntr" for every stat and
falls back to "stat". An easy remedy...
I guess I misspoke as this error is happening in all our staging
environments. I seem to remember not and did not revalidate before sending
my earlier email.
thanks for quick response and solving my issue,
lex
On Tue, Jan 17, 2012 at 9:11 PM, Patrick Hunt <ph...@apache.org> wrote:
> On Tue, Jan 17, 2012 at 6:19 PM, Ted Dunning <te...@gmail.com>
> wrote:
> > I think that Pat is right:
> >
> > 1835955314 = 6D6E7472 = 'mntr'
> >
>
> Nice. :-) "mntr" is new in 3.4, it's not available/known in 3.3
> http://zookeeper.apache.org/doc/r3.4.2/zookeeperAdmin.html#sc_zkCommands
>
> Patrick
>
> > On Wed, Jan 18, 2012 at 2:00 AM, Patrick Hunt <ph...@apache.org> wrote:
> >
> >> Hi Alexius, it looks like localhost (127.0.0.1) is making a socket
> >> connection and then sending a request with a supposed size of 1.8GB.
> >> (the first 4 bytes of each zk request are the length)
> >>
> >> It's likely that this is a "4letterword" attempt gone wrong (or some
> >> other host incorrectly pinging this port). Which application are you
> >> pointing at the ZK server's client port? Could this be an invalid
> >> monitoring request? (say someone trying to do a "stat" but spelling it
> >> wrong?)
> >>
> >> Patrick
> >>
> >> On Tue, Jan 17, 2012 at 5:00 PM, Alexius Ludeman <le...@lexinator.com>
> >> wrote:
> >> > hi,
> >> >
> >> > I am getting the error from all 5 zookeeperd, "Exception causing
> close of
> >> > session 0x0 due to java.io.IOException: Len error 1835955314" every
> >> > microsecond(?).
> >> >
> >> > I have a 5 node cluster on ubuntu maverick in aws. I've backported
> >> > zookeeperd and now am running 3.3.4+dfsg1-1. My apps are
> >> > using zookeeper-3.3.3.jar, but within the next 3-4 weeks I will be
> >> > upgrading to zookeeper-3.3.4.jar. This problem also existed with
> >> > zookeeperd on 3.3.3, in hopes of upgrading would solve my problem.
> >> >
> >> > I have stopped the whole world, rm -rf /var/lib/zookeeper and
> restarted
> >> > without any change to this log message. I have several other
> >> environments
> >> > without this error showing, same versions of client and server but
> only 3
> >> > zookeeperd, and much fewer clients.
> >> >
> >> > Our use of zookeeper is: appName is a key and the value is a
> server:port.
> >> > After the app reads in its required appName(s), it puts watches on
> these
> >> > appNames looking for changes. So we have a bunch of apps with many
> >> > watches. The client count is below.
> >> >
> >> > I would be grateful if anyone has some insight on how to debug
> further.
> >> >
> >> > thanks
> >> > lex
> >> >
> >> > zookeeper-0.p2:/var/log/zookeeper/zookeeper.log:
> >> >
> >> > 2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> >> > connection from /127.0.0.1:38653
> >> > 2012-01-17 22:23:59,097 - INFO [Thread-2309197:NIOServerCnxn@1435][]
> -
> >> > Closed socket connection for client /127.0.0.1:38652 (no session
> >> > established for client)
> >> > 2012-01-17 22:23:59,097 - WARN [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close
> of
> >> > session 0x0 due to java.io.IOException: Len error 1835955314
> >> > 2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection
> >> for
> >> > client /127.0.0.1:38653 (no session established for client)
> >> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> >> > connection from /127.0.0.1:38654
> >> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command
> >> from /
> >> > 127.0.0.1:38654
> >> > 2012-01-17 22:23:59,098 - INFO
> >> > [Thread-2309198:NIOServerCnxn$StatCommand@1153][] - Stat command
> output
> >> > 2012-01-17 22:23:59,098 - INFO [Thread-2309198:NIOServerCnxn@1435][]
> -
> >> > Closed socket connection for client /127.0.0.1:38654 (no session
> >> > established for client)
> >> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> >> > connection from /127.0.0.1:38655
> >> > 2012-01-17 22:23:59,099 - WARN [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close
> of
> >> > session 0x0 due to java.io.IOException: Len error 1835955314
> >> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection
> >> for
> >> > client /127.0.0.1:38655 (no session established for client)
> >> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> >> > connection from /127.0.0.1:38656
> >> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> >> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command
> >> from /
> >> > 127.0.0.1:38656
> >> >
> >> >
> >> > Connected to zookeeper-0.p2
> >> > envi
> >> >
> >> > Environment:
> >> > zookeeper.version=3.3.4--1, built on 12/09/2011 00:03 GMT
> >> > host.name=zookeeper-0.p2.use.color.internal
> >> > java.version=1.6.0_20
> >> > java.vendor=Sun Microsystems Inc.
> >> > java.home=/usr/lib/jvm/java-6-openjdk/jre
> >> >
> >>
> java.class.path=/etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar
> >> >
> >>
> java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
> >> > java.io.tmpdir=/tmp
> >> > java.compiler=<NA>
> >> > os.name=Linux
> >> > os.arch=amd64
> >> > os.version=2.6.35-24-virtual
> >> > user.name=zookeeper
> >> > user.home=/var/lib/zookeeper
> >> > user.dir=/
> >> >
> >> > Trying 10.118.145.79...Connected to zookeeper-0.p2
> >> > stats
> >> >
> >> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> >> > Clients:
> >> > /10.40.50.163:45526[1](queued=0,recved=2819,sent=2820)
> >> > /10.124.143.97:46512[0](queued=0,recved=1,sent=0)
> >> > /10.40.37.34:33308[1](queued=0,recved=2795,sent=2796)
> >> > /10.205.49.18:35339[1](queued=0,recved=2671,sent=2672)
> >> > /10.205.49.18:42419[1](queued=0,recved=2718,sent=2719)
> >> > /10.80.82.184:43535[1](queued=0,recved=2819,sent=2820)
> >> > /10.84.133.213:52533[1](queued=0,recved=2817,sent=2844)
> >> >
> >> > Latency min/avg/max: 0/0/200
> >> > Received: 7330810
> >> > Sent: 7331408
> >> > Outstanding: 0
> >> > Zxid: 0x100014e31
> >> > Mode: follower
> >> > Node count: 197
> >> >
> >> > Trying 10.34.37.214...Connected to zookeeper-1.p2
> >> > stat
> >> >
> >> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> >> > Clients:
> >> > /10.79.6.245:38198[1](queued=0,recved=2680,sent=2681)
> >> > /10.98.119.91:37301[1](queued=0,recved=2689,sent=2690)
> >> > /10.79.35.125:59744[1](queued=0,recved=2681,sent=2682)
> >> > /10.39.81.28:38047[1](queued=0,recved=2666,sent=2667)
> >> > /10.124.143.97:48969[0](queued=0,recved=1,sent=0)
> >> >
> >> > Latency min/avg/max: 0/0/81
> >> > Received: 4984011
> >> > Sent: 4984267
> >> > Outstanding: 0
> >> > Zxid: 0x100014e31
> >> > Mode: follower
> >> > Node count: 197
> >> >
> >> > Trying 10.38.85.114...Connected to zookeeper-2.p2
> >> >
> >> > stat
> >> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> >> > Clients:
> >> > /10.84.133.213:51499[1](queued=0,recved=2655,sent=2656)
> >> > /10.124.143.97:39412[0](queued=0,recved=1,sent=0)
> >> > /10.39.81.28:56431[1](queued=0,recved=2715,sent=2716)
> >> > /10.204.183.195:40180[1](queued=0,recved=2664,sent=2665)
> >> > /10.32.51.156:59390[1](queued=0,recved=2634,sent=2635)
> >> > /10.79.6.245:42161[1](queued=0,recved=2731,sent=2732)
> >> > /10.118.14.138:40355[1](queued=0,recved=2643,sent=2644)
> >> >
> >> > Latency min/avg/max: 0/0/33
> >> > Received: 6650278
> >> > Sent: 6651369
> >> > Outstanding: 0
> >> > Zxid: 0x100014e31
> >> > Mode: leader
> >> > Node count: 197
> >> >
> >> > Trying 10.101.42.61...Connected to zookeeper-3.p2
> >> >
> >> > stat
> >> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> >> > Clients:
> >> > /10.204.230.212:57523[1](queued=0,recved=2719,sent=2720)
> >> > /10.85.19.23:52916[1](queued=0,recved=2836,sent=2837)
> >> > /10.79.35.125:54742[1](queued=0,recved=2734,sent=2735)
> >> > /10.124.143.97:60909[0](queued=0,recved=1,sent=0)
> >> > /10.204.230.212:57918[1](queued=0,recved=2674,sent=2675)
> >> > /10.100.90.101:35977[1](queued=0,recved=2680,sent=2681)
> >> > /10.98.119.91:42147[1](queued=0,recved=2745,sent=2746)
> >> >
> >> > Latency min/avg/max: 0/0/123
> >> > Received: 8430215
> >> > Sent: 8430724
> >> > Outstanding: 0
> >> > Zxid: 0x100014e31
> >> > Mode: follower
> >> > Node count: 197
> >> >
> >> > Trying 10.204.59.55...Connected to zookeeper-4.p2
> >> >
> >> > stat
> >> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> >> > Clients:
> >> > /10.100.90.101:34437[1](queued=0,recved=2728,sent=2729)
> >> > /10.204.183.195:49961[1](queued=0,recved=2706,sent=2707)
> >> > /10.124.143.97:34797[0](queued=0,recved=1,sent=0)
> >> >
> >> > Latency min/avg/max: 0/0/19
> >> > Received: 46410
> >> > Sent: 46411
> >> > Outstanding: 0
> >> > Zxid: 0x100014e31
> >> > Mode: follower
> >> > Node count: 197
> >>
>
Re: chatty error: Exception causing close of session 0x0 due to
java.io.IOException: Len error 1835955314
Posted by Patrick Hunt <ph...@apache.org>.
On Tue, Jan 17, 2012 at 6:19 PM, Ted Dunning <te...@gmail.com> wrote:
> I think that Pat is right:
>
> 1835955314 = 6D6E7472 = 'mntr'
>
Nice. :-) "mntr" is new in 3.4, it's not available/known in 3.3
http://zookeeper.apache.org/doc/r3.4.2/zookeeperAdmin.html#sc_zkCommands
Patrick
> On Wed, Jan 18, 2012 at 2:00 AM, Patrick Hunt <ph...@apache.org> wrote:
>
>> Hi Alexius, it looks like localhost (127.0.0.1) is making a socket
>> connection and then sending a request with a supposed size of 1.8GB.
>> (the first 4 bytes of each zk request are the length)
>>
>> It's likely that this is a "4letterword" attempt gone wrong (or some
>> other host incorrectly pinging this port). Which application are you
>> pointing at the ZK server's client port? Could this be an invalid
>> monitoring request? (say someone trying to do a "stat" but spelling it
>> wrong?)
>>
>> Patrick
>>
>> On Tue, Jan 17, 2012 at 5:00 PM, Alexius Ludeman <le...@lexinator.com>
>> wrote:
>> > hi,
>> >
>> > I am getting the error from all 5 zookeeperd, "Exception causing close of
>> > session 0x0 due to java.io.IOException: Len error 1835955314" every
>> > microsecond(?).
>> >
>> > I have a 5 node cluster on ubuntu maverick in aws. I've backported
>> > zookeeperd and now am running 3.3.4+dfsg1-1. My apps are
>> > using zookeeper-3.3.3.jar, but within the next 3-4 weeks I will be
>> > upgrading to zookeeper-3.3.4.jar. This problem also existed with
>> > zookeeperd on 3.3.3, in hopes of upgrading would solve my problem.
>> >
>> > I have stopped the whole world, rm -rf /var/lib/zookeeper and restarted
>> > without any change to this log message. I have several other
>> environments
>> > without this error showing, same versions of client and server but only 3
>> > zookeeperd, and much fewer clients.
>> >
>> > Our use of zookeeper is: appName is a key and the value is a server:port.
>> > After the app reads in its required appName(s), it puts watches on these
>> > appNames looking for changes. So we have a bunch of apps with many
>> > watches. The client count is below.
>> >
>> > I would be grateful if anyone has some insight on how to debug further.
>> >
>> > thanks
>> > lex
>> >
>> > zookeeper-0.p2:/var/log/zookeeper/zookeeper.log:
>> >
>> > 2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
>> > connection from /127.0.0.1:38653
>> > 2012-01-17 22:23:59,097 - INFO [Thread-2309197:NIOServerCnxn@1435][] -
>> > Closed socket connection for client /127.0.0.1:38652 (no session
>> > established for client)
>> > 2012-01-17 22:23:59,097 - WARN [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
>> > session 0x0 due to java.io.IOException: Len error 1835955314
>> > 2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection
>> for
>> > client /127.0.0.1:38653 (no session established for client)
>> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
>> > connection from /127.0.0.1:38654
>> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command
>> from /
>> > 127.0.0.1:38654
>> > 2012-01-17 22:23:59,098 - INFO
>> > [Thread-2309198:NIOServerCnxn$StatCommand@1153][] - Stat command output
>> > 2012-01-17 22:23:59,098 - INFO [Thread-2309198:NIOServerCnxn@1435][] -
>> > Closed socket connection for client /127.0.0.1:38654 (no session
>> > established for client)
>> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
>> > connection from /127.0.0.1:38655
>> > 2012-01-17 22:23:59,099 - WARN [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
>> > session 0x0 due to java.io.IOException: Len error 1835955314
>> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection
>> for
>> > client /127.0.0.1:38655 (no session established for client)
>> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
>> > connection from /127.0.0.1:38656
>> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
>> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command
>> from /
>> > 127.0.0.1:38656
>> >
>> >
>> > Connected to zookeeper-0.p2
>> > envi
>> >
>> > Environment:
>> > zookeeper.version=3.3.4--1, built on 12/09/2011 00:03 GMT
>> > host.name=zookeeper-0.p2.use.color.internal
>> > java.version=1.6.0_20
>> > java.vendor=Sun Microsystems Inc.
>> > java.home=/usr/lib/jvm/java-6-openjdk/jre
>> >
>> java.class.path=/etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar
>> >
>> java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
>> > java.io.tmpdir=/tmp
>> > java.compiler=<NA>
>> > os.name=Linux
>> > os.arch=amd64
>> > os.version=2.6.35-24-virtual
>> > user.name=zookeeper
>> > user.home=/var/lib/zookeeper
>> > user.dir=/
>> >
>> > Trying 10.118.145.79...Connected to zookeeper-0.p2
>> > stats
>> >
>> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
>> > Clients:
>> > /10.40.50.163:45526[1](queued=0,recved=2819,sent=2820)
>> > /10.124.143.97:46512[0](queued=0,recved=1,sent=0)
>> > /10.40.37.34:33308[1](queued=0,recved=2795,sent=2796)
>> > /10.205.49.18:35339[1](queued=0,recved=2671,sent=2672)
>> > /10.205.49.18:42419[1](queued=0,recved=2718,sent=2719)
>> > /10.80.82.184:43535[1](queued=0,recved=2819,sent=2820)
>> > /10.84.133.213:52533[1](queued=0,recved=2817,sent=2844)
>> >
>> > Latency min/avg/max: 0/0/200
>> > Received: 7330810
>> > Sent: 7331408
>> > Outstanding: 0
>> > Zxid: 0x100014e31
>> > Mode: follower
>> > Node count: 197
>> >
>> > Trying 10.34.37.214...Connected to zookeeper-1.p2
>> > stat
>> >
>> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
>> > Clients:
>> > /10.79.6.245:38198[1](queued=0,recved=2680,sent=2681)
>> > /10.98.119.91:37301[1](queued=0,recved=2689,sent=2690)
>> > /10.79.35.125:59744[1](queued=0,recved=2681,sent=2682)
>> > /10.39.81.28:38047[1](queued=0,recved=2666,sent=2667)
>> > /10.124.143.97:48969[0](queued=0,recved=1,sent=0)
>> >
>> > Latency min/avg/max: 0/0/81
>> > Received: 4984011
>> > Sent: 4984267
>> > Outstanding: 0
>> > Zxid: 0x100014e31
>> > Mode: follower
>> > Node count: 197
>> >
>> > Trying 10.38.85.114...Connected to zookeeper-2.p2
>> >
>> > stat
>> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
>> > Clients:
>> > /10.84.133.213:51499[1](queued=0,recved=2655,sent=2656)
>> > /10.124.143.97:39412[0](queued=0,recved=1,sent=0)
>> > /10.39.81.28:56431[1](queued=0,recved=2715,sent=2716)
>> > /10.204.183.195:40180[1](queued=0,recved=2664,sent=2665)
>> > /10.32.51.156:59390[1](queued=0,recved=2634,sent=2635)
>> > /10.79.6.245:42161[1](queued=0,recved=2731,sent=2732)
>> > /10.118.14.138:40355[1](queued=0,recved=2643,sent=2644)
>> >
>> > Latency min/avg/max: 0/0/33
>> > Received: 6650278
>> > Sent: 6651369
>> > Outstanding: 0
>> > Zxid: 0x100014e31
>> > Mode: leader
>> > Node count: 197
>> >
>> > Trying 10.101.42.61...Connected to zookeeper-3.p2
>> >
>> > stat
>> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
>> > Clients:
>> > /10.204.230.212:57523[1](queued=0,recved=2719,sent=2720)
>> > /10.85.19.23:52916[1](queued=0,recved=2836,sent=2837)
>> > /10.79.35.125:54742[1](queued=0,recved=2734,sent=2735)
>> > /10.124.143.97:60909[0](queued=0,recved=1,sent=0)
>> > /10.204.230.212:57918[1](queued=0,recved=2674,sent=2675)
>> > /10.100.90.101:35977[1](queued=0,recved=2680,sent=2681)
>> > /10.98.119.91:42147[1](queued=0,recved=2745,sent=2746)
>> >
>> > Latency min/avg/max: 0/0/123
>> > Received: 8430215
>> > Sent: 8430724
>> > Outstanding: 0
>> > Zxid: 0x100014e31
>> > Mode: follower
>> > Node count: 197
>> >
>> > Trying 10.204.59.55...Connected to zookeeper-4.p2
>> >
>> > stat
>> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
>> > Clients:
>> > /10.100.90.101:34437[1](queued=0,recved=2728,sent=2729)
>> > /10.204.183.195:49961[1](queued=0,recved=2706,sent=2707)
>> > /10.124.143.97:34797[0](queued=0,recved=1,sent=0)
>> >
>> > Latency min/avg/max: 0/0/19
>> > Received: 46410
>> > Sent: 46411
>> > Outstanding: 0
>> > Zxid: 0x100014e31
>> > Mode: follower
>> > Node count: 197
>>
Re: chatty error: Exception causing close of session 0x0 due to
java.io.IOException: Len error 1835955314
Posted by Ted Dunning <te...@gmail.com>.
I think that Pat is right:
1835955314 = 6D6E7472 = 'mntr'
On Wed, Jan 18, 2012 at 2:00 AM, Patrick Hunt <ph...@apache.org> wrote:
> Hi Alexius, it looks like localhost (127.0.0.1) is making a socket
> connection and then sending a request with a supposed size of 1.8GB.
> (the first 4 bytes of each zk request are the length)
>
> It's likely that this is a "4letterword" attempt gone wrong (or some
> other host incorrectly pinging this port). Which application are you
> pointing at the ZK server's client port? Could this be an invalid
> monitoring request? (say someone trying to do a "stat" but spelling it
> wrong?)
>
> Patrick
>
> On Tue, Jan 17, 2012 at 5:00 PM, Alexius Ludeman <le...@lexinator.com>
> wrote:
> > hi,
> >
> > I am getting the error from all 5 zookeeperd, "Exception causing close of
> > session 0x0 due to java.io.IOException: Len error 1835955314" every
> > microsecond(?).
> >
> > I have a 5 node cluster on ubuntu maverick in aws. I've backported
> > zookeeperd and now am running 3.3.4+dfsg1-1. My apps are
> > using zookeeper-3.3.3.jar, but within the next 3-4 weeks I will be
> > upgrading to zookeeper-3.3.4.jar. This problem also existed with
> > zookeeperd on 3.3.3, in hopes of upgrading would solve my problem.
> >
> > I have stopped the whole world, rm -rf /var/lib/zookeeper and restarted
> > without any change to this log message. I have several other
> environments
> > without this error showing, same versions of client and server but only 3
> > zookeeperd, and much fewer clients.
> >
> > Our use of zookeeper is: appName is a key and the value is a server:port.
> > After the app reads in its required appName(s), it puts watches on these
> > appNames looking for changes. So we have a bunch of apps with many
> > watches. The client count is below.
> >
> > I would be grateful if anyone has some insight on how to debug further.
> >
> > thanks
> > lex
> >
> > zookeeper-0.p2:/var/log/zookeeper/zookeeper.log:
> >
> > 2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> > connection from /127.0.0.1:38653
> > 2012-01-17 22:23:59,097 - INFO [Thread-2309197:NIOServerCnxn@1435][] -
> > Closed socket connection for client /127.0.0.1:38652 (no session
> > established for client)
> > 2012-01-17 22:23:59,097 - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
> > session 0x0 due to java.io.IOException: Len error 1835955314
> > 2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection
> for
> > client /127.0.0.1:38653 (no session established for client)
> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> > connection from /127.0.0.1:38654
> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command
> from /
> > 127.0.0.1:38654
> > 2012-01-17 22:23:59,098 - INFO
> > [Thread-2309198:NIOServerCnxn$StatCommand@1153][] - Stat command output
> > 2012-01-17 22:23:59,098 - INFO [Thread-2309198:NIOServerCnxn@1435][] -
> > Closed socket connection for client /127.0.0.1:38654 (no session
> > established for client)
> > 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> > connection from /127.0.0.1:38655
> > 2012-01-17 22:23:59,099 - WARN [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
> > session 0x0 due to java.io.IOException: Len error 1835955314
> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection
> for
> > client /127.0.0.1:38655 (no session established for client)
> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> > connection from /127.0.0.1:38656
> > 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command
> from /
> > 127.0.0.1:38656
> >
> >
> > Connected to zookeeper-0.p2
> > envi
> >
> > Environment:
> > zookeeper.version=3.3.4--1, built on 12/09/2011 00:03 GMT
> > host.name=zookeeper-0.p2.use.color.internal
> > java.version=1.6.0_20
> > java.vendor=Sun Microsystems Inc.
> > java.home=/usr/lib/jvm/java-6-openjdk/jre
> >
> java.class.path=/etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar
> >
> java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
> > java.io.tmpdir=/tmp
> > java.compiler=<NA>
> > os.name=Linux
> > os.arch=amd64
> > os.version=2.6.35-24-virtual
> > user.name=zookeeper
> > user.home=/var/lib/zookeeper
> > user.dir=/
> >
> > Trying 10.118.145.79...Connected to zookeeper-0.p2
> > stats
> >
> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> > Clients:
> > /10.40.50.163:45526[1](queued=0,recved=2819,sent=2820)
> > /10.124.143.97:46512[0](queued=0,recved=1,sent=0)
> > /10.40.37.34:33308[1](queued=0,recved=2795,sent=2796)
> > /10.205.49.18:35339[1](queued=0,recved=2671,sent=2672)
> > /10.205.49.18:42419[1](queued=0,recved=2718,sent=2719)
> > /10.80.82.184:43535[1](queued=0,recved=2819,sent=2820)
> > /10.84.133.213:52533[1](queued=0,recved=2817,sent=2844)
> >
> > Latency min/avg/max: 0/0/200
> > Received: 7330810
> > Sent: 7331408
> > Outstanding: 0
> > Zxid: 0x100014e31
> > Mode: follower
> > Node count: 197
> >
> > Trying 10.34.37.214...Connected to zookeeper-1.p2
> > stat
> >
> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> > Clients:
> > /10.79.6.245:38198[1](queued=0,recved=2680,sent=2681)
> > /10.98.119.91:37301[1](queued=0,recved=2689,sent=2690)
> > /10.79.35.125:59744[1](queued=0,recved=2681,sent=2682)
> > /10.39.81.28:38047[1](queued=0,recved=2666,sent=2667)
> > /10.124.143.97:48969[0](queued=0,recved=1,sent=0)
> >
> > Latency min/avg/max: 0/0/81
> > Received: 4984011
> > Sent: 4984267
> > Outstanding: 0
> > Zxid: 0x100014e31
> > Mode: follower
> > Node count: 197
> >
> > Trying 10.38.85.114...Connected to zookeeper-2.p2
> >
> > stat
> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> > Clients:
> > /10.84.133.213:51499[1](queued=0,recved=2655,sent=2656)
> > /10.124.143.97:39412[0](queued=0,recved=1,sent=0)
> > /10.39.81.28:56431[1](queued=0,recved=2715,sent=2716)
> > /10.204.183.195:40180[1](queued=0,recved=2664,sent=2665)
> > /10.32.51.156:59390[1](queued=0,recved=2634,sent=2635)
> > /10.79.6.245:42161[1](queued=0,recved=2731,sent=2732)
> > /10.118.14.138:40355[1](queued=0,recved=2643,sent=2644)
> >
> > Latency min/avg/max: 0/0/33
> > Received: 6650278
> > Sent: 6651369
> > Outstanding: 0
> > Zxid: 0x100014e31
> > Mode: leader
> > Node count: 197
> >
> > Trying 10.101.42.61...Connected to zookeeper-3.p2
> >
> > stat
> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> > Clients:
> > /10.204.230.212:57523[1](queued=0,recved=2719,sent=2720)
> > /10.85.19.23:52916[1](queued=0,recved=2836,sent=2837)
> > /10.79.35.125:54742[1](queued=0,recved=2734,sent=2735)
> > /10.124.143.97:60909[0](queued=0,recved=1,sent=0)
> > /10.204.230.212:57918[1](queued=0,recved=2674,sent=2675)
> > /10.100.90.101:35977[1](queued=0,recved=2680,sent=2681)
> > /10.98.119.91:42147[1](queued=0,recved=2745,sent=2746)
> >
> > Latency min/avg/max: 0/0/123
> > Received: 8430215
> > Sent: 8430724
> > Outstanding: 0
> > Zxid: 0x100014e31
> > Mode: follower
> > Node count: 197
> >
> > Trying 10.204.59.55...Connected to zookeeper-4.p2
> >
> > stat
> > Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> > Clients:
> > /10.100.90.101:34437[1](queued=0,recved=2728,sent=2729)
> > /10.204.183.195:49961[1](queued=0,recved=2706,sent=2707)
> > /10.124.143.97:34797[0](queued=0,recved=1,sent=0)
> >
> > Latency min/avg/max: 0/0/19
> > Received: 46410
> > Sent: 46411
> > Outstanding: 0
> > Zxid: 0x100014e31
> > Mode: follower
> > Node count: 197
>
Re: chatty error: Exception causing close of session 0x0 due to
java.io.IOException: Len error 1835955314
Posted by Patrick Hunt <ph...@apache.org>.
Hi Alexius, it looks like localhost (127.0.0.1) is making a socket
connection and then sending a request with a supposed size of 1.8GB.
(the first 4 bytes of each zk request are the length)
It's likely that this is a "4letterword" attempt gone wrong (or some
other host incorrectly pinging this port). Which application are you
pointing at the ZK server's client port? Could this be an invalid
monitoring request? (say someone trying to do a "stat" but spelling it
wrong?)
Patrick
On Tue, Jan 17, 2012 at 5:00 PM, Alexius Ludeman <le...@lexinator.com> wrote:
> hi,
>
> I am getting the error from all 5 zookeeperd, "Exception causing close of
> session 0x0 due to java.io.IOException: Len error 1835955314" every
> microsecond(?).
>
> I have a 5 node cluster on ubuntu maverick in aws. I've backported
> zookeeperd and now am running 3.3.4+dfsg1-1. My apps are
> using zookeeper-3.3.3.jar, but within the next 3-4 weeks I will be
> upgrading to zookeeper-3.3.4.jar. This problem also existed with
> zookeeperd on 3.3.3, in hopes of upgrading would solve my problem.
>
> I have stopped the whole world, rm -rf /var/lib/zookeeper and restarted
> without any change to this log message. I have several other environments
> without this error showing, same versions of client and server but only 3
> zookeeperd, and much fewer clients.
>
> Our use of zookeeper is: appName is a key and the value is a server:port.
> After the app reads in its required appName(s), it puts watches on these
> appNames looking for changes. So we have a bunch of apps with many
> watches. The client count is below.
>
> I would be grateful if anyone has some insight on how to debug further.
>
> thanks
> lex
>
> zookeeper-0.p2:/var/log/zookeeper/zookeeper.log:
>
> 2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> connection from /127.0.0.1:38653
> 2012-01-17 22:23:59,097 - INFO [Thread-2309197:NIOServerCnxn@1435][] -
> Closed socket connection for client /127.0.0.1:38652 (no session
> established for client)
> 2012-01-17 22:23:59,097 - WARN [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
> session 0x0 due to java.io.IOException: Len error 1835955314
> 2012-01-17 22:23:59,097 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection for
> client /127.0.0.1:38653 (no session established for client)
> 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> connection from /127.0.0.1:38654
> 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command from /
> 127.0.0.1:38654
> 2012-01-17 22:23:59,098 - INFO
> [Thread-2309198:NIOServerCnxn$StatCommand@1153][] - Stat command output
> 2012-01-17 22:23:59,098 - INFO [Thread-2309198:NIOServerCnxn@1435][] -
> Closed socket connection for client /127.0.0.1:38654 (no session
> established for client)
> 2012-01-17 22:23:59,098 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> connection from /127.0.0.1:38655
> 2012-01-17 22:23:59,099 - WARN [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
> session 0x0 due to java.io.IOException: Len error 1835955314
> 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection for
> client /127.0.0.1:38655 (no session established for client)
> 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> connection from /127.0.0.1:38656
> 2012-01-17 22:23:59,099 - INFO [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command from /
> 127.0.0.1:38656
>
>
> Connected to zookeeper-0.p2
> envi
>
> Environment:
> zookeeper.version=3.3.4--1, built on 12/09/2011 00:03 GMT
> host.name=zookeeper-0.p2.use.color.internal
> java.version=1.6.0_20
> java.vendor=Sun Microsystems Inc.
> java.home=/usr/lib/jvm/java-6-openjdk/jre
> java.class.path=/etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar
> java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
> java.io.tmpdir=/tmp
> java.compiler=<NA>
> os.name=Linux
> os.arch=amd64
> os.version=2.6.35-24-virtual
> user.name=zookeeper
> user.home=/var/lib/zookeeper
> user.dir=/
>
> Trying 10.118.145.79...Connected to zookeeper-0.p2
> stats
>
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
> /10.40.50.163:45526[1](queued=0,recved=2819,sent=2820)
> /10.124.143.97:46512[0](queued=0,recved=1,sent=0)
> /10.40.37.34:33308[1](queued=0,recved=2795,sent=2796)
> /10.205.49.18:35339[1](queued=0,recved=2671,sent=2672)
> /10.205.49.18:42419[1](queued=0,recved=2718,sent=2719)
> /10.80.82.184:43535[1](queued=0,recved=2819,sent=2820)
> /10.84.133.213:52533[1](queued=0,recved=2817,sent=2844)
>
> Latency min/avg/max: 0/0/200
> Received: 7330810
> Sent: 7331408
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: follower
> Node count: 197
>
> Trying 10.34.37.214...Connected to zookeeper-1.p2
> stat
>
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
> /10.79.6.245:38198[1](queued=0,recved=2680,sent=2681)
> /10.98.119.91:37301[1](queued=0,recved=2689,sent=2690)
> /10.79.35.125:59744[1](queued=0,recved=2681,sent=2682)
> /10.39.81.28:38047[1](queued=0,recved=2666,sent=2667)
> /10.124.143.97:48969[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0/81
> Received: 4984011
> Sent: 4984267
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: follower
> Node count: 197
>
> Trying 10.38.85.114...Connected to zookeeper-2.p2
>
> stat
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
> /10.84.133.213:51499[1](queued=0,recved=2655,sent=2656)
> /10.124.143.97:39412[0](queued=0,recved=1,sent=0)
> /10.39.81.28:56431[1](queued=0,recved=2715,sent=2716)
> /10.204.183.195:40180[1](queued=0,recved=2664,sent=2665)
> /10.32.51.156:59390[1](queued=0,recved=2634,sent=2635)
> /10.79.6.245:42161[1](queued=0,recved=2731,sent=2732)
> /10.118.14.138:40355[1](queued=0,recved=2643,sent=2644)
>
> Latency min/avg/max: 0/0/33
> Received: 6650278
> Sent: 6651369
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: leader
> Node count: 197
>
> Trying 10.101.42.61...Connected to zookeeper-3.p2
>
> stat
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
> /10.204.230.212:57523[1](queued=0,recved=2719,sent=2720)
> /10.85.19.23:52916[1](queued=0,recved=2836,sent=2837)
> /10.79.35.125:54742[1](queued=0,recved=2734,sent=2735)
> /10.124.143.97:60909[0](queued=0,recved=1,sent=0)
> /10.204.230.212:57918[1](queued=0,recved=2674,sent=2675)
> /10.100.90.101:35977[1](queued=0,recved=2680,sent=2681)
> /10.98.119.91:42147[1](queued=0,recved=2745,sent=2746)
>
> Latency min/avg/max: 0/0/123
> Received: 8430215
> Sent: 8430724
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: follower
> Node count: 197
>
> Trying 10.204.59.55...Connected to zookeeper-4.p2
>
> stat
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
> /10.100.90.101:34437[1](queued=0,recved=2728,sent=2729)
> /10.204.183.195:49961[1](queued=0,recved=2706,sent=2707)
> /10.124.143.97:34797[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0/19
> Received: 46410
> Sent: 46411
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: follower
> Node count: 197