You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by iain wright <ia...@gmail.com> on 2012/10/08 23:41:33 UTC
flume+hbase+zookeeper, can't hang on to zookeeper session
We're having some trouble with the Flume & the HbaseSink working. Seems we
cannot hang on to zookeeper sessions. The flume list recommended I cross
post here as this seems to be something up with the hbaseclient &/or
zookeeper. In all honesty I am probably just doing something dumb and not
seeing it.
Using latest flume-ng 1.3.0, hbase 0.94.0. Hbase & ZK both look fine and
are in normal prod use. I don't think we are hitting our
hbase.zookeeper.property.maxClientCnxns as we only have about 95 sessions
in ZK DUMP and I believe it defaults to 2k. I can establish new sessions
using the CLI from the same server and idle there for 10 minutes without
getting dropped. When I start up flume I see the session show up in the ZK
DUMP, see active TCP connection on the port etc., it just keeps
dropping/reconnecting.
Flume & zookeeper logs below, using rsync'd copies of our hadoop & hbase
directories from one of our datanode+regionserver's.
*Searched the flume list, this user appeared to have the same problem, not
sure how he fixed it though:*
http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%3CCADGpRhcgztwhuUJi1izMRX2UOKeAWwhYOkKSKkHYmKopAkwHAQ@mail.gmail.com%3E
*startup cmd
*/app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf -f
conf/brian.properties -Dflume.root.logger=INFO,console*
*
*flume-env.sh*
$ cat conf/flume-env.sh
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.
# Enviroment variables can be set here.
#JAVA_HOME=/usr/lib/jvm/java-6-sun
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
#JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH="/app/flume/lib"
FLUME_CLASSPATH="/app/apache-flume-1.3.0-SNAPSHOT/lib"
HBASE_HOME="/app/hbase-0.94.0"
HADOOP_HOME="/app/hadoop-1.0.1"
*config*
$ cat conf/brian.properties
#example.conf: A single-node Flume configuration
# Name the components on this agent
agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1
# Describe/configure source1
agent1.sources.source1.type = exec
agent1.sources.source1.command = tail -F /tank/log_dev.log
agent1.sources.source1.batchSize = 1
# Describe sink1
#agent1.sinks.sink1.type = logger
agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
agent1.sinks.sink1.table = brian_test
agent1.sinks.sink1.columnFamily = f1
#agent1.sinks.sink1.serializer =
org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
#agent1.sinks.sink1.serializer =
org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.capacity = 1000
agent1.channels.channel1.transactionCapactiy = 100
# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1
*flume console log*
2012-10-08 12:06:19,007 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.io.tmpdir=/var/tmp/
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.compiler=<NA>
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.name=FreeBSD
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.arch=amd64
2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.version=9.0-RELEASE
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.name=czhang
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.home=/users/czhang
2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
2012-10-08 12:06:19,010 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.com:2181sessionTimeout=180000
watcher=hconnection
2012-10-08 12:06:19,036 (lifecycleSupervisor-1-1) [INFO -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:97)]
The identifier of this process is 12494@app14.app.domain.com
2012-10-08 12:06:19,041 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 12:06:19,049 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 12:06:19,072 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bbce, negotiated timeout = 180000
2012-10-08 12:08:19,070 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
Client session timed out, have not heard from server in 120001ms for
sessionid 0x138a0620045bbce, closing socket connection and attempting
reconnect
2012-10-08 12:08:19,233 (lifecycleSupervisor-1-1) [WARN -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
Possibly transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
2012-10-08 12:08:19,234 (lifecycleSupervisor-1-1) [INFO -
org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
Sleeping 2000ms before retry #1...
2012-10-08 12:08:20,691 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 12:08:20,692 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 12:08:20,695 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bbce, negotiated timeout = 180000
2012-10-08 12:10:20,695 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
Client session timed out, have not heard from server in 120000ms for
sessionid 0x138a0620045bbce, closing socket connection and attempting
reconnect
2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [WARN -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
Possibly transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [INFO -
org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
Sleeping 4000ms before retry #2...
2012-10-08 12:10:22,792 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 12:10:22,794 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 12:10:22,799 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bbce, negotiated timeout = 180000
2012-10-08 12:12:22,800 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
Client session timed out, have not heard from server in 120001ms for
sessionid 0x138a0620045bbce, closing socket connection and attempting
reconnect
2012-10-08 12:12:22,902 (lifecycleSupervisor-1-1) [WARN -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
Possibly transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
2012-10-08 12:12:22,903 (lifecycleSupervisor-1-1) [INFO -
org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
Sleeping 8000ms before retry #3...
2012-10-08 12:12:24,289 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 12:12:24,291 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 12:12:24,294 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bbce, negotiated timeout = 180000
2012-10-08 12:14:24,294 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
Client session timed out, have not heard from server in 120000ms for
sessionid 0x138a0620045bbce, closing socket connection and attempting
reconnect
2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [WARN -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
Possibly transient ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [ERROR -
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:197)]
ZooKeeper exists failed after 3 retries
2012-10-08 12:14:24,398 (lifecycleSupervisor-1-1) [WARN -
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:239)]
hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce
Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107)
at
org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2012-10-08 12:14:24,405 (lifecycleSupervisor-1-1) [ERROR -
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.keeperException(ZooKeeperWatcher.java:408)]
hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce
Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107)
at
org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2012-10-08 12:14:24,407 (lifecycleSupervisor-1-1) [INFO -
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1654)]
This client just lost it's session with ZooKeeper, will automatically
reconnect when needed.
2012-10-08 12:14:26,166 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 12:14:26,167 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 12:14:26,170 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bbce, negotiated timeout = 180000
*zookeeper log*
2012-10-08 19:06:19,049 INFO
org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
connection from /10.10.10.64:16755
2012-10-08 19:06:19,053 INFO org.apache.zookeeper.server.ZooKeeperServer:
Client attempting to establish new session at /10.10.10.64:16755
2012-10-08 19:06:19,069 INFO org.apache.zookeeper.server.ZooKeeperServer:
Established session 0x138a0620045bbce with negotiated timeout 180000 for
client /10.10.10.64:16755
2012-10-08 19:08:19,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
Closed socket connection for client /10.10.10.64:16755 which had sessionid
0x138a0620045bbce
2012-10-08 19:08:20,692 INFO
org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
connection from /10.10.10.64:33245
2012-10-08 19:08:20,693 INFO org.apache.zookeeper.server.ZooKeeperServer:
Client attempting to renew session 0x138a0620045bbce at /10.10.10.64:33245
2012-10-08 19:08:20,694 INFO org.apache.zookeeper.server.ZooKeeperServer:
Established session 0x138a0620045bbce with negotiated timeout 180000 for
client /10.10.10.64:33245
2012-10-08 19:10:20,695 WARN org.apache.zookeeper.server.NIOServerCnxn:
caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x138a0620045bbce, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
at java.lang.Thread.run(Thread.java:619)
2012-10-08 19:10:20,698 INFO org.apache.zookeeper.server.NIOServerCnxn:
Closed socket connection for client /10.10.10.64:33245 which had sessionid
0x138a0620045bbce
2012-10-08 19:10:22,794 INFO
org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
connection from /10.10.10.64:44993
2012-10-08 19:10:22,796 INFO org.apache.zookeeper.server.ZooKeeperServer:
Client attempting to renew session 0x138a0620045bbce at /10.10.10.64:44993
2012-10-08 19:10:22,798 INFO org.apache.zookeeper.server.ZooKeeperServer:
Established session 0x138a0620045bbce with negotiated timeout 180000 for
client /10.10.10.64:44993
*We also tried the hbase async sink as that is recommended over in the
flume group, and got similar ** results (but seemingly more serious
CONNECTIONLOSS errors)*
2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)]
Starting new configuration:{
sourceRunners:{source1=EventDrivenSourceRunner: {
source:org.apache.flume.source.ExecSource{name:source1,state:IDLE} }}
sinkRunners:{sink1=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@3968a6fe counterGroup:{
name:null counters:{} } }}
channels:{channel1=org.apache.flume.channel.MemoryChannel{name: channel1}} }
2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
Starting Channel channel1
2012-10-08 14:38:29,641 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)]
Component type: CHANNEL, name: channel1 started
2012-10-08 14:38:29,642 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
Starting Sink sink1
2012-10-08 14:38:29,642 (lifecycleSupervisor-1-1) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)]
Component type: SINK, name: sink1 started
2012-10-08 14:38:29,643 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
Starting Source source1
2012-10-08 14:38:29,643 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.source.ExecSource.start(ExecSource.java:155)] Exec source
starting with command:tail -F
/tank/clients/telescope/users/czhang/connecttwitter_dev.log
2012-10-08 14:38:29,816 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:zookeeper.version=3.4.4-1386507, built on 09/17/2012 08:33 GMT
2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:host.name=app14.app.domain.com
2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.version=1.7.0_02
2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.vendor=Oracle Corporation
2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.home=/usr/local/openjdk7/jre
2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.class.path=/app/flume/conf:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-avro-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-log4jappender-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-lang-2.5.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.2.7.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-scribe-source-1.3.0-20120929.012216-17.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-pool-1.5.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-ipc-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-thrift-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpclient-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/mina-core-2.0.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-mapper-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/servlet-api-2.5-20110124.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jcl-over-slf4j-1.7.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jsr305-1.3.9.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.4.0.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-util-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-hdfs-sink-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-collections-3.2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/async-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/paranamer-2.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/velocity-1.7.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-codec-1.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-hbase-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/irclib-1.10.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-jdbc-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-dbcp-1.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-api-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/protobuf-java-2.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-irc-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/joda-time-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-io-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-file-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-recoverable-memory-channel-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-cli-1.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/asynchbase-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/gson-2.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/derby-10.8.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-logging-1.1.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/guava-10.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-log4j12-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpcore-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/libthrift-0.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/log4j-1.2.16.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/zookeeper-3.4.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/snappy-java-1.0.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-core-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib:/app/hadoop-1.0.1/libexec/../conf:/usr/local/lib/tools.jar:/app/hadoop-1.0.1/libexec/..:/app/hadoop-1.0.1/libexec/../hadoop-core-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/asm-3.2.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjrt-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjtools-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-1.7.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-cli-1.2.jar:/app/hadoop-1.0.1/libexec/../lib/commons-codec-1.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-collections-3.2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-configuration-1.6.jar:/app/hadoop-1.0.1/libexec/../lib/commons-daemon-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-digester-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/commons-el-1.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-httpclient-3.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-lang-2.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-1.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-api-1.0.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-math-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-net-1.4.1.jar:/app/hadoop-1.0.1/libexec/../lib/core-3.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-capacity-scheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-fairscheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-thriftfs-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hsqldb-1.8.0.10.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-compiler-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-runtime-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jdeb-0.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-core-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-json-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-server-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jets3t-0.6.1.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-util-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jsch-0.1.42.jar:/app/hadoop-1.0.1/libexec/../lib/junit-4.5.jar:/app/hadoop-1.0.1/libexec/../lib/kfs-0.2.2.jar:/app/hadoop-1.0.1/libexec/../lib/log4j-1.2.15.jar:/app/hadoop-1.0.1/libexec/../lib/mockito-all-1.8.5.jar:/app/hadoop-1.0.1/libexec/../lib/oro-2.0.8.jar:/app/hadoop-1.0.1/libexec/../lib/servlet-api-2.5-20081211.jar:/app/hadoop-1.0.1/libexec/../lib/xmlenc-0.52.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/app/hbase-0.94.0/bin/../conf:/usr/local/lib/tools.jar:/app/hbase-0.94.0/bin/..:/app/hbase-0.94.0/bin/../hbase-0.94.0-tests.jar:/app/hbase-0.94.0/bin/../hbase-0.94.0.jar:/app/hbase-0.94.0/bin/../lib/activation-1.1.jar:/app/hbase-0.94.0/bin/../lib/asm-3.1.jar:/app/hbase-0.94.0/bin/../lib/avro-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/avro-ipc-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-1.7.0.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-core-1.8.0.jar:/app/hbase-0.94.0/bin/../lib/commons-cli-1.2.jar:/app/hbase-0.94.0/bin/../lib/commons-codec-1.4.jar:/app/hbase-0.94.0/bin/../lib/commons-collections-3.2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-configuration-1.6.jar:/app/hbase-0.94.0/bin/../lib/commons-digester-1.8.jar:/app/hbase-0.94.0/bin/../lib/commons-el-1.0.jar:/app/hbase-0.94.0/bin/../lib/commons-httpclient-3.1.jar:/app/hbase-0.94.0/bin/../lib/commons-io-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-lang-2.5.jar:/app/hbase-0.94.0/bin/../lib/commons-logging-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/commons-math-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-net-1.4.1.jar:/app/hbase-0.94.0/bin/../lib/core-3.1.1.jar:/app/hbase-0.94.0/bin/../lib/guava-r09.jar:/app/hbase-0.94.0/bin/../lib/hadoop-core-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/high-scale-lib-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/httpclient-4.1.2.jar:/app/hbase-0.94.0/bin/../lib/httpcore-4.1.3.jar:/app/hbase-0.94.0/bin/../lib/jackson-core-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-jaxrs-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-mapper-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-xc-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jamon-runtime-2.3.1.jar:/app/hbase-0.94.0/bin/../lib/jasper-compiler-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jasper-runtime-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jaxb-api-2.1.jar:/app/hbase-0.94.0/bin/../lib/jaxb-impl-2.1.12.jar:/app/hbase-0.94.0/bin/../lib/jersey-core-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-json-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-server-1.4.jar:/app/hbase-0.94.0/bin/../lib/jettison-1.1.jar:/app/hbase-0.94.0/bin/../lib/jetty-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jetty-util-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jruby-complete-1.6.5.jar:/app/hbase-0.94.0/bin/../lib/jsp-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/libthrift-0.8.0.jar:/app/hbase-0.94.0/bin/../lib/log4j-1.2.16.jar:/app/hbase-0.94.0/bin/../lib/netty-3.2.4.Final.jar:/app/hbase-0.94.0/bin/../lib/protobuf-java-2.4.0a.jar:/app/hbase-0.94.0/bin/../lib/servlet-api-2.5-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/snappy-java-1.0.3.2.jar:/app/hbase-0.94.0/bin/../lib/stax-api-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/velocity-1.7.jar:/app/hbase-0.94.0/bin/../lib/xmlenc-0.52.jar:/app/hbase-0.94.0/bin/../lib/zookeeper-3.4.3.jar:.:/app/hbase-0.94.0/conf
2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.io.tmpdir=/var/tmp/
2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:java.compiler=<NA>
2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.name=FreeBSD
2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.arch=amd64
2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:os.version=9.0-RELEASE
2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.name=czhang
2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.home=/tank/clients/telescope/users/czhang
2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
2012-10-08 14:38:29,822 (lifecycleSupervisor-1-1) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
2012-10-08 14:38:29,863 (lifecycleSupervisor-1-1) [INFO -
org.hbase.async.HBaseClient$ZKClient.getDeferredRoot(HBaseClient.java:2238)]
Need to find the -ROOT- region
2012-10-08 14:38:29,870 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 14:38:29,879 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 14:38:29,908 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bdb5, negotiated timeout = 6000
2012-10-08 14:38:33,906 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
Client session timed out, have not heard from server in 4001ms for
sessionid 0x138a0620045bdb5, closing socket connection and attempting
reconnect
2012-10-08 14:38:34,013 (lifecycleSupervisor-1-1-EventThread) [ERROR -
org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
0x138a0620045bdb5 closed
2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
2012-10-08 14:38:34,971 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 14:38:34,972 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 14:38:34,986 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bdba, negotiated timeout = 6000
2012-10-08 14:38:39,088 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
0x138a0620045bdba closed
2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [WARN -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
type:None path:null
2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
EventThread shut down
2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
EventThread shut down
2012-10-08 14:38:39,091 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 14:38:39,092 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 14:38:39,163 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bdbb, negotiated timeout = 6000
2012-10-08 14:38:43,164 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
Client session timed out, have not heard from server in 4001ms for
sessionid 0x138a0620045bdbb, closing socket connection and attempting
reconnect
2012-10-08 14:38:43,265 (lifecycleSupervisor-1-1-EventThread) [ERROR -
org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
2012-10-08 14:38:43,683 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
0x138a0620045bdbb closed
2012-10-08 14:38:43,684 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
2012-10-08 14:38:43,686 (lifecycleSupervisor-1-1-EventThread) [ERROR -
org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
2012-10-08 14:38:43,687 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 14:38:43,688 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 14:38:43,701 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bdbc, negotiated timeout = 6000
2012-10-08 14:38:47,802 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
0x138a0620045bdbc closed
2012-10-08 14:38:47,804 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
2012-10-08 14:38:47,807 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
EventThread shut down
2012-10-08 14:38:47,810 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 14:38:47,812 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 14:38:47,821 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bdbd, negotiated timeout = 6000
2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
0x138a0620045bdbd closed
2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [WARN -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
type:None path:null
2012-10-08 14:38:51,923 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [INFO -
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
EventThread shut down
2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [ERROR -
org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
2012-10-08 14:38:51,925 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 14:38:51,926 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
2012-10-08 14:38:51,945 (lifecycleSupervisor-1-1-SendThread(
hbasemaster0.hadoop.domain.com:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
0x138a0620045bdbe, negotiated timeout = 6000
Thanks for the continued help from the community on getting us going w/flume
Cheers,
--
Iain Wright
Cell: (562) 852-5916
<http://www.labctsi.org/>
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
Re: flume+hbase+zookeeper, can't hang on to zookeeper session
Posted by anil gupta <an...@gmail.com>.
My 2 cents:
In past when i have used OpenJDK, i ran into a DNS resolution trouble with
Hadoop/HBase. The problem was resolved when i started using Sun Jdk. I
would suggest you to try the preferred version of Sun Java 1.6. You can
find this info on HBase website. It's worth a try although i am not sure
that this is the problem.
HTH,
Anil Gupta
On Mon, Oct 8, 2012 at 3:56 PM, iain wright <ia...@gmail.com> wrote:
> Below is a debug level log of flume trying to connect. Could we
> experiencing this bug:
> https://issues.apache.org/jira/browse/ZOOKEEPER-1550
>
> We are using openjdk version "1.7.0_02"
> OpenJDK Runtime Environment (build 1.7.0_02-b13)
> OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
>
> debug log:
>
> 2012-10-08 15:28:01,324 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
>
> 2012-10-08 15:28:01,325 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:01,326 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:01,327 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:01,328 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:03,322 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:03,323 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:03,332 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:03,334 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:05,333 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:05,334 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:05,334 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
> exception was thrown while closing send thread for session
> 0x138a0620045bf8f : Client session timed out, have not heard from server in
> 4009ms for sessionid 0x138a0620045bf8f
>
> 2012-10-08 15:28:05,410 (conf-file-poller-0) [DEBUG -
>
> org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
> Checking file:conf/brian.properties for changes
>
> 2012-10-08 15:28:05,435 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
> Disconnecting client for session: 0x138a0620045bf8f
>
> 2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bf8f closed
>
> 2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
> ZooKeeper#close completed in 4123621708ns
>
> 2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection,
> connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
>
> 2012-10-08 15:28:05,438 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.getRootRegion(HBaseClient.java:2512)]
> Finding the -ROOT- region in ZooKeeper
>
> 2012-10-08 15:28:05,438 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
>
> 2012-10-08 15:28:05,439 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2314)] Done
> handling ZooKeeper event: WatchedEvent state:SyncConnected type:None
> path:null
>
> 2012-10-08 15:28:05,439 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
> Ignore any DEBUG exception from ZooKeeper
>
> 2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
> 2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-EventThread) [ERROR -
>
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
>
> 2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
> 0x0
>
> 2012-10-08 15:28:05,442 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
> for session: 0x0
>
> 2012-10-08 15:28:05,442 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
>
> 2012-10-08 15:28:05,443 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
> Session establishment request sent on
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
>
> 2012-10-08 15:28:05,444 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:05,445 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:05,455 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
> 0x138a0620045bf90, negotiated timeout = 6000
>
> 2012-10-08 15:28:05,456 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
>
> 2012-10-08 15:28:05,456 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:05,457 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:05,458 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:05,459 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:07,450 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:07,452 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:07,454 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:07,455 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,456 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:09,458 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,460 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:09,462 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,462 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
> exception was thrown while closing send thread for session
> 0x138a0620045bf90 : Client session timed out, have not heard from server in
> 4003ms for sessionid 0x138a0620045bf90
>
> 2012-10-08 15:28:09,564 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
> Disconnecting client for session: 0x138a0620045bf90
>
> 2012-10-08 15:28:09,566 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bf90 closed
>
> 2012-10-08 15:28:09,566 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
> ZooKeeper#close completed in 4126386193ns
>
> 2012-10-08 15:28:09,567 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection,
> connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
>
> 2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
> Ignore any DEBUG exception from ZooKeeper
>
> 2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
>
> 2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
> 0x0
>
> 2012-10-08 15:28:09,575 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
> for session: 0x0
>
> 2012-10-08 15:28:09,576 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
> 2012-10-08 15:28:09,579 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
>
> 2012-10-08 15:28:09,580 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
> Session establishment request sent on
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
>
> 2012-10-08 15:28:09,583 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,599 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
> 0x138a0620045bf91, negotiated timeout = 6000
>
> 2012-10-08 15:28:09,601 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
>
> 2012-10-08 15:28:09,602 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:09,603 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:11,588 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:11,591 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,592 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,596 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,604 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,606 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
> exception was thrown while closing send thread for session
> 0x138a0620045bf91 : Client session timed out, have not heard from server in
> 4003ms for sessionid 0x138a0620045bf91
>
> 2012-10-08 15:28:13,708 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
> Disconnecting client for session: 0x138a0620045bf91
>
> 2012-10-08 15:28:13,710 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bf91 closed
>
> 2012-10-08 15:28:13,711 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
> ZooKeeper#close completed in 4137007581ns
>
> 2012-10-08 15:28:13,712 (lifecycleSupervisor-1-0-EventThread) [WARN -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
> longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
> type:None path:null
>
> 2012-10-08 15:28:13,713 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection,
> connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
>
> 2012-10-08 15:28:13,721 (lifecycleSupervisor-1-0-EventThread) [INFO -
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> EventThread shut down
>
> 2012-10-08 15:28:13,721 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.getRootRegion(HBaseClient.java:2512)]
> Finding the -ROOT- region in ZooKeeper
>
> 2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
>
> 2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2314)] Done
> handling ZooKeeper event: WatchedEvent state:SyncConnected type:None
> path:null
>
> 2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
> Ignore any DEBUG exception from ZooKeeper
>
> 2012-10-08 15:28:13,727 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
>
> 2012-10-08 15:28:13,730 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
>
> 2012-10-08 15:28:13,730 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
> 0x0
>
> 2012-10-08 15:28:13,733 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
> Session establishment request sent on
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
>
> 2012-10-08 15:28:13,734 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
> for session: 0x0
>
> 2012-10-08 15:28:13,761 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:13,772 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
>
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
> 0x138a0620045bf92, negotiated timeout = 6000
>
> 2012-10-08 15:28:13,773 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
> ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
>
> 2012-10-08 15:28:13,776 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
>
> 2012-10-08 15:28:13,779 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
>
> 2012-10-08 15:28:13,783 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:/hbase/root-region-server
> serverPath:/hbase/root-region-server finished:false header:: 0,4
> replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
> until SASL authentication completes.
> 2012-10-08 15:28:13,786 (lifecycleSupervisor-1-0-SendThread(
> hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
>
> org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
> deferring non-priming packet: clientPath:null serverPath:null
> finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
> response:: nulluntil SASL authentication completes.
> --
> Iain Wright
> Cell: (562) 852-5916
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
>
> On Mon, Oct 8, 2012 at 2:41 PM, iain wright <ia...@gmail.com> wrote:
>
> > We're having some trouble with the Flume & the HbaseSink working. Seems
> we
> > cannot hang on to zookeeper sessions. The flume list recommended I cross
> > post here as this seems to be something up with the hbaseclient &/or
> > zookeeper. In all honesty I am probably just doing something dumb and not
> > seeing it.
> >
> > Using latest flume-ng 1.3.0, hbase 0.94.0. Hbase & ZK both look fine and
> > are in normal prod use. I don't think we are hitting our
> > hbase.zookeeper.property.maxClientCnxns as we only have about 95 sessions
> > in ZK DUMP and I believe it defaults to 2k. I can establish new sessions
> > using the CLI from the same server and idle there for 10 minutes without
> > getting dropped. When I start up flume I see the session show up in the
> ZK
> > DUMP, see active TCP connection on the port etc., it just keeps
> > dropping/reconnecting.
> >
> > Flume & zookeeper logs below, using rsync'd copies of our hadoop & hbase
> > directories from one of our datanode+regionserver's.
> >
> > *Searched the flume list, this user appeared to have the same problem,
> > not sure how he fixed it though:*
> >
> >
> http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%3CCADGpRhcgztwhuUJi1izMRX2UOKeAWwhYOkKSKkHYmKopAkwHAQ@mail.gmail.com%3E
> >
> > *startup cmd
> > */app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf
> > -f conf/brian.properties -Dflume.root.logger=INFO,console*
> > *
> > *flume-env.sh*
> > $ cat conf/flume-env.sh
> >
> > # Licensed to the Apache Software Foundation (ASF) under one
> > # or more contributor license agreements. See the NOTICE file
> > # distributed with this work for additional information
> > # regarding copyright ownership. The ASF licenses this file
> > # to you under the Apache License, Version 2.0 (the
> > # "License"); you may not use this file except in compliance
> > # with the License. You may obtain a copy of the License at
> > #
> > # http://www.apache.org/licenses/LICENSE-2.0
> > #
> > # Unless required by applicable law or agreed to in writing, software
> > # distributed under the License is distributed on an "AS IS" BASIS,
> > # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> > # See the License for the specific language governing permissions and
> > # limitations under the License.
> >
> > # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be
> sourced
> > # during Flume startup.
> >
> > # Enviroment variables can be set here.
> >
> > #JAVA_HOME=/usr/lib/jvm/java-6-sun
> >
> > # Give Flume more memory and pre-allocate, enable remote monitoring via
> JMX
> > #JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
> >
> > # Note that the Flume conf directory is always included in the classpath.
> > #FLUME_CLASSPATH="/app/flume/lib"
> >
> > FLUME_CLASSPATH="/app/apache-flume-1.3.0-SNAPSHOT/lib"
> >
> > HBASE_HOME="/app/hbase-0.94.0"
> >
> > HADOOP_HOME="/app/hadoop-1.0.1"
> >
> >
> > *config*
> > $ cat conf/brian.properties
> > #example.conf: A single-node Flume configuration
> >
> > # Name the components on this agent
> > agent1.sources = source1
> > agent1.sinks = sink1
> > agent1.channels = channel1
> >
> > # Describe/configure source1
> > agent1.sources.source1.type = exec
> > agent1.sources.source1.command = tail -F /tank/log_dev.log
> > agent1.sources.source1.batchSize = 1
> > # Describe sink1
> > #agent1.sinks.sink1.type = logger
> > agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> > agent1.sinks.sink1.table = brian_test
> > agent1.sinks.sink1.columnFamily = f1
> > #agent1.sinks.sink1.serializer =
> > org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
> > #agent1.sinks.sink1.serializer =
> > org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
> >
> >
> > # Use a channel which buffers events in memory
> > agent1.channels.channel1.type = memory
> > agent1.channels.channel1.capacity = 1000
> > agent1.channels.channel1.transactionCapactiy = 100
> >
> > # Bind the source and sink to the channel
> > agent1.sources.source1.channels = channel1
> > agent1.sinks.sink1.channel = channel1
> >
> >
> > *flume console log*
> > 2012-10-08 12:06:19,007 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> >
> environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.io.tmpdir=/var/tmp/
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.compiler=<NA>
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.name=FreeBSD
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.arch=amd64
> > 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.version=9.0-RELEASE
> > 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.name=czhang
> > 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.home=/users/czhang
> > 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
> > 2012-10-08 12:06:19,010 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection, connectString=hbasemaster0.hadoop.domain.com:2181sessionTimeout=180000
> watcher=hconnection
> > 2012-10-08 12:06:19,036 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:97)]
> > The identifier of this process is 12494@app14.app.domain.com
> > 2012-10-08 12:06:19,041 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:06:19,049 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:06:19,072 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> > 2012-10-08 12:08:19,070 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 120001ms for
> > sessionid 0x138a0620045bbce, closing socket connection and attempting
> > reconnect
> > 2012-10-08 12:08:19,233 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> > Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-10-08 12:08:19,234 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> > Sleeping 2000ms before retry #1...
> > 2012-10-08 12:08:20,691 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:08:20,692 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:08:20,695 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> > 2012-10-08 12:10:20,695 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 120000ms for
> > sessionid 0x138a0620045bbce, closing socket connection and attempting
> > reconnect
> > 2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> > Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> > Sleeping 4000ms before retry #2...
> > 2012-10-08 12:10:22,792 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:10:22,794 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:10:22,799 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> > 2012-10-08 12:12:22,800 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 120001ms for
> > sessionid 0x138a0620045bbce, closing socket connection and attempting
> > reconnect
> > 2012-10-08 12:12:22,902 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> > Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-10-08 12:12:22,903 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> > Sleeping 8000ms before retry #3...
> > 2012-10-08 12:12:24,289 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:12:24,291 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:12:24,294 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> > 2012-10-08 12:14:24,294 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 120000ms for
> > sessionid 0x138a0620045bbce, closing socket connection and attempting
> > reconnect
> > 2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> > Possibly transient ZooKeeper exception:
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > 2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [ERROR -
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:197)]
> > ZooKeeper exists failed after 3 retries
> > 2012-10-08 12:14:24,398 (lifecycleSupervisor-1-1) [WARN -
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:239)]
> >
> hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce
> > Unable to set watcher on znode /hbase/master
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> > at
> > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> > at
> org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107)
> > at
> >
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> > at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> > at
> >
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> > at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > at java.lang.Thread.run(Thread.java:722)
> > 2012-10-08 12:14:24,405 (lifecycleSupervisor-1-1) [ERROR -
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.keeperException(ZooKeeperWatcher.java:408)]
> >
> hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce
> > Received unexpected KeeperException, re-throwing exception
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode = ConnectionLoss for /hbase/master
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> > at
> > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> > at
> org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107)
> > at
> >
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> > at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> > at
> >
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> > at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > at java.lang.Thread.run(Thread.java:722)
> > 2012-10-08 12:14:24,407 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1654)]
> > This client just lost it's session with ZooKeeper, will automatically
> > reconnect when needed.
> > 2012-10-08 12:14:26,166 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 12:14:26,167 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 12:14:26,170 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bbce, negotiated timeout = 180000
> >
> > *zookeeper log*
> > 2012-10-08 19:06:19,049 INFO
> > org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> > connection from /10.10.10.64:16755
> > 2012-10-08 19:06:19,053 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Client attempting to establish new session at /10.10.10.64:16755
> > 2012-10-08 19:06:19,069 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Established session 0x138a0620045bbce with negotiated timeout 180000 for
> > client /10.10.10.64:16755
> > 2012-10-08 19:08:19,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /10.10.10.64:16755 which had
> > sessionid 0x138a0620045bbce
> > 2012-10-08 19:08:20,692 INFO
> > org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> > connection from /10.10.10.64:33245
> > 2012-10-08 19:08:20,693 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Client attempting to renew session 0x138a0620045bbce at /
> 10.10.10.64:33245
> > 2012-10-08 19:08:20,694 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Established session 0x138a0620045bbce with negotiated timeout 180000 for
> > client /10.10.10.64:33245
> > 2012-10-08 19:10:20,695 WARN org.apache.zookeeper.server.NIOServerCnxn:
> > caught end of stream exception
> > EndOfStreamException: Unable to read additional data from client
> sessionid
> > 0x138a0620045bbce, likely client has closed socket
> > at
> > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
> > at
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
> > at java.lang.Thread.run(Thread.java:619)
> > 2012-10-08 19:10:20,698 INFO org.apache.zookeeper.server.NIOServerCnxn:
> > Closed socket connection for client /10.10.10.64:33245 which had
> > sessionid 0x138a0620045bbce
> > 2012-10-08 19:10:22,794 INFO
> > org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> > connection from /10.10.10.64:44993
> > 2012-10-08 19:10:22,796 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Client attempting to renew session 0x138a0620045bbce at /
> 10.10.10.64:44993
> > 2012-10-08 19:10:22,798 INFO org.apache.zookeeper.server.ZooKeeperServer:
> > Established session 0x138a0620045bbce with negotiated timeout 180000 for
> > client /10.10.10.64:44993
> >
> > *We also tried the hbase async sink as that is recommended over in the
> > flume group, and got similar ** results (but seemingly more serious
> > CONNECTIONLOSS errors)*
> >
> > 2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO -
> >
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)]
> > Starting new configuration:{
> > sourceRunners:{source1=EventDrivenSourceRunner: {
> > source:org.apache.flume.source.ExecSource{name:source1,state:IDLE} }}
> > sinkRunners:{sink1=SinkRunner: {
> > policy:org.apache.flume.sink.DefaultSinkProcessor@3968a6fecounterGroup:{
> > name:null counters:{} } }}
> > channels:{channel1=org.apache.flume.channel.MemoryChannel{name:
> channel1}} }
> > 2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO -
> >
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
> > Starting Channel channel1
> > 2012-10-08 14:38:29,641 (lifecycleSupervisor-1-0) [INFO -
> >
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)]
> > Component type: CHANNEL, name: channel1 started
> > 2012-10-08 14:38:29,642 (conf-file-poller-0) [INFO -
> >
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
> > Starting Sink sink1
> > 2012-10-08 14:38:29,642 (lifecycleSupervisor-1-1) [INFO -
> >
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)]
> > Component type: SINK, name: sink1 started
> > 2012-10-08 14:38:29,643 (conf-file-poller-0) [INFO -
> >
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
> > Starting Source source1
> > 2012-10-08 14:38:29,643 (lifecycleSupervisor-1-0) [INFO -
> > org.apache.flume.source.ExecSource.start(ExecSource.java:155)] Exec
> source
> > starting with command:tail -F
> > /tank/clients/telescope/users/czhang/connecttwitter_dev.log
> > 2012-10-08 14:38:29,816 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:zookeeper.version=3.4.4-1386507, built on 09/17/2012 08:33
> GMT
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:host.name=app14.app.domain.com
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.version=1.7.0_02
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.vendor=Oracle Corporation
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.home=/usr/local/openjdk7/jre
> > 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> >
> environment:java.class.path=/app/flume/conf:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-avro-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-log4jappender-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-lang-2.5.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.2.7.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-scribe-source-1.3.0-20120929.012216-17.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-pool-1.5.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-ipc-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-thrift-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpclient-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/mina-core-2.0.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-mapper-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/servlet-api-2.5-20110124.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jcl-over-slf4j-1.7.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jsr305-1.3.9.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.4.0.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-util-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-hdfs-sink-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-collections-3.2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/async-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/paranamer-2.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/velocity-1.7.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-codec-1.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-hbase-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/irclib-1.10.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-jdbc-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-dbcp-1.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-api-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/protobuf-java-2.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-irc-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/joda-time-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-io-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-file-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-recoverable-memory-channel-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-cli-1.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/asynchbase-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/gson-2.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/derby-10.8.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-logging-1.1.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/guava-10.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-log4j12-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpcore-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/libthrift-0.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/log4j-1.2.16.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/zookeeper-3.4.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/snappy-java-1.0.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-core-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib:/app/hadoop-1.0.1/libexec/../conf:/usr/local/lib/tools.jar:/app/hadoop-1.0.1/libexec/..:/app/hadoop-1.0.1/libexec/../hadoop-core-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/asm-3.2.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjrt-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjtools-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-1.7.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-cli-1.2.jar:/app/hadoop-1.0.1/libexec/../lib/commons-codec-1.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-collections-3.2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-configuration-1.6.jar:/app/hadoop-1.0.1/libexec/../lib/commons-daemon-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-digester-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/commons-el-1.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-httpclient-3.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-lang-2.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-1.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-api-1.0.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-math-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-net-1.4.1.jar:/app/hadoop-1.0.1/libexec/../lib/core-3.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-capacity-scheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-fairscheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-thriftfs-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hsqldb-1.8.0.10.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-compiler-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-runtime-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jdeb-0.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-core-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-json-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-server-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jets3t-0.6.1.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-util-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jsch-0.1.42.jar:/app/hadoop-1.0.1/libexec/../lib/junit-4.5.jar:/app/hadoop-1.0.1/libexec/../lib/kfs-0.2.2.jar:/app/hadoop-1.0.1/libexec/../lib/log4j-1.2.15.jar:/app/hadoop-1.0.1/libexec/../lib/mockito-all-1.8.5.jar:/app/hadoop-1.0.1/libexec/../lib/oro-2.0.8.jar:/app/hadoop-1.0.1/libexec/../lib/servlet-api-2.5-20081211.jar:/app/hadoop-1.0.1/libexec/../lib/xmlenc-0.52.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/app/hbase-0.94.0/bin/../conf:/usr/local/lib/tools.jar:/app/hbase-0.94.0/bin/..:/app/hbase-0.94.0/bin/../hbase-0.94.0-tests.jar:/app/hbase-0.94.0/bin/../hbase-0.94.0.jar:/app/hbase-0.94.0/bin/../lib/activation-1.1.jar:/app/hbase-0.94.0/bin/../lib/asm-3.1.jar:/app/hbase-0.94.0/bin/../lib/avro-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/avro-ipc-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-1.7.0.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-core-1.8.0.jar:/app/hbase-0.94.0/bin/../lib/commons-cli-1.2.jar:/app/hbase-0.94.0/bin/../lib/commons-codec-1.4.jar:/app/hbase-0.94.0/bin/../lib/commons-collections-3.2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-configuration-1.6.jar:/app/hbase-0.94.0/bin/../lib/commons-digester-1.8.jar:/app/hbase-0.94.0/bin/../lib/commons-el-1.0.jar:/app/hbase-0.94.0/bin/../lib/commons-httpclient-3.1.jar:/app/hbase-0.94.0/bin/../lib/commons-io-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-lang-2.5.jar:/app/hbase-0.94.0/bin/../lib/commons-logging-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/commons-math-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-net-1.4.1.jar:/app/hbase-0.94.0/bin/../lib/core-3.1.1.jar:/app/hbase-0.94.0/bin/../lib/guava-r09.jar:/app/hbase-0.94.0/bin/../lib/hadoop-core-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/high-scale-lib-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/httpclient-4.1.2.jar:/app/hbase-0.94.0/bin/../lib/httpcore-4.1.3.jar:/app/hbase-0.94.0/bin/../lib/jackson-core-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-jaxrs-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-mapper-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-xc-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jamon-runtime-2.3.1.jar:/app/hbase-0.94.0/bin/../lib/jasper-compiler-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jasper-runtime-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jaxb-api-2.1.jar:/app/hbase-0.94.0/bin/../lib/jaxb-impl-2.1.12.jar:/app/hbase-0.94.0/bin/../lib/jersey-core-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-json-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-server-1.4.jar:/app/hbase-0.94.0/bin/../lib/jettison-1.1.jar:/app/hbase-0.94.0/bin/../lib/jetty-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jetty-util-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jruby-complete-1.6.5.jar:/app/hbase-0.94.0/bin/../lib/jsp-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/libthrift-0.8.0.jar:/app/hbase-0.94.0/bin/../lib/log4j-1.2.16.jar:/app/hbase-0.94.0/bin/../lib/netty-3.2.4.Final.jar:/app/hbase-0.94.0/bin/../lib/protobuf-java-2.4.0a.jar:/app/hbase-0.94.0/bin/../lib/servlet-api-2.5-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/snappy-java-1.0.3.2.jar:/app/hbase-0.94.0/bin/../lib/stax-api-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/velocity-1.7.jar:/app/hbase-0.94.0/bin/../lib/xmlenc-0.52.jar:/app/hbase-0.94.0/bin/../lib/zookeeper-3.4.3.jar:.:/app/hbase-0.94.0/conf
> > 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> >
> environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
> > 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.io.tmpdir=/var/tmp/
> > 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:java.compiler=<NA>
> > 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.name=FreeBSD
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.arch=amd64
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:os.version=9.0-RELEASE
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.name=czhang
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.home=/tank/clients/telescope/users/czhang
> > 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> > environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
> > 2012-10-08 14:38:29,822 (lifecycleSupervisor-1-1) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:29,863 (lifecycleSupervisor-1-1) [INFO -
> >
> org.hbase.async.HBaseClient$ZKClient.getDeferredRoot(HBaseClient.java:2238)]
> > Need to find the -ROOT- region
> > 2012-10-08 14:38:29,870 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:29,879 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:29,908 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdb5, negotiated timeout = 6000
> > 2012-10-08 14:38:33,906 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 4001ms for
> > sessionid 0x138a0620045bdb5, closing socket connection and attempting
> > reconnect
> > 2012-10-08 14:38:34,013 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> >
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> > Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> > 2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdb5 closed
> > 2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:34,971 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:34,972 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:34,986 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdba, negotiated timeout = 6000
> > 2012-10-08 14:38:39,088 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdba closed
> > 2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [WARN -
> > org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
> > longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
> > type:None path:null
> > 2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> > EventThread shut down
> > 2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> > EventThread shut down
> > 2012-10-08 14:38:39,091 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:39,092 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:39,163 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdbb, negotiated timeout = 6000
> > 2012-10-08 14:38:43,164 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> > Client session timed out, have not heard from server in 4001ms for
> > sessionid 0x138a0620045bdbb, closing socket connection and attempting
> > reconnect
> > 2012-10-08 14:38:43,265 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> >
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> > Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> > 2012-10-08 14:38:43,683 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdbb closed
> > 2012-10-08 14:38:43,684 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:43,686 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> >
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> > Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> > 2012-10-08 14:38:43,687 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:43,688 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:43,701 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdbc, negotiated timeout = 6000
> > 2012-10-08 14:38:47,802 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdbc closed
> > 2012-10-08 14:38:47,804 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:47,807 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> > EventThread shut down
> > 2012-10-08 14:38:47,810 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:47,812 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:47,821 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdbd, negotiated timeout = 6000
> > 2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> > 0x138a0620045bdbd closed
> > 2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [WARN -
> > org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
> > longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
> > type:None path:null
> > 2012-10-08 14:38:51,923 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> > client connection,
> connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000
> watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> > 2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [INFO -
> > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> > EventThread shut down
> > 2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> >
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> > Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> > 2012-10-08 14:38:51,925 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> > Opening socket connection to server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> > authenticate using SASL (unknown error)
> > 2012-10-08 14:38:51,926 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> > Socket connection established to
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> > 2012-10-08 14:38:51,945 (lifecycleSupervisor-1-1-SendThread(
> > hbasemaster0.hadoop.domain.com:2181)) [INFO -
> >
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> > Session establishment complete on server
> > hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> > 0x138a0620045bdbe, negotiated timeout = 6000
> >
> > Thanks for the continued help from the community on getting us going
> > w/flume
> >
> > Cheers,
> >
> > --
> > Iain Wright
> > Cell: (562) 852-5916
> >
> > <http://www.labctsi.org/>
> > This email message is confidential, intended only for the recipient(s)
> > named above and may contain information that is privileged, exempt from
> > disclosure under applicable law. If you are not the intended recipient,
> do
> > not disclose or disseminate the message to anyone except the intended
> > recipient. If you have received this message in error, or are not the
> named
> > recipient(s), please immediately notify the sender by return email, and
> > delete all copies of this message.
> >
>
--
Thanks & Regards,
Anil Gupta
Re: flume+hbase+zookeeper, can't hang on to zookeeper session
Posted by iain wright <ia...@gmail.com>.
Below is a debug level log of flume trying to connect. Could we
experiencing this bug: https://issues.apache.org/jira/browse/ZOOKEEPER-1550
We are using openjdk version "1.7.0_02"
OpenJDK Runtime Environment (build 1.7.0_02-b13)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
debug log:
2012-10-08 15:28:01,324 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2012-10-08 15:28:01,325 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:01,326 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:01,327 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:01,328 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:03,322 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:03,323 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:03,332 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:03,334 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:05,333 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:05,334 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:05,334 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
exception was thrown while closing send thread for session
0x138a0620045bf8f : Client session timed out, have not heard from server in
4009ms for sessionid 0x138a0620045bf8f
2012-10-08 15:28:05,410 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
Checking file:conf/brian.properties for changes
2012-10-08 15:28:05,435 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
Disconnecting client for session: 0x138a0620045bf8f
2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
0x138a0620045bf8f closed
2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
ZooKeeper#close completed in 4123621708ns
2012-10-08 15:28:05,436 (lifecycleSupervisor-1-0-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
2012-10-08 15:28:05,438 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.getRootRegion(HBaseClient.java:2512)]
Finding the -ROOT- region in ZooKeeper
2012-10-08 15:28:05,438 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
2012-10-08 15:28:05,439 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2314)] Done
handling ZooKeeper event: WatchedEvent state:SyncConnected type:None
path:null
2012-10-08 15:28:05,439 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
Ignore any DEBUG exception from ZooKeeper
2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-EventThread) [ERROR -
org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
2012-10-08 15:28:05,440 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
0x0
2012-10-08 15:28:05,442 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
for session: 0x0
2012-10-08 15:28:05,442 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
2012-10-08 15:28:05,443 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
Session establishment request sent on
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
2012-10-08 15:28:05,444 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:05,445 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:05,455 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
0x138a0620045bf90, negotiated timeout = 6000
2012-10-08 15:28:05,456 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2012-10-08 15:28:05,456 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:05,457 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:05,458 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:05,459 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:07,450 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:07,452 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:07,454 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:07,455 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:09,456 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:09,458 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:09,460 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:09,462 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:09,462 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
exception was thrown while closing send thread for session
0x138a0620045bf90 : Client session timed out, have not heard from server in
4003ms for sessionid 0x138a0620045bf90
2012-10-08 15:28:09,564 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
Disconnecting client for session: 0x138a0620045bf90
2012-10-08 15:28:09,566 (lifecycleSupervisor-1-0-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
0x138a0620045bf90 closed
2012-10-08 15:28:09,566 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
ZooKeeper#close completed in 4126386193ns
2012-10-08 15:28:09,567 (lifecycleSupervisor-1-0-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
Ignore any DEBUG exception from ZooKeeper
2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
2012-10-08 15:28:09,574 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
0x0
2012-10-08 15:28:09,575 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
for session: 0x0
2012-10-08 15:28:09,576 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 15:28:09,579 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
2012-10-08 15:28:09,580 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
Session establishment request sent on
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
2012-10-08 15:28:09,583 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:09,599 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
0x138a0620045bf91, negotiated timeout = 6000
2012-10-08 15:28:09,601 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2012-10-08 15:28:09,602 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:09,603 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:11,588 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:11,591 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:13,592 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:13,596 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:13,604 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:13,606 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1073)] An
exception was thrown while closing send thread for session
0x138a0620045bf91 : Client session timed out, have not heard from server in
4003ms for sessionid 0x138a0620045bf91
2012-10-08 15:28:13,708 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ClientCnxn.disconnect(ClientCnxn.java:1257)]
Disconnecting client for session: 0x138a0620045bf91
2012-10-08 15:28:13,710 (lifecycleSupervisor-1-0-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
0x138a0620045bf91 closed
2012-10-08 15:28:13,711 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2365)]
ZooKeeper#close completed in 4137007581ns
2012-10-08 15:28:13,712 (lifecycleSupervisor-1-0-EventThread) [WARN -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
type:None path:null
2012-10-08 15:28:13,713 (lifecycleSupervisor-1-0-EventThread) [INFO -
org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
client connection,
connectString=hbasemaster0.hadoop.telescope.tvsessionTimeout=5000
watcher=org.hbase.async.HBaseClient$ZKClient@19adc78d
2012-10-08 15:28:13,721 (lifecycleSupervisor-1-0-EventThread) [INFO -
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
EventThread shut down
2012-10-08 15:28:13,721 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.getRootRegion(HBaseClient.java:2512)]
Finding the -ROOT- region in ZooKeeper
2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
Opening socket connection to server
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181. Will not attempt to
authenticate using SASL (unknown error)
2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2314)] Done
handling ZooKeeper event: WatchedEvent state:SyncConnected type:None
path:null
2012-10-08 15:28:13,724 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.disconnectZK(HBaseClient.java:2362)]
Ignore any DEBUG exception from ZooKeeper
2012-10-08 15:28:13,727 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
ZooKeeper event: WatchedEvent state:Disconnected type:None path:null
2012-10-08 15:28:13,730 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
Socket connection established to
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, initiating session
2012-10-08 15:28:13,730 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:668)] Closing session:
0x0
2012-10-08 15:28:13,733 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:889)]
Session establishment request sent on
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181
2012-10-08 15:28:13,734 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1273)] Closing client
for session: 0x0
2012-10-08 15:28:13,761 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:13,772 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [INFO -
org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
Session establishment complete on server
hbasemaster0.hadoop.telescope.tv/10.10.11.30:2181, sessionid =
0x138a0620045bf92, negotiated timeout = 6000
2012-10-08 15:28:13,773 (lifecycleSupervisor-1-0-EventThread) [DEBUG -
org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2293)] Got
ZooKeeper event: WatchedEvent state:SyncConnected type:None path:null
2012-10-08 15:28:13,776 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:13,779 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
2012-10-08 15:28:13,783 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:/hbase/root-region-server
serverPath:/hbase/root-region-server finished:false header:: 0,4
replyHeader:: 0,0,0 request:: '/hbase/root-region-server,T response::
until SASL authentication completes.
2012-10-08 15:28:13,786 (lifecycleSupervisor-1-0-SendThread(
hbasemaster0.hadoop.telescope.tv:2181)) [DEBUG -
org.apache.zookeeper.ClientCnxnSocketNIO.findSendablePacket(ClientCnxnSocketNIO.java:164)]
deferring non-priming packet: clientPath:null serverPath:null
finished:false header:: 0,-11 replyHeader:: 0,0,0 request:: null
response:: nulluntil SASL authentication completes.
--
Iain Wright
Cell: (562) 852-5916
<http://www.labctsi.org/>
This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.
On Mon, Oct 8, 2012 at 2:41 PM, iain wright <ia...@gmail.com> wrote:
> We're having some trouble with the Flume & the HbaseSink working. Seems we
> cannot hang on to zookeeper sessions. The flume list recommended I cross
> post here as this seems to be something up with the hbaseclient &/or
> zookeeper. In all honesty I am probably just doing something dumb and not
> seeing it.
>
> Using latest flume-ng 1.3.0, hbase 0.94.0. Hbase & ZK both look fine and
> are in normal prod use. I don't think we are hitting our
> hbase.zookeeper.property.maxClientCnxns as we only have about 95 sessions
> in ZK DUMP and I believe it defaults to 2k. I can establish new sessions
> using the CLI from the same server and idle there for 10 minutes without
> getting dropped. When I start up flume I see the session show up in the ZK
> DUMP, see active TCP connection on the port etc., it just keeps
> dropping/reconnecting.
>
> Flume & zookeeper logs below, using rsync'd copies of our hadoop & hbase
> directories from one of our datanode+regionserver's.
>
> *Searched the flume list, this user appeared to have the same problem,
> not sure how he fixed it though:*
>
> http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%3CCADGpRhcgztwhuUJi1izMRX2UOKeAWwhYOkKSKkHYmKopAkwHAQ@mail.gmail.com%3E
>
> *startup cmd
> */app/apache-flume-1.3.0-SNAPSHOT/bin/flume-ng agent -n agent1 -c ./conf
> -f conf/brian.properties -Dflume.root.logger=INFO,console*
> *
> *flume-env.sh*
> $ cat conf/flume-env.sh
>
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
> # during Flume startup.
>
> # Enviroment variables can be set here.
>
> #JAVA_HOME=/usr/lib/jvm/java-6-sun
>
> # Give Flume more memory and pre-allocate, enable remote monitoring via JMX
> #JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"
>
> # Note that the Flume conf directory is always included in the classpath.
> #FLUME_CLASSPATH="/app/flume/lib"
>
> FLUME_CLASSPATH="/app/apache-flume-1.3.0-SNAPSHOT/lib"
>
> HBASE_HOME="/app/hbase-0.94.0"
>
> HADOOP_HOME="/app/hadoop-1.0.1"
>
>
> *config*
> $ cat conf/brian.properties
> #example.conf: A single-node Flume configuration
>
> # Name the components on this agent
> agent1.sources = source1
> agent1.sinks = sink1
> agent1.channels = channel1
>
> # Describe/configure source1
> agent1.sources.source1.type = exec
> agent1.sources.source1.command = tail -F /tank/log_dev.log
> agent1.sources.source1.batchSize = 1
> # Describe sink1
> #agent1.sinks.sink1.type = logger
> agent1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
> agent1.sinks.sink1.table = brian_test
> agent1.sinks.sink1.columnFamily = f1
> #agent1.sinks.sink1.serializer =
> org.apache.flume.sink.hbase.SimpleHBaseEventSerializer
> #agent1.sinks.sink1.serializer =
> org.apache.flume.sink.hbase.SimpleHbaseEventSerializer
>
>
> # Use a channel which buffers events in memory
> agent1.channels.channel1.type = memory
> agent1.channels.channel1.capacity = 1000
> agent1.channels.channel1.transactionCapactiy = 100
>
> # Bind the source and sink to the channel
> agent1.sources.source1.channels = channel1
> agent1.sinks.sink1.channel = channel1
>
>
> *flume console log*
> 2012-10-08 12:06:19,007 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
> 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.io.tmpdir=/var/tmp/
> 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.compiler=<NA>
> 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.name=FreeBSD
> 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.arch=amd64
> 2012-10-08 12:06:19,008 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.version=9.0-RELEASE
> 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.name=czhang
> 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.home=/users/czhang
> 2012-10-08 12:06:19,009 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
> 2012-10-08 12:06:19,010 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection, connectString=hbasemaster0.hadoop.domain.com:2181sessionTimeout=180000 watcher=hconnection
> 2012-10-08 12:06:19,036 (lifecycleSupervisor-1-1) [INFO -
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:97)]
> The identifier of this process is 12494@app14.app.domain.com
> 2012-10-08 12:06:19,041 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 12:06:19,049 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 12:06:19,072 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bbce, negotiated timeout = 180000
> 2012-10-08 12:08:19,070 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> Client session timed out, have not heard from server in 120001ms for
> sessionid 0x138a0620045bbce, closing socket connection and attempting
> reconnect
> 2012-10-08 12:08:19,233 (lifecycleSupervisor-1-1) [WARN -
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2012-10-08 12:08:19,234 (lifecycleSupervisor-1-1) [INFO -
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> Sleeping 2000ms before retry #1...
> 2012-10-08 12:08:20,691 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 12:08:20,692 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 12:08:20,695 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bbce, negotiated timeout = 180000
> 2012-10-08 12:10:20,695 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> Client session timed out, have not heard from server in 120000ms for
> sessionid 0x138a0620045bbce, closing socket connection and attempting
> reconnect
> 2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [WARN -
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2012-10-08 12:10:20,796 (lifecycleSupervisor-1-1) [INFO -
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> Sleeping 4000ms before retry #2...
> 2012-10-08 12:10:22,792 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 12:10:22,794 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 12:10:22,799 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bbce, negotiated timeout = 180000
> 2012-10-08 12:12:22,800 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> Client session timed out, have not heard from server in 120001ms for
> sessionid 0x138a0620045bbce, closing socket connection and attempting
> reconnect
> 2012-10-08 12:12:22,902 (lifecycleSupervisor-1-1) [WARN -
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2012-10-08 12:12:22,903 (lifecycleSupervisor-1-1) [INFO -
> org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:53)]
> Sleeping 8000ms before retry #3...
> 2012-10-08 12:12:24,289 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 12:12:24,291 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 12:12:24,294 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bbce, negotiated timeout = 180000
> 2012-10-08 12:14:24,294 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> Client session timed out, have not heard from server in 120000ms for
> sessionid 0x138a0620045bbce, closing socket connection and attempting
> reconnect
> 2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [WARN -
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:195)]
> Possibly transient ZooKeeper exception:
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> 2012-10-08 12:14:24,395 (lifecycleSupervisor-1-1) [ERROR -
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.retryOrThrow(RecoverableZooKeeper.java:197)]
> ZooKeeper exists failed after 3 retries
> 2012-10-08 12:14:24,398 (lifecycleSupervisor-1-1) [WARN -
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:239)]
> hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce
> Unable to set watcher on znode /hbase/master
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> at
> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2012-10-08 12:14:24,405 (lifecycleSupervisor-1-1) [ERROR -
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.keeperException(ZooKeeperWatcher.java:408)]
> hconnection-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce-0x138a0620045bbce
> Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:150)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:590)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:818)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> at
> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> at org.apache.flume.sink.hbase.HBaseSink.start(HBaseSink.java:107)
> at
> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2012-10-08 12:14:24,407 (lifecycleSupervisor-1-1) [INFO -
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1654)]
> This client just lost it's session with ZooKeeper, will automatically
> reconnect when needed.
> 2012-10-08 12:14:26,166 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 12:14:26,167 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 12:14:26,170 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bbce, negotiated timeout = 180000
>
> *zookeeper log*
> 2012-10-08 19:06:19,049 INFO
> org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> connection from /10.10.10.64:16755
> 2012-10-08 19:06:19,053 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Client attempting to establish new session at /10.10.10.64:16755
> 2012-10-08 19:06:19,069 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Established session 0x138a0620045bbce with negotiated timeout 180000 for
> client /10.10.10.64:16755
> 2012-10-08 19:08:19,072 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /10.10.10.64:16755 which had
> sessionid 0x138a0620045bbce
> 2012-10-08 19:08:20,692 INFO
> org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> connection from /10.10.10.64:33245
> 2012-10-08 19:08:20,693 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Client attempting to renew session 0x138a0620045bbce at /10.10.10.64:33245
> 2012-10-08 19:08:20,694 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Established session 0x138a0620045bbce with negotiated timeout 180000 for
> client /10.10.10.64:33245
> 2012-10-08 19:10:20,695 WARN org.apache.zookeeper.server.NIOServerCnxn:
> caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x138a0620045bbce, likely client has closed socket
> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
> at java.lang.Thread.run(Thread.java:619)
> 2012-10-08 19:10:20,698 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /10.10.10.64:33245 which had
> sessionid 0x138a0620045bbce
> 2012-10-08 19:10:22,794 INFO
> org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket
> connection from /10.10.10.64:44993
> 2012-10-08 19:10:22,796 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Client attempting to renew session 0x138a0620045bbce at /10.10.10.64:44993
> 2012-10-08 19:10:22,798 INFO org.apache.zookeeper.server.ZooKeeperServer:
> Established session 0x138a0620045bbce with negotiated timeout 180000 for
> client /10.10.10.64:44993
>
> *We also tried the hbase async sink as that is recommended over in the
> flume group, and got similar ** results (but seemingly more serious
> CONNECTIONLOSS errors)*
>
> 2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:106)]
> Starting new configuration:{
> sourceRunners:{source1=EventDrivenSourceRunner: {
> source:org.apache.flume.source.ExecSource{name:source1,state:IDLE} }}
> sinkRunners:{sink1=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@3968a6fe counterGroup:{
> name:null counters:{} } }}
> channels:{channel1=org.apache.flume.channel.MemoryChannel{name: channel1}} }
> 2012-10-08 14:38:29,640 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:113)]
> Starting Channel channel1
> 2012-10-08 14:38:29,641 (lifecycleSupervisor-1-0) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)]
> Component type: CHANNEL, name: channel1 started
> 2012-10-08 14:38:29,642 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:141)]
> Starting Sink sink1
> 2012-10-08 14:38:29,642 (lifecycleSupervisor-1-1) [INFO -
> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)]
> Component type: SINK, name: sink1 started
> 2012-10-08 14:38:29,643 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
> Starting Source source1
> 2012-10-08 14:38:29,643 (lifecycleSupervisor-1-0) [INFO -
> org.apache.flume.source.ExecSource.start(ExecSource.java:155)] Exec source
> starting with command:tail -F
> /tank/clients/telescope/users/czhang/connecttwitter_dev.log
> 2012-10-08 14:38:29,816 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:zookeeper.version=3.4.4-1386507, built on 09/17/2012 08:33 GMT
> 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:host.name=app14.app.domain.com
> 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.version=1.7.0_02
> 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.vendor=Oracle Corporation
> 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.home=/usr/local/openjdk7/jre
> 2012-10-08 14:38:29,817 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.class.path=/app/flume/conf:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-avro-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-log4jappender-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-lang-2.5.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.2.7.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-scribe-source-1.3.0-20120929.012216-17.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-pool-1.5.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/avro-ipc-1.7.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-thrift-source-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpclient-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-core-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/mina-core-2.0.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-mapper-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/servlet-api-2.5-20110124.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jcl-over-slf4j-1.7.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jsr305-1.3.9.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/netty-3.4.0.Final.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-util-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-hdfs-sink-1.3.0-20120929.012214-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-collections-3.2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/async-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/paranamer-2.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/velocity-1.7.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-codec-1.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-hbase-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/irclib-1.10.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-jdbc-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-dbcp-1.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-api-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/protobuf-java-2.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-irc-sink-1.3.0-20120929.012215-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-configuration-1.3.0-20120929.012205-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/joda-time-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-io-2.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-file-channel-1.3.0-20120929.012212-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-node-1.3.0-20120929.012209-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-recoverable-memory-channel-1.3.0-20120929.012213-20.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-cli-1.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jetty-6.1.26.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/asynchbase-1.2.0.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/gson-2.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/derby-10.8.2.2.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/commons-logging-1.1.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/guava-10.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/flume-ng-sdk-1.3.0-SNAPSHOT.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/slf4j-log4j12-1.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/httpcore-4.0.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/libthrift-0.6.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/log4j-1.2.16.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/zookeeper-3.4.4.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/snappy-java-1.0.4.1.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib/jackson-core-asl-1.9.3.jar:/app/apache-flume-1.3.0-SNAPSHOT/lib:/app/hadoop-1.0.1/libexec/../conf:/usr/local/lib/tools.jar:/app/hadoop-1.0.1/libexec/..:/app/hadoop-1.0.1/libexec/../hadoop-core-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/asm-3.2.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjrt-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/aspectjtools-1.6.5.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-1.7.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-cli-1.2.jar:/app/hadoop-1.0.1/libexec/../lib/commons-codec-1.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-collections-3.2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-configuration-1.6.jar:/app/hadoop-1.0.1/libexec/../lib/commons-daemon-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-digester-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/commons-el-1.0.jar:/app/hadoop-1.0.1/libexec/../lib/commons-httpclient-3.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-lang-2.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-1.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-logging-api-1.0.4.jar:/app/hadoop-1.0.1/libexec/../lib/commons-math-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/commons-net-1.4.1.jar:/app/hadoop-1.0.1/libexec/../lib/core-3.1.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-capacity-scheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-fairscheduler-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hadoop-thriftfs-1.0.1.jar:/app/hadoop-1.0.1/libexec/../lib/hsqldb-1.8.0.10.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-compiler-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jasper-runtime-5.5.12.jar:/app/hadoop-1.0.1/libexec/../lib/jdeb-0.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-core-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-json-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jersey-server-1.8.jar:/app/hadoop-1.0.1/libexec/../lib/jets3t-0.6.1.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jetty-util-6.1.26.jar:/app/hadoop-1.0.1/libexec/../lib/jsch-0.1.42.jar:/app/hadoop-1.0.1/libexec/../lib/junit-4.5.jar:/app/hadoop-1.0.1/libexec/../lib/kfs-0.2.2.jar:/app/hadoop-1.0.1/libexec/../lib/log4j-1.2.15.jar:/app/hadoop-1.0.1/libexec/../lib/mockito-all-1.8.5.jar:/app/hadoop-1.0.1/libexec/../lib/oro-2.0.8.jar:/app/hadoop-1.0.1/libexec/../lib/servlet-api-2.5-20081211.jar:/app/hadoop-1.0.1/libexec/../lib/xmlenc-0.52.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/app/hadoop-1.0.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar:/app/hbase-0.94.0/bin/../conf:/usr/local/lib/tools.jar:/app/hbase-0.94.0/bin/..:/app/hbase-0.94.0/bin/../hbase-0.94.0-tests.jar:/app/hbase-0.94.0/bin/../hbase-0.94.0.jar:/app/hbase-0.94.0/bin/../lib/activation-1.1.jar:/app/hbase-0.94.0/bin/../lib/asm-3.1.jar:/app/hbase-0.94.0/bin/../lib/avro-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/avro-ipc-1.5.3.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-1.7.0.jar:/app/hbase-0.94.0/bin/../lib/commons-beanutils-core-1.8.0.jar:/app/hbase-0.94.0/bin/../lib/commons-cli-1.2.jar:/app/hbase-0.94.0/bin/../lib/commons-codec-1.4.jar:/app/hbase-0.94.0/bin/../lib/commons-collections-3.2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-configuration-1.6.jar:/app/hbase-0.94.0/bin/../lib/commons-digester-1.8.jar:/app/hbase-0.94.0/bin/../lib/commons-el-1.0.jar:/app/hbase-0.94.0/bin/../lib/commons-httpclient-3.1.jar:/app/hbase-0.94.0/bin/../lib/commons-io-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-lang-2.5.jar:/app/hbase-0.94.0/bin/../lib/commons-logging-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/commons-math-2.1.jar:/app/hbase-0.94.0/bin/../lib/commons-net-1.4.1.jar:/app/hbase-0.94.0/bin/../lib/core-3.1.1.jar:/app/hbase-0.94.0/bin/../lib/guava-r09.jar:/app/hbase-0.94.0/bin/../lib/hadoop-core-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/high-scale-lib-1.1.1.jar:/app/hbase-0.94.0/bin/../lib/httpclient-4.1.2.jar:/app/hbase-0.94.0/bin/../lib/httpcore-4.1.3.jar:/app/hbase-0.94.0/bin/../lib/jackson-core-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-jaxrs-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-mapper-asl-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jackson-xc-1.5.5.jar:/app/hbase-0.94.0/bin/../lib/jamon-runtime-2.3.1.jar:/app/hbase-0.94.0/bin/../lib/jasper-compiler-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jasper-runtime-5.5.23.jar:/app/hbase-0.94.0/bin/../lib/jaxb-api-2.1.jar:/app/hbase-0.94.0/bin/../lib/jaxb-impl-2.1.12.jar:/app/hbase-0.94.0/bin/../lib/jersey-core-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-json-1.4.jar:/app/hbase-0.94.0/bin/../lib/jersey-server-1.4.jar:/app/hbase-0.94.0/bin/../lib/jettison-1.1.jar:/app/hbase-0.94.0/bin/../lib/jetty-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jetty-util-6.1.26.jar:/app/hbase-0.94.0/bin/../lib/jruby-complete-1.6.5.jar:/app/hbase-0.94.0/bin/../lib/jsp-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/libthrift-0.8.0.jar:/app/hbase-0.94.0/bin/../lib/log4j-1.2.16.jar:/app/hbase-0.94.0/bin/../lib/netty-3.2.4.Final.jar:/app/hbase-0.94.0/bin/../lib/protobuf-java-2.4.0a.jar:/app/hbase-0.94.0/bin/../lib/servlet-api-2.5-6.1.14.jar:/app/hbase-0.94.0/bin/../lib/snappy-java-1.0.3.2.jar:/app/hbase-0.94.0/bin/../lib/stax-api-1.0.1.jar:/app/hbase-0.94.0/bin/../lib/velocity-1.7.jar:/app/hbase-0.94.0/bin/../lib/xmlenc-0.52.jar:/app/hbase-0.94.0/bin/../lib/zookeeper-3.4.3.jar:.:/app/hbase-0.94.0/conf
> 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.library.path=:/app/hadoop-1.0.1/libexec/../lib/native/FreeBSD-amd64-64:/app/hbase-0.94.0/bin/../lib/native/FreeBSD-amd64-64
> 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.io.tmpdir=/var/tmp/
> 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:java.compiler=<NA>
> 2012-10-08 14:38:29,819 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.name=FreeBSD
> 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.arch=amd64
> 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:os.version=9.0-RELEASE
> 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.name=czhang
> 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.home=/tank/clients/telescope/users/czhang
> 2012-10-08 14:38:29,820 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.Environment.logEnv(Environment.java:100)] Client
> environment:user.dir=/app/apache-flume-1.3.0-SNAPSHOT
> 2012-10-08 14:38:29,822 (lifecycleSupervisor-1-1) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> 2012-10-08 14:38:29,863 (lifecycleSupervisor-1-1) [INFO -
> org.hbase.async.HBaseClient$ZKClient.getDeferredRoot(HBaseClient.java:2238)]
> Need to find the -ROOT- region
> 2012-10-08 14:38:29,870 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 14:38:29,879 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 14:38:29,908 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bdb5, negotiated timeout = 6000
> 2012-10-08 14:38:33,906 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> Client session timed out, have not heard from server in 4001ms for
> sessionid 0x138a0620045bdb5, closing socket connection and attempting
> reconnect
> 2012-10-08 14:38:34,013 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> 2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bdb5 closed
> 2012-10-08 14:38:34,969 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> 2012-10-08 14:38:34,971 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 14:38:34,972 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 14:38:34,986 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bdba, negotiated timeout = 6000
> 2012-10-08 14:38:39,088 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bdba closed
> 2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [WARN -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
> longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
> type:None path:null
> 2012-10-08 14:38:39,089 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> 2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> EventThread shut down
> 2012-10-08 14:38:39,090 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> EventThread shut down
> 2012-10-08 14:38:39,091 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 14:38:39,092 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 14:38:39,163 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bdbb, negotiated timeout = 6000
> 2012-10-08 14:38:43,164 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1083)]
> Client session timed out, have not heard from server in 4001ms for
> sessionid 0x138a0620045bdbb, closing socket connection and attempting
> reconnect
> 2012-10-08 14:38:43,265 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> 2012-10-08 14:38:43,683 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bdbb closed
> 2012-10-08 14:38:43,684 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> 2012-10-08 14:38:43,686 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> 2012-10-08 14:38:43,687 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 14:38:43,688 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 14:38:43,701 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bdbc, negotiated timeout = 6000
> 2012-10-08 14:38:47,802 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bdbc closed
> 2012-10-08 14:38:47,804 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> 2012-10-08 14:38:47,807 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> EventThread shut down
> 2012-10-08 14:38:47,810 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 14:38:47,812 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 14:38:47,821 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bdbd, negotiated timeout = 6000
> 2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:679)] Session:
> 0x138a0620045bdbd closed
> 2012-10-08 14:38:51,922 (lifecycleSupervisor-1-1-EventThread) [WARN -
> org.hbase.async.HBaseClient$ZKClient.process(HBaseClient.java:2304)] No
> longer connected to ZooKeeper, event=WatchedEvent state:Disconnected
> type:None path:null
> 2012-10-08 14:38:51,923 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:433)] Initiating
> client connection, connectString=hbasemaster0.hadoop.domain.comsessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@4d1c2914
> 2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [INFO -
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:509)]
> EventThread shut down
> 2012-10-08 14:38:51,924 (lifecycleSupervisor-1-1-EventThread) [ERROR -
> org.hbase.async.HBaseClient$ZKClient$2.processResult(HBaseClient.java:2407)]
> Looks like our ZK session expired or is broken, rc=-4: CONNECTIONLOSS
> 2012-10-08 14:38:51,925 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.logStartConnect(ClientCnxn.java:966)]
> Opening socket connection to server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2012-10-08 14:38:51,926 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:849)]
> Socket connection established to
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, initiating session
> 2012-10-08 14:38:51,945 (lifecycleSupervisor-1-1-SendThread(
> hbasemaster0.hadoop.domain.com:2181)) [INFO -
> org.apache.zookeeper.ClientCnxn$SendThread.onConnected(ClientCnxn.java:1207)]
> Session establishment complete on server
> hbasemaster0.hadoop.domain.com/10.10.11.30:2181, sessionid =
> 0x138a0620045bdbe, negotiated timeout = 6000
>
> Thanks for the continued help from the community on getting us going
> w/flume
>
> Cheers,
>
> --
> Iain Wright
> Cell: (562) 852-5916
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>