You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2010/12/03 20:47:12 UTC
[jira] Commented: (CHUKWA-563) Collector Crashes when HBase is not
started
[ https://issues.apache.org/jira/browse/CHUKWA-563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966644#action_12966644 ]
Eric Yang commented on CHUKWA-563:
----------------------------------
HBaseWriter runs a schema validation to make sure there is no misconfiguration. The schema validation is currently mandatory. If the validation does not pass, collector will bail out. We can add optional parameter to turn off schema validation.
> Collector Crashes when HBase is not started
> -------------------------------------------
>
> Key: CHUKWA-563
> URL: https://issues.apache.org/jira/browse/CHUKWA-563
> Project: Chukwa
> Issue Type: Bug
> Components: data collection
> Affects Versions: 0.4.0
> Environment: Ubuntu 8.04
> Reporter: Ahmed Fathalla
> Fix For: 0.5.0
>
>
> When using HBaseWriter, collectors crash if they start and do not find HBase running. Collectors also crash if HBase is running and Tables are not found
> Here are the exceptions in collector.log when HBase is not running
> 2010-12-03 18:20:38,857 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
> 2010-12-03 18:20:39,188 INFO main root - initing servletCollector
> 2010-12-03 18:20:39,190 INFO main PipelineStageWriter - using pipelined writers, pipe length is 2
> 2010-12-03 18:20:39,198 INFO Thread-6 SocketTeeWriter - listen thread started
> 2010-12-03 18:20:39,200 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
> 2010-12-03 18:20:39,396 INFO main ZooKeeper - Client environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
> 2010-12-03 18:20:39,396 INFO main ZooKeeper - Client environment:host.name=ahmed-laptop
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.version=1.6.0_20
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.20/jre
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.class.path=/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf:/usr/local/hadoop/conf::/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../chukwa-core-0.5.0.jar:/usr/local/hadoop/hadoop-0.20.2-core.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-core-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-protobuf-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/aopalliance-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/asm-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-beanutils-1.8.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-codec-1.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-fileupload-1.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-httpclient-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-io-1.4.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-lang-2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-api-1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-net-2.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/confspellcheck.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/core-3.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/ezmorph-1.0.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-j2ee-management_1.0_spec-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-jms_1.1_spec-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/guava-r05.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6-test.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/html-filter-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jchronic-0.2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jersey-bundle-1.1.0-ea.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-util-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json-lib-2.2.3-jdk15.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-api-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsr311-api-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/junit-3.8.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/kahadb-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/log4j-1.2.14.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/mysql-connector-java-5.1.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/NagiosAppender-1.5.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/org.osgi.core-4.1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/prefuse.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/servlet-api-2.5-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/sigar.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-beans-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-context-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-core-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-mock-1.2.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-core-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-io-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/zookeeper-3.2.2.jar:
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.library.path=/usr/local/hadoop/lib/native/Linux-i386-32
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.io.tmpdir=/tmp
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.compiler=<NA>
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.name=Linux
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.arch=i386
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.version=2.6.24-19-generic
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.name=hadoop
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.home=/home/hadoop
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.dir=/home/hadoop/workspace/chukwa-0.5.0-incubating
> 2010-12-03 18:20:39,398 INFO main ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@1428ea
> 2010-12-03 18:20:39,400 INFO main ClientCnxn - zookeeper.disableAutoWatchReset is false
> 2010-12-03 18:20:39,418 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:39,422 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a68ef9
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:39,423 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:39,423 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:39,545 INFO main HConnectionManager$TableServers - getMaster attempt 0 of 10 failed; retrying after sleep of 1000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:40,188 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2010-12-03 18:20:40,312 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:20:40,640 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1977b9b
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:40,742 INFO main HConnectionManager$TableServers - getMaster attempt 1 of 10 failed; retrying after sleep of 1000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:42,329 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@2e1f1f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:42,431 INFO main HConnectionManager$TableServers - getMaster attempt 2 of 10 failed; retrying after sleep of 1000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:43,903 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@fefe3f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:44,007 INFO main HConnectionManager$TableServers - getMaster attempt 3 of 10 failed; retrying after sleep of 2000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:45,178 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:45,178 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@c2b2f6
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:45,179 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:45,179 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:46,610 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:46,610 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a001ff
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:46,611 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:46,611 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:46,711 INFO main HConnectionManager$TableServers - getMaster attempt 4 of 10 failed; retrying after sleep of 2000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:48,252 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:48,252 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@90c06f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:48,253 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:48,253 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:49,973 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:49,973 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1267649
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:49,973 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:49,974 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:50,074 INFO main HConnectionManager$TableServers - getMaster attempt 5 of 10 failed; retrying after sleep of 4000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:50,312 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:20:51,749 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@dcb03b
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:53,639 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1b1fbf4
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:55,094 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:55,094 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1829e6f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:55,094 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:55,095 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:55,195 INFO main HConnectionManager$TableServers - getMaster attempt 6 of 10 failed; retrying after sleep of 4000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:56,869 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:56,869 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@100ebec
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:56,869 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:56,870 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:58,099 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7736bd
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:59,363 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@178460d
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:59,465 INFO main HConnectionManager$TableServers - getMaster attempt 7 of 10 failed; retrying after sleep of 8000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:21:00,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:21:00,703 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1938039
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:02,221 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@3c342b
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:03,664 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@419d05
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:04,922 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:04,922 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1deeb40
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:04,923 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:04,923 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:06,779 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:06,779 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@cafb56
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:06,779 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:06,780 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:07,942 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:07,943 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a7ddcf
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:07,944 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:07,944 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:08,044 INFO main HConnectionManager$TableServers - getMaster attempt 8 of 10 failed; retrying after sleep of 16000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:21:09,964 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:09,965 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1da669c
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:09,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:09,966 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:10,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:21:11,285 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@17f242c
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:12,919 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:12,920 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@12f9ee
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:12,920 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:12,921 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:14,754 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@106fc94
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:15,947 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7f5580
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:17,483 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@84cc09
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:18,793 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@14db0e3
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:19,964 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@676437
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:20,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:21:21,899 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1e808ca
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:23,916 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@2bd3a
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:25,857 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:25,857 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1329642
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:25,858 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:25,858 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:25,958 INFO main HConnectionManager$TableServers - getMaster attempt 9 of 10 failed; no more retrying.
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:21:25,961 ERROR main HBaseWriter - org.apache.hadoop.hbase.MasterNotRunningException
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> 2010-12-03 18:21:25,961 ERROR main HBaseWriter - Hbase schema mismatch with demux parser.
> 2010-12-03 18:21:25,961 ERROR main HBaseWriter - Exiting...
> 2010-12-03 18:21:25,963 INFO HCM.shutdownHook ZooKeeper - Closing session: 0x0
> 2010-12-03 18:21:25,964 INFO HCM.shutdownHook ClientCnxn - Closing ClientCnxn for session: 0x0
> 2010-12-03 18:21:27,098 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:27,099 INFO main-SendThread ClientCnxn - Exception while closing send thread for session 0x0 : Connection refused
> 2010-12-03 18:21:27,099 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
> 2010-12-03 18:21:27,099 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
> 2010-12-03 18:21:27,199 INFO HCM.shutdownHook ClientCnxn - Disconnecting ClientCnxn for session: 0x0
> 2010-12-03 18:21:27,200 INFO HCM.shutdownHook ZooKeeper - Session: 0x0 closed
> 2010-12-03 18:21:27,200 INFO main-EventThread ClientCnxn - EventThread shut down
> Here are the exceptions when HBase is running but tables do not exist
> 010-12-03 18:24:40,846 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
> 2010-12-03 18:24:41,230 INFO main root - initing servletCollector
> 2010-12-03 18:24:41,232 INFO main PipelineStageWriter - using pipelined writers, pipe length is 2
> 2010-12-03 18:24:41,235 INFO Thread-6 SocketTeeWriter - listen thread started
> 2010-12-03 18:24:41,238 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:host.name=ahmed-laptop
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.version=1.6.0_20
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.20/jre
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.class.path=/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf:/usr/local/hadoop/conf::/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../chukwa-core-0.5.0.jar:/usr/local/hadoop/hadoop-0.20.2-core.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-core-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-protobuf-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/aopalliance-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/asm-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-beanutils-1.8.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-codec-1.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-fileupload-1.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-httpclient-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-io-1.4.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-lang-2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-api-1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-net-2.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/confspellcheck.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/core-3.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/ezmorph-1.0.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-j2ee-management_1.0_spec-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-jms_1.1_spec-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/guava-r05.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6-test.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/html-filter-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jchronic-0.2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jersey-bundle-1.1.0-ea.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-util-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json-lib-2.2.3-jdk15.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-api-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsr311-api-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/junit-3.8.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/kahadb-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/log4j-1.2.14.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/mysql-connector-java-5.1.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/NagiosAppender-1.5.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/org.osgi.core-4.1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/prefuse.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/servlet-api-2.5-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/sigar.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-beans-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-context-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-core-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-mock-1.2.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-core-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-io-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/zookeeper-3.2.2.jar:
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.library.path=/usr/local/hadoop/lib/native/Linux-i386-32
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:java.io.tmpdir=/tmp
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:java.compiler=<NA>
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.name=Linux
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.arch=i386
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.version=2.6.24-19-generic
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.name=hadoop
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.home=/home/hadoop
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.dir=/home/hadoop/workspace/chukwa-0.5.0-incubating
> 2010-12-03 18:24:41,420 INFO main ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@1326484
> 2010-12-03 18:24:41,422 INFO main ClientCnxn - zookeeper.disableAutoWatchReset is false
> 2010-12-03 18:24:41,440 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:24:41,442 INFO main-SendThread ClientCnxn - Priming connection to java.nio.channels.SocketChannel[connected local=/127.0.0.1:58057 remote=localhost/127.0.0.1:2181]
> 2010-12-03 18:24:41,500 INFO main-SendThread ClientCnxn - Server connection successful
> 2010-12-03 18:24:41,984 ERROR main HBaseWriter - java.lang.Exception: HBase table: Hadoop does not exist.
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseTable(HBaseWriter.java:133)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:158)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> 2010-12-03 18:24:41,984 ERROR main HBaseWriter - java.lang.Exception: Validation failed - table: Hadoop column family: ClientTrace does not exist.
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:159)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> 2010-12-03 18:24:41,984 ERROR main HBaseWriter - Hbase schema mismatch with demux parser.
> 2010-12-03 18:24:41,984 ERROR main HBaseWriter - Exiting...
> 2010-12-03 18:24:42,087 INFO HCM.shutdownHook ZooKeeper - Closing session: 0x12cad0c07fd0002
> 2010-12-03 18:24:42,087 INFO HCM.shutdownHook ClientCnxn - Closing ClientCnxn for session: 0x12cad0c07fd0002
> 2010-12-03 18:24:42,094 INFO main-SendThread ClientCnxn - Exception while closing send thread for session 0x12cad0c07fd0002 : Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> 2010-12-03 18:24:42,202 INFO HCM.shutdownHook ClientCnxn - Disconnecting ClientCnxn for session: 0x12cad0c07fd0002
> 2010-12-03 18:24:42,202 INFO HCM.shutdownHook ZooKeeper - Session: 0x12cad0c07fd0002 closed
> 2010-12-03 18:24:42,202 INFO main-EventThread ClientCnxn - EventThread shut down
> 2010-12-03 18:24:42,231 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.