You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2010/12/05 02:34:10 UTC
[jira] Updated: (CHUKWA-563) Collector Crashes when HBase is not
started
[ https://issues.apache.org/jira/browse/CHUKWA-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-563:
-----------------------------
Attachment: CHUKWA-563.patch
Added option:
- hbase.writer.verify.schema - Verify HBase Table schema with demux parser schema, log warning if there are mismatch between hbase schema and demux parsers.
- hbase.writer.halt.on.schema.mismatch - If this option is set to true, and HBase table schema is mismatched with demux parser, collector will shut down itself.
> Collector Crashes when HBase is not started
> -------------------------------------------
>
> Key: CHUKWA-563
> URL: https://issues.apache.org/jira/browse/CHUKWA-563
> Project: Chukwa
> Issue Type: Bug
> Components: data collection
> Affects Versions: 0.4.0
> Environment: Ubuntu 8.04
> Reporter: Ahmed Fathalla
> Fix For: 0.5.0
>
> Attachments: CHUKWA-563.patch
>
>
> When using HBaseWriter, collectors crash if they start and do not find HBase running. Collectors also crash if HBase is running and Tables are not found
> Here are the exceptions in collector.log when HBase is not running
> 2010-12-03 18:20:38,857 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
> 2010-12-03 18:20:39,188 INFO main root - initing servletCollector
> 2010-12-03 18:20:39,190 INFO main PipelineStageWriter - using pipelined writers, pipe length is 2
> 2010-12-03 18:20:39,198 INFO Thread-6 SocketTeeWriter - listen thread started
> 2010-12-03 18:20:39,200 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
> 2010-12-03 18:20:39,396 INFO main ZooKeeper - Client environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
> 2010-12-03 18:20:39,396 INFO main ZooKeeper - Client environment:host.name=ahmed-laptop
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.version=1.6.0_20
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.20/jre
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.class.path=/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf:/usr/local/hadoop/conf::/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../chukwa-core-0.5.0.jar:/usr/local/hadoop/hadoop-0.20.2-core.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-core-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-protobuf-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/aopalliance-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/asm-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-beanutils-1.8.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-codec-1.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-fileupload-1.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-httpclient-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-io-1.4.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-lang-2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-api-1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-net-2.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/confspellcheck.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/core-3.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/ezmorph-1.0.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-j2ee-management_1.0_spec-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-jms_1.1_spec-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/guava-r05.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6-test.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/html-filter-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jchronic-0.2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jersey-bundle-1.1.0-ea.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-util-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json-lib-2.2.3-jdk15.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-api-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsr311-api-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/junit-3.8.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/kahadb-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/log4j-1.2.14.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/mysql-connector-java-5.1.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/NagiosAppender-1.5.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/org.osgi.core-4.1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/prefuse.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/servlet-api-2.5-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/sigar.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-beans-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-context-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-core-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-mock-1.2.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-core-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-io-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/zookeeper-3.2.2.jar:
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.library.path=/usr/local/hadoop/lib/native/Linux-i386-32
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.io.tmpdir=/tmp
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.compiler=<NA>
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.name=Linux
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.arch=i386
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.version=2.6.24-19-generic
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.name=hadoop
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.home=/home/hadoop
> 2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.dir=/home/hadoop/workspace/chukwa-0.5.0-incubating
> 2010-12-03 18:20:39,398 INFO main ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@1428ea
> 2010-12-03 18:20:39,400 INFO main ClientCnxn - zookeeper.disableAutoWatchReset is false
> 2010-12-03 18:20:39,418 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:39,422 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a68ef9
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:39,423 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:39,423 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:39,545 INFO main HConnectionManager$TableServers - getMaster attempt 0 of 10 failed; retrying after sleep of 1000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:40,188 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
> 2010-12-03 18:20:40,312 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:20:40,640 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1977b9b
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:40,742 INFO main HConnectionManager$TableServers - getMaster attempt 1 of 10 failed; retrying after sleep of 1000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:42,329 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@2e1f1f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:42,431 INFO main HConnectionManager$TableServers - getMaster attempt 2 of 10 failed; retrying after sleep of 1000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:43,903 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@fefe3f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:44,007 INFO main HConnectionManager$TableServers - getMaster attempt 3 of 10 failed; retrying after sleep of 2000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:45,178 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:45,178 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@c2b2f6
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:45,179 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:45,179 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:46,610 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:46,610 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a001ff
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:46,611 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:46,611 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:46,711 INFO main HConnectionManager$TableServers - getMaster attempt 4 of 10 failed; retrying after sleep of 2000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:48,252 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:48,252 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@90c06f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:48,253 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:48,253 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:49,973 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:49,973 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1267649
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:49,973 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:49,974 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:50,074 INFO main HConnectionManager$TableServers - getMaster attempt 5 of 10 failed; retrying after sleep of 4000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:50,312 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:20:51,749 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@dcb03b
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:53,639 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1b1fbf4
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:55,094 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:55,094 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1829e6f
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:55,094 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:55,095 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:55,195 INFO main HConnectionManager$TableServers - getMaster attempt 6 of 10 failed; retrying after sleep of 4000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:20:56,869 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:56,869 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@100ebec
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:56,869 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:56,870 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:58,099 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7736bd
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:59,363 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@178460d
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:20:59,465 INFO main HConnectionManager$TableServers - getMaster attempt 7 of 10 failed; retrying after sleep of 8000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:21:00,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:21:00,703 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1938039
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:02,221 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@3c342b
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:03,664 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@419d05
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:04,922 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:04,922 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1deeb40
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:04,923 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:04,923 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:06,779 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:06,779 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@cafb56
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:06,779 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:06,780 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:07,942 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:07,943 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a7ddcf
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:07,944 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:07,944 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:08,044 INFO main HConnectionManager$TableServers - getMaster attempt 8 of 10 failed; retrying after sleep of 16000
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:21:09,964 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:09,965 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1da669c
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:09,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:09,966 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:10,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:21:11,285 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@17f242c
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:12,919 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:12,920 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@12f9ee
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:12,920 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:12,921 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:14,754 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@106fc94
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:15,947 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7f5580
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:17,483 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@84cc09
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:18,793 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@14db0e3
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:19,964 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@676437
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:20,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
> 2010-12-03 18:21:21,899 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1e808ca
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:23,916 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@2bd3a
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:25,857 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:25,857 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1329642
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
> 2010-12-03 18:21:25,858 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:25,858 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
> 2010-12-03 18:21:25,958 INFO main HConnectionManager$TableServers - getMaster attempt 9 of 10 failed; no more retrying.
> java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
> ... 18 more
> 2010-12-03 18:21:25,961 ERROR main HBaseWriter - org.apache.hadoop.hbase.MasterNotRunningException
> at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374)
> at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> 2010-12-03 18:21:25,961 ERROR main HBaseWriter - Hbase schema mismatch with demux parser.
> 2010-12-03 18:21:25,961 ERROR main HBaseWriter - Exiting...
> 2010-12-03 18:21:25,963 INFO HCM.shutdownHook ZooKeeper - Closing session: 0x0
> 2010-12-03 18:21:25,964 INFO HCM.shutdownHook ClientCnxn - Closing ClientCnxn for session: 0x0
> 2010-12-03 18:21:27,098 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:21:27,099 INFO main-SendThread ClientCnxn - Exception while closing send thread for session 0x0 : Connection refused
> 2010-12-03 18:21:27,099 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
> 2010-12-03 18:21:27,099 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
> java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
> at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
> 2010-12-03 18:21:27,199 INFO HCM.shutdownHook ClientCnxn - Disconnecting ClientCnxn for session: 0x0
> 2010-12-03 18:21:27,200 INFO HCM.shutdownHook ZooKeeper - Session: 0x0 closed
> 2010-12-03 18:21:27,200 INFO main-EventThread ClientCnxn - EventThread shut down
> Here are the exceptions when HBase is running but tables do not exist
> 010-12-03 18:24:40,846 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
> 2010-12-03 18:24:41,230 INFO main root - initing servletCollector
> 2010-12-03 18:24:41,232 INFO main PipelineStageWriter - using pipelined writers, pipe length is 2
> 2010-12-03 18:24:41,235 INFO Thread-6 SocketTeeWriter - listen thread started
> 2010-12-03 18:24:41,238 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:host.name=ahmed-laptop
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.version=1.6.0_20
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.20/jre
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.class.path=/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf:/usr/local/hadoop/conf::/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../chukwa-core-0.5.0.jar:/usr/local/hadoop/hadoop-0.20.2-core.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-core-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-protobuf-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/aopalliance-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/asm-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-beanutils-1.8.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-codec-1.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-fileupload-1.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-httpclient-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-io-1.4.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-lang-2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-api-1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-net-2.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/confspellcheck.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/core-3.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/ezmorph-1.0.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-j2ee-management_1.0_spec-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-jms_1.1_spec-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/guava-r05.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6-test.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/html-filter-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jchronic-0.2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jersey-bundle-1.1.0-ea.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-util-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json-lib-2.2.3-jdk15.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-api-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsr311-api-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/junit-3.8.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/kahadb-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/log4j-1.2.14.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/mysql-connector-java-5.1.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/NagiosAppender-1.5.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/org.osgi.core-4.1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/prefuse.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/servlet-api-2.5-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/sigar.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-beans-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-context-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-core-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-mock-1.2.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-core-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-io-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/zookeeper-3.2.2.jar:
> 2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.library.path=/usr/local/hadoop/lib/native/Linux-i386-32
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:java.io.tmpdir=/tmp
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:java.compiler=<NA>
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.name=Linux
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.arch=i386
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.version=2.6.24-19-generic
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.name=hadoop
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.home=/home/hadoop
> 2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.dir=/home/hadoop/workspace/chukwa-0.5.0-incubating
> 2010-12-03 18:24:41,420 INFO main ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@1326484
> 2010-12-03 18:24:41,422 INFO main ClientCnxn - zookeeper.disableAutoWatchReset is false
> 2010-12-03 18:24:41,440 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
> 2010-12-03 18:24:41,442 INFO main-SendThread ClientCnxn - Priming connection to java.nio.channels.SocketChannel[connected local=/127.0.0.1:58057 remote=localhost/127.0.0.1:2181]
> 2010-12-03 18:24:41,500 INFO main-SendThread ClientCnxn - Server connection successful
> 2010-12-03 18:24:41,984 ERROR main HBaseWriter - java.lang.Exception: HBase table: Hadoop does not exist.
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseTable(HBaseWriter.java:133)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:158)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> 2010-12-03 18:24:41,984 ERROR main HBaseWriter - java.lang.Exception: Validation failed - table: Hadoop column family: ClientTrace does not exist.
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:159)
> at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
> at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
> at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
> at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
> at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:222)
> at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
> at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
> 2010-12-03 18:24:41,984 ERROR main HBaseWriter - Hbase schema mismatch with demux parser.
> 2010-12-03 18:24:41,984 ERROR main HBaseWriter - Exiting...
> 2010-12-03 18:24:42,087 INFO HCM.shutdownHook ZooKeeper - Closing session: 0x12cad0c07fd0002
> 2010-12-03 18:24:42,087 INFO HCM.shutdownHook ClientCnxn - Closing ClientCnxn for session: 0x12cad0c07fd0002
> 2010-12-03 18:24:42,094 INFO main-SendThread ClientCnxn - Exception while closing send thread for session 0x12cad0c07fd0002 : Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
> 2010-12-03 18:24:42,202 INFO HCM.shutdownHook ClientCnxn - Disconnecting ClientCnxn for session: 0x12cad0c07fd0002
> 2010-12-03 18:24:42,202 INFO HCM.shutdownHook ZooKeeper - Session: 0x12cad0c07fd0002 closed
> 2010-12-03 18:24:42,202 INFO main-EventThread ClientCnxn - EventThread shut down
> 2010-12-03 18:24:42,231 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.