You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@chukwa.apache.org by "Kumar, Suresh" <Su...@emc.com> on 2012/09/17 23:00:29 UTC
agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
I have the following in my local host :
hadoop-1.0.3
chukwa-incubating-0.5.0
hbase-0.92.2
Here are some relevant properties from chukwa-collector-conf.xml
<property>
<name>writer.hdfs.filesystem</name>
<value>hdfs://localhost:9000</value>
<description>HDFS to dump to</description>
</property>
.........
<property>
<name>chukwaCollector.http.port</name>
<value>8081</value>
<description>The HTTP port number the collector will listen
on</description>
</property>
and collectors
http://localhost:8081
I see the following in agent.log and http://localhost:4080/hicc is
spinning, any help appreciated.
2012-09-17 16:00:28,698 INFO main Log4JMetricsContext -
Log4JMetricsContext.chukwaAgent.period=60
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME:
[/opt/chukwa-incubating-0.5.0]
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config -
CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config -
CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir:
[/opt/chukwa/log]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
CHECKPOINT_INTERVAL_MS: [5000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
DO_CHECKPOINT_RESTORE: [true]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
STATS_INTERVAL_MS: [10000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags:
[cluster="demo"]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled,
period is 5000
2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed
in /opt/chukwa/log/chukwa_agent_checkpoint0
2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_93df4746476c9a4b624f6755b122f9dc
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@
30384065]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_79203ebcc495e11b747254e660380620
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@
212bcd4b]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_e2b39549cac6e966126df286a79ae6ad
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@
296f25a7]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_c79bf882974a14286cffe29d3d4cf0d6
function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMe
trics@a13f991]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor,
id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@
7dc05ffd]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed
in /opt/chukwa/etc/chukwa/initial_adaptors
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener -
AgentControlSocketListerner ask for port: 9093
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket
bound to 9093
2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started
on port 9093
2012-09-17 16:00:29,713 INFO main log - Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26
2012-09-17 16:00:29,821 INFO main log - Started
SelectChannelConnector@0.0.0.0:9090
2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http
agent interface on port 9090
2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1
2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on
port 9093
2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector -
HttpConnector started at time:1347912029827
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config -
System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config -
System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up
collectors file: /opt/chukwa/etc/chukwa/collectors
2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using
collectors from collectors file
2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks
ACK'ed since last report: 0
2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender -
collected 1 chunks for post_0
2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>>
HTTP post_0 to http://localhost:8081/ length = 248
2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http
post exception on http://localhost:8081/: java.net.ConnectException:
Connection refused
2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more
collectors to try rolling over to; waiting 20000 ms (144000 retries
left)
2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught
java.io.EOFException closing conneciton.
RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Posted by "Kumar, Suresh" <Su...@emc.com>.
Hi Saisai:
I have the hbase-site.xml in the etc/chukwa folder.
I upgraded to 0.6.0, the exception is gone,
Thanks,
Suresh
From: Shao, Saisai [mailto:saisai.shao@intel.com]
Sent: Wednesday, September 19, 2012 5:10 PM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi Suresh,
Removing disabled file can only fix this exception(WidgetBean - java.lang.NullPointerException).
For the zookeeper problem, I agreed with you that it is originated from HBase, but I have no clues about this exception. Did you config HBase related environment variables in chukwa-env.sh, or did you copy hbase-site.xml to etc/chukwa folder.
My chukwa version 0.6.0, currently trunk version, and my HBase version is 0.94.0. Sorry for not helping you with the zookeeper problems.
Thanks
Saisai
From: Kumar, Suresh [mailto:Suresh.Kumar4@emc.com]
Sent: Thursday, September 20, 2012 4:27 AM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Thanks Saisai, I removed the disabled file, still see the exception.
What version of chukwa are you running? What version of HBase are you running?
Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably
originates from HBase.
2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000
2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
Suresh
From: Shao, Saisai [mailto:saisai.shao@intel.com]
Sent: Tuesday, September 18, 2012 6:06 PM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi Suresh,
I met this exception before.
ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.
I��m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.
Thanks
Saisai
From: Kumar, Suresh [mailto:Suresh.Kumar4@emc.com]
Sent: Wednesday, September 19, 2012 1:45 AM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
I had checked these before posting, the problem was in the ��initial_adaptors�� file, I had copied
��add FileTailer FooData /tmp/foo 0�� from the Chukwa agent documentation, it didn��t work. I had
to change to ��add filetailer.FileTailingAdaptor FooData /tmp/foo 0�� to make it work. I found the
exception in ��agent.log�� and fixed the problem.
However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?
Thanks,
Suresh
2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread ClientCnxn - Error while calling watcher
java.lang.NullPointerException: ZK is null
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper - The identifier of this process is 18719@localhost
2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init
2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
From: ������ [mailto:sai.sai.shao@gmail.com]
Sent: Monday, September 17, 2012 6:28 PM
To: chukwa-user@incubator.apache.org
Subject: Re: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind correct port.
2. maybe port 8081 has been occupied by other process, so please check the collecor.log to see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is correct.
Thanks
Saisai
2012/9/18 Kumar, Suresh <Su...@emc.com>
I have the following in my local host :
?u>
hadoop-1.0.3
chukwa-incubating-0.5.0
hbase-0.92.2
?u>
Here are some relevant properties from ?b>chukwa-collector-conf.xml
?u>
<property>
��?<name>writer.hdfs.filesystem</name>
��?<value>hdfs://localhost:9000</value>
��?<description>HDFS to dump to</description>
?</property>
��?u>
<property>
��?<name>chukwaCollector.http.port</name>
��?<value>8081</value>
��?<description>The HTTP port number the collector will listen on</description>
?</property>
?u>
and collectors
?u>
http://localhost:8081
?u>
I see the following in agent.log and http://localhost:4080/hicc is spinning, any help appreciated.
?u>
2012-09-17 16:00:28,698 INFO main Log4JMetricsContext - Log4JMetricsContext.chukwaAgent.period=60
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME: [/opt/chukwa-incubating-0.5.0]
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir: [/opt/chukwa/log]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - CHECKPOINT_INTERVAL_MS: [5000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE: [true]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - STATS_INTERVAL_MS: [10000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags: [cluster="demo"]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled, period is 5000
2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/log/chukwa_agent_checkpoint0
2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id = adaptor_93df4746476c9a4b624f6755b122f9dc function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_79203ebcc495e11b747254e660380620 function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_e2b39549cac6e966126df286a79ae6ad function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c79bf882974a14286cffe29d3d4cf0d6 function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/etc/chukwa/initial_adaptors
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - AgentControlSocketListerner ask for port: 9093
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket bound to 9093
2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on port 9093
2012-09-17 16:00:29,713 INFO main log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26
2012-09-17 16:00:29,821 INFO main log - Started SelectChannelConnector@0.0.0.0:9090
2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent interface on port 9090
2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1
2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on port 9093
2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector - HttpConnector started at time:1347912029827
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up collectors file: /opt/chukwa/etc/chukwa/collectors
2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using collectors from collectors file
2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed since last report: 0
2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected 1 chunks for post_0
2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>> HTTP post_0 to http://localhost:8081/ length = 248
2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http post exception on http://localhost:8081/: java.net.ConnectException: Connection refused
2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more collectors to try rolling over to; waiting 20000 ms (144000 retries left)
2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught java.io.EOFException closing conneciton.
RE: agent.log I/O exception (java.net.SocketTimeoutException)
caught when processing request: Read timed out
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Suresh,
Removing disabled file can only fix this exception(WidgetBean - java.lang.NullPointerException).
For the zookeeper problem, I agreed with you that it is originated from HBase, but I have no clues about this exception. Did you config HBase related environment variables in chukwa-env.sh, or did you copy hbase-site.xml to etc/chukwa folder.
My chukwa version 0.6.0, currently trunk version, and my HBase version is 0.94.0. Sorry for not helping you with the zookeeper problems.
Thanks
Saisai
From: Kumar, Suresh [mailto:Suresh.Kumar4@emc.com]
Sent: Thursday, September 20, 2012 4:27 AM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Thanks Saisai, I removed the disabled file, still see the exception.
What version of chukwa are you running? What version of HBase are you running?
Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably
originates from HBase.
2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000
2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
Suresh
From: Shao, Saisai [mailto:saisai.shao@intel.com]<mailto:[mailto:saisai.shao@intel.com]>
Sent: Tuesday, September 18, 2012 6:06 PM
To: chukwa-user@incubator.apache.org<ma...@incubator.apache.org>
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi Suresh,
I met this exception before.
ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.
I��m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.
Thanks
Saisai
From: Kumar, Suresh [mailto:Suresh.Kumar4@emc.com]<mailto:[mailto:Suresh.Kumar4@emc.com]>
Sent: Wednesday, September 19, 2012 1:45 AM
To: chukwa-user@incubator.apache.org<ma...@incubator.apache.org>
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
I had checked these before posting, the problem was in the ��initial_adaptors�� file, I had copied
��add FileTailer FooData /tmp/foo 0�� from the Chukwa agent documentation, it didn��t work. I had
to change to ��add filetailer.FileTailingAdaptor FooData /tmp/foo 0�� to make it work. I found the
exception in ��agent.log�� and fixed the problem.
However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?
Thanks,
Suresh
2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread ClientCnxn - Error while calling watcher
java.lang.NullPointerException: ZK is null
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper - The identifier of this process is 18719@localhost
2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init
2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
From: ������ [mailto:sai.sai.shao@gmail.com]
Sent: Monday, September 17, 2012 6:28 PM
To: chukwa-user@incubator.apache.org<ma...@incubator.apache.org>
Subject: Re: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind correct port.
2. maybe port 8081 has been occupied by other process, so please check the collecor.log to see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is correct.
Thanks
Saisai
2012/9/18 Kumar, Suresh <Su...@emc.com>>
I have the following in my local host :
?u>
hadoop-1.0.3
chukwa-incubating-0.5.0
hbase-0.92.2
?u>
Here are some relevant properties from ?b>chukwa-collector-conf.xml
?u>
<property>
��?<name>writer.hdfs.filesystem</name>
��?<value>hdfs://localhost:9000</value>
��?<description>HDFS to dump to</description>
?</property>
��?u>
<property>
��?<name>chukwaCollector.http.port</name>
��?<value>8081</value>
��?<description>The HTTP port number the collector will listen on</description>
?</property>
?u>
and collectors
?u>
http://localhost:8081
?u>
I see the following in agent.log and http://localhost:4080/hicc is spinning, any help appreciated.
?u>
2012-09-17 16:00:28,698 INFO main Log4JMetricsContext - Log4JMetricsContext.chukwaAgent.period=60
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME: [/opt/chukwa-incubating-0.5.0]
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir: [/opt/chukwa/log]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - CHECKPOINT_INTERVAL_MS: [5000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE: [true]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - STATS_INTERVAL_MS: [10000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags: [cluster="demo"]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled, period is 5000
2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/log/chukwa_agent_checkpoint0
2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id = adaptor_93df4746476c9a4b624f6755b122f9dc function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_79203ebcc495e11b747254e660380620 function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_e2b39549cac6e966126df286a79ae6ad function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c79bf882974a14286cffe29d3d4cf0d6 function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/etc/chukwa/initial_adaptors
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - AgentControlSocketListerner ask for port: 9093
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket bound to 9093
2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on port 9093
2012-09-17 16:00:29,713 INFO main log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26
2012-09-17 16:00:29,821 INFO main log - Started SelectChannelConnector@0.0.0.0:9090<http://SelectChannelConnector@0.0.0.0:9090>
2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent interface on port 9090
2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1
2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on port 9093
2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector - HttpConnector started at time:1347912029827
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up collectors file: /opt/chukwa/etc/chukwa/collectors
2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using collectors from collectors file
2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed since last report: 0
2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected 1 chunks for post_0
2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>> HTTP post_0 to http://localhost:8081/ length = 248
2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http post exception on http://localhost:8081/: java.net.ConnectException: Connection refused
2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more collectors to try rolling over to; waiting 20000 ms (144000 retries left)
2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught java.io.EOFException closing conneciton.
RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Posted by "Kumar, Suresh" <Su...@emc.com>.
Thanks Saisai, I removed the disabled file, still see the exception.
What version of chukwa are you running? What version of HBase are you running?
Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably
originates from HBase.
2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005, negotiated timeout = 40000
2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
Suresh
From: Shao, Saisai [mailto:saisai.shao@intel.com]
Sent: Tuesday, September 18, 2012 6:06 PM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi Suresh,
I met this exception before.
ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.
I��m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.
Thanks
Saisai
From: Kumar, Suresh [mailto:Suresh.Kumar4@emc.com]
Sent: Wednesday, September 19, 2012 1:45 AM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
I had checked these before posting, the problem was in the ��initial_adaptors�� file, I had copied
��add FileTailer FooData /tmp/foo 0�� from the Chukwa agent documentation, it didn��t work. I had
to change to ��add filetailer.FileTailingAdaptor FooData /tmp/foo 0�� to make it work. I found the
exception in ��agent.log�� and fixed the problem.
However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?
Thanks,
Suresh
2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread ClientCnxn - Error while calling watcher
java.lang.NullPointerException: ZK is null
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper - The identifier of this process is 18719@localhost
2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init
2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
From: ������ [mailto:sai.sai.shao@gmail.com]
Sent: Monday, September 17, 2012 6:28 PM
To: chukwa-user@incubator.apache.org
Subject: Re: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind correct port.
2. maybe port 8081 has been occupied by other process, so please check the collecor.log to see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is correct.
Thanks
Saisai
2012/9/18 Kumar, Suresh <Su...@emc.com>
I have the following in my local host :
?u>
hadoop-1.0.3
chukwa-incubating-0.5.0
hbase-0.92.2
?u>
Here are some relevant properties from ?b>chukwa-collector-conf.xml
?u>
<property>
��?<name>writer.hdfs.filesystem</name>
��?<value>hdfs://localhost:9000</value>
��?<description>HDFS to dump to</description>
?</property>
��?u>
<property>
��?<name>chukwaCollector.http.port</name>
��?<value>8081</value>
��?<description>The HTTP port number the collector will listen on</description>
?</property>
?u>
and collectors
?u>
http://localhost:8081
?u>
I see the following in agent.log and http://localhost:4080/hicc is spinning, any help appreciated.
?u>
2012-09-17 16:00:28,698 INFO main Log4JMetricsContext - Log4JMetricsContext.chukwaAgent.period=60
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME: [/opt/chukwa-incubating-0.5.0]
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir: [/opt/chukwa/log]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - CHECKPOINT_INTERVAL_MS: [5000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE: [true]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - STATS_INTERVAL_MS: [10000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags: [cluster="demo"]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled, period is 5000
2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/log/chukwa_agent_checkpoint0
2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id = adaptor_93df4746476c9a4b624f6755b122f9dc function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_79203ebcc495e11b747254e660380620 function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_e2b39549cac6e966126df286a79ae6ad function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c79bf882974a14286cffe29d3d4cf0d6 function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/etc/chukwa/initial_adaptors
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - AgentControlSocketListerner ask for port: 9093
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket bound to 9093
2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on port 9093
2012-09-17 16:00:29,713 INFO main log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26
2012-09-17 16:00:29,821 INFO main log - Started SelectChannelConnector@0.0.0.0:9090
2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent interface on port 9090
2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1
2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on port 9093
2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector - HttpConnector started at time:1347912029827
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up collectors file: /opt/chukwa/etc/chukwa/collectors
2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using collectors from collectors file
2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed since last report: 0
2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected 1 chunks for post_0
2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>> HTTP post_0 to http://localhost:8081/ length = 248
2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http post exception on http://localhost:8081/: java.net.ConnectException: Connection refused
2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more collectors to try rolling over to; waiting 20000 ms (144000 retries left)
2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught java.io.EOFException closing conneciton.
RE: agent.log I/O exception (java.net.SocketTimeoutException)
caught when processing request: Read timed out
Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Suresh,
I met this exception before.
ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.
I��m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several minutes to load.
Thanks
Saisai
From: Kumar, Suresh [mailto:Suresh.Kumar4@emc.com]
Sent: Wednesday, September 19, 2012 1:45 AM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
I had checked these before posting, the problem was in the ��initial_adaptors�� file, I had copied
��add FileTailer FooData /tmp/foo 0�� from the Chukwa agent documentation, it didn��t work. I had
to change to ��add filetailer.FileTailingAdaptor FooData /tmp/foo 0�� to make it work. I found the
exception in ��agent.log�� and fixed the problem.
However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?
Thanks,
Suresh
2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread ClientCnxn - Error while calling watcher
java.lang.NullPointerException: ZK is null
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper - The identifier of this process is 18719@localhost
2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init
2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
From: ������ [mailto:sai.sai.shao@gmail.com]
Sent: Monday, September 17, 2012 6:28 PM
To: chukwa-user@incubator.apache.org<ma...@incubator.apache.org>
Subject: Re: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind correct port.
2. maybe port 8081 has been occupied by other process, so please check the collecor.log to see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is correct.
Thanks
Saisai
2012/9/18 Kumar, Suresh <Su...@emc.com>>
I have the following in my local host :
?u>
hadoop-1.0.3
chukwa-incubating-0.5.0
hbase-0.92.2
?u>
Here are some relevant properties from ?b>chukwa-collector-conf.xml
?u>
<property>
��?<name>writer.hdfs.filesystem</name>
��?<value>hdfs://localhost:9000</value>
��?<description>HDFS to dump to</description>
?</property>
��?u>
<property>
��?<name>chukwaCollector.http.port</name>
��?<value>8081</value>
��?<description>The HTTP port number the collector will listen on</description>
?</property>
?u>
and collectors
?u>
http://localhost:8081
?u>
I see the following in agent.log and http://localhost:4080/hicc is spinning, any help appreciated.
?u>
2012-09-17 16:00:28,698 INFO main Log4JMetricsContext - Log4JMetricsContext.chukwaAgent.period=60
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME: [/opt/chukwa-incubating-0.5.0]
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir: [/opt/chukwa/log]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - CHECKPOINT_INTERVAL_MS: [5000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE: [true]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - STATS_INTERVAL_MS: [10000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags: [cluster="demo"]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled, period is 5000
2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/log/chukwa_agent_checkpoint0
2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id = adaptor_93df4746476c9a4b624f6755b122f9dc function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_79203ebcc495e11b747254e660380620 function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_e2b39549cac6e966126df286a79ae6ad function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c79bf882974a14286cffe29d3d4cf0d6 function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/etc/chukwa/initial_adaptors
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - AgentControlSocketListerner ask for port: 9093
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket bound to 9093
2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on port 9093
2012-09-17 16:00:29,713 INFO main log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26
2012-09-17 16:00:29,821 INFO main log - Started SelectChannelConnector@0.0.0.0:9090<http://SelectChannelConnector@0.0.0.0:9090>
2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent interface on port 9090
2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1
2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on port 9093
2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector - HttpConnector started at time:1347912029827
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up collectors file: /opt/chukwa/etc/chukwa/collectors
2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using collectors from collectors file
2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed since last report: 0
2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected 1 chunks for post_0
2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>> HTTP post_0 to http://localhost:8081/ length = 248
2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http post exception on http://localhost:8081/: java.net.ConnectException: Connection refused
2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more collectors to try rolling over to; waiting 20000 ms (144000 retries left)
2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught java.io.EOFException closing conneciton.
RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Posted by "Kumar, Suresh" <Su...@emc.com>.
I had checked these before posting, the problem was in the ��initial_adaptors�� file, I had copied
��add FileTailer FooData /tmp/foo 0�� from the Chukwa agent documentation, it didn��t work. I had
to change to ��add filetailer.FileTailingAdaptor FooData /tmp/foo 0�� to make it work. I found the
exception in ��agent.log�� and fixed the problem.
However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any clues?
Thanks,
Suresh
2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK is null on connection event -- see stack trace for the stack trace when constructor was called on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread ClientCnxn - Error while calling watcher
java.lang.NullPointerException: ZK is null
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper - The identifier of this process is 18719@localhost
2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init
2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
From: ������ [mailto:sai.sai.shao@gmail.com]
Sent: Monday, September 17, 2012 6:28 PM
To: chukwa-user@incubator.apache.org
Subject: Re: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind correct port.
2. maybe port 8081 has been occupied by other process, so please check the collecor.log to see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is correct.
Thanks
Saisai
2012/9/18 Kumar, Suresh <Su...@emc.com>
I have the following in my local host :
?u>
hadoop-1.0.3
chukwa-incubating-0.5.0
hbase-0.92.2
?u>
Here are some relevant properties from ?b>chukwa-collector-conf.xml
?u>
<property>
��?<name>writer.hdfs.filesystem</name>
��?<value>hdfs://localhost:9000</value>
��?<description>HDFS to dump to</description>
?</property>
��?u>
<property>
��?<name>chukwaCollector.http.port</name>
��?<value>8081</value>
��?<description>The HTTP port number the collector will listen on</description>
?</property>
?u>
and collectors
?u>
http://localhost:8081
?u>
I see the following in agent.log and http://localhost:4080/hicc is spinning, any help appreciated.
?u>
2012-09-17 16:00:28,698 INFO main Log4JMetricsContext - Log4JMetricsContext.chukwaAgent.period=60
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME: [/opt/chukwa-incubating-0.5.0]
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir: [/opt/chukwa/log]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - CHECKPOINT_INTERVAL_MS: [5000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE: [true]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - STATS_INTERVAL_MS: [10000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags: [cluster="demo"]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled, period is 5000
2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/log/chukwa_agent_checkpoint0
2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id = adaptor_93df4746476c9a4b624f6755b122f9dc function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_79203ebcc495e11b747254e660380620 function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_e2b39549cac6e966126df286a79ae6ad function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c79bf882974a14286cffe29d3d4cf0d6 function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/etc/chukwa/initial_adaptors
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - AgentControlSocketListerner ask for port: 9093
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket bound to 9093
2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on port 9093
2012-09-17 16:00:29,713 INFO main log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26
2012-09-17 16:00:29,821 INFO main log - Started SelectChannelConnector@0.0.0.0:9090
2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent interface on port 9090
2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1
2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on port 9093
2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector - HttpConnector started at time:1347912029827
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up collectors file: /opt/chukwa/etc/chukwa/collectors
2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using collectors from collectors file
2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed since last report: 0
2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected 1 chunks for post_0
2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>> HTTP post_0 to http://localhost:8081/ length = 248
2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http post exception on http://localhost:8081/: java.net.ConnectException: Connection refused
2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more collectors to try rolling over to; waiting 20000 ms (144000 retries left)
2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught java.io.EOFException closing conneciton.
Re: agent.log I/O exception (java.net.SocketTimeoutException) caught
when processing request: Read timed out
Posted by 邵赛赛 <sa...@gmail.com>.
Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind
correct port.
2. maybe port 8081 has been occupied by other process, so please check the
collecor.log to see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is
correct.
Thanks
Saisai
2012/9/18 Kumar, Suresh <Su...@emc.com>
> I have the following in my local host :****
>
> ** **
>
> hadoop-1.0.3****
>
> chukwa-incubating-0.5.0****
>
> hbase-0.92.2****
>
> ** **
>
> Here are some relevant properties from *chukwa-collector-conf.xml*****
>
> ** **
>
> <property>****
>
> <name>writer.hdfs.filesystem</name>****
>
> <value>hdfs://localhost:9000</value>****
>
> <description>HDFS to dump to</description>****
>
> </property>****
>
> ………****
>
> <property>****
>
> <name>chukwaCollector.http.port</name>****
>
> <value>8081</value>****
>
> <description>The HTTP port number the collector will listen
> on</description>****
>
> </property>****
>
> ** **
>
> and *collectors*
>
> * *
>
> http://localhost:8081****
>
> ** **
>
> I see the following in agent.log and http://localhost:4080/hicc is
> spinning, any help appreciated.****
>
> ** **
>
> 2012-09-17 16:00:28,698 INFO main Log4JMetricsContext -
> Log4JMetricsContext.chukwaAgent.period=60****
>
> 2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME:
> [/opt/chukwa-incubating-0.5.0]****
>
> 2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR:
> [/opt/chukwa/etc/chukwa]****
>
> 2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config -
> CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir:
> [/opt/chukwa/log]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
> CHECKPOINT_INTERVAL_MS: [5000]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
> DO_CHECKPOINT_RESTORE: [true]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config -
> STATS_INTERVAL_MS: [10000]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags:
> [cluster="demo"]****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled,
> period is 5000****
>
> 2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed
> in /opt/chukwa/log/chukwa_agent_checkpoint0****
>
> 2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_93df4746476c9a4b624f6755b122f9dc
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065
> ]****
>
> 2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_79203ebcc495e11b747254e660380620
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b
> ]****
>
> 2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_e2b39549cac6e966126df286a79ae6ad
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7
> ]****
>
> 2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_c79bf882974a14286cffe29d3d4cf0d6
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991
> ]****
>
> 2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id
> = adaptor_c6fd1e5021ad8023477ef0b56a24a29f
> function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd
> ]****
>
> 2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed
> in /opt/chukwa/etc/chukwa/initial_adaptors****
>
> 2012-09-17 16:00:29,675 INFO main AgentControlSocketListener -
> AgentControlSocketListerner ask for port: 9093****
>
> 2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket
> bound to 9093****
>
> 2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on
> port 9093****
>
> 2012-09-17 16:00:29,713 INFO main log - Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog****
>
> 2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26****
>
> 2012-09-17 16:00:29,821 INFO main log - Started
> SelectChannelConnector@0.0.0.0:9090****
>
> 2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent
> interface on port 9090****
>
> 2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1***
> *
>
> 2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on
> port 9093****
>
> 2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector -
> HttpConnector started at time:1347912029827****
>
> 2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config -
> System.getenv("CHUKWA_HOME"): [/opt/chukwa-incubating-0.5.0/]****
>
> 2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config -
> System.getenv("chukwaConf"): [/opt/chukwa/etc/chukwa]****
>
> 2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up
> collectors file: /opt/chukwa/etc/chukwa/collectors****
>
> 2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using
> collectors from collectors file****
>
> 2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed
> since last report: 0****
>
> 2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected
> 1 chunks for post_0****
>
> 2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>>
> HTTP post_0 to http://localhost:8081/ length = 248****
>
> 2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http
> post exception on http://localhost:8081/: java.net.ConnectException:
> Connection refused****
>
> 2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more
> collectors to try rolling over to; waiting 20000 ms (144000 retries left)*
> ***
>
> 2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught
> java.io.EOFException closing conneciton.****
>