You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Hadi Sotudeh <ha...@gmail.com> on 2015/05/15 20:41:42 UTC
error in the cluster
Hi
I've submitted my project to the cluster after a while I've got the
following output:
Any idea?
2015-05-15 22:47:33 o.a.c.f.s.ConnectionStateManager [INFO] State change:
SUSPENDED
2015-05-15 22:47:34 o.a.z.ClientCnxn [INFO] Opening socket connection to
server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-05-15 22:47:34 o.a.z.ClientCnxn [INFO] Socket connection established
to user1-HVM-domU.local/213.233.170.200:2181, initiating session
2015-05-15 22:47:38 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2015-05-15 22:47:49 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2015-05-15 22:47:51 o.a.z.ClientCnxn [WARN] Session 0x14d572f6d9e08f7 for
server user1-HVM-domU.local/213.233.170.200:2181, unexpected error, closing
socket connection and attempting reconnect
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method) ~[na:1.6.0_35]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
~[na:1.6.0_35]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122) ~[na:1.6.0_35]
at sun.nio.ch.IOUtil.write(IOUtil.java:93) ~[na:1.6.0_35]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
~[na:1.6.0_35]
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
~[zookeeper-3.4.5.jar:3.4.5-1392090]
2015-05-15 22:47:55 o.a.z.ClientCnxn [INFO] Opening socket connection to
server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-05-15 22:47:55 o.a.z.ClientCnxn [INFO] Socket connection established
to user1-HVM-domU.local/213.233.170.200:2181, initiating session
2015-05-15 22:47:56 o.a.z.ClientCnxn [INFO] Unable to reconnect to
ZooKeeper service, session 0x14d572f6d9e08f7 has expired, closing socket
connection
2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [INFO] State change:
LOST
2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2015-05-15 22:47:59 b.s.cluster [WARN] Received event :expired::none: with
disconnected Zookeeper.
2015-05-15 22:47:59 o.a.c.ConnectionState [WARN] Session expired event
received
2015-05-15 22:47:59 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=213.233.170.200:2181/storm sessionTimeout=20000
watcher=org.apache.curator.ConnectionState@4be4c643
2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Opening socket connection to
server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Socket connection established
to user1-HVM-domU.local/213.233.170.200:2181, initiating session
2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Session establishment complete
on server user1-HVM-domU.local/213.233.170.200:2181, sessionid =
0x14d572f6d9e08fa, negotiated timeout = 20000
2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [INFO] State change:
RECONNECTED
2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2015-05-15 22:48:56 o.a.z.ClientCnxn [INFO] Client session timed out, have
not heard from server in 47980ms for sessionid 0x14d572f6d9e08fa, closing
socket connection and attempting reconnect
2015-05-15 22:48:59 o.a.c.f.s.ConnectionStateManager [INFO] State change:
SUSPENDED
2015-05-15 22:48:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2015-05-15 22:48:59 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Opening socket connection to
server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Socket connection established
to user1-HVM-domU.local/213.233.170.200:2181, initiating session
2015-05-15 22:49:01 o.a.c.f.s.ConnectionStateManager [INFO] State change:
LOST
2015-05-15 22:49:01 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2015-05-15 22:49:01 b.s.cluster [WARN] Received event :expired::none: with
disconnected Zookeeper.
2015-05-15 22:49:01 o.a.c.ConnectionState [WARN] Session expired event
received
2015-05-15 22:49:01 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=213.233.170.200:2181/storm sessionTimeout=20000
watcher=org.apache.curator.ConnectionState@4be4c643
2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Unable to reconnect to
ZooKeeper service, session 0x14d572f6d9e08fa has expired, closing socket
connection
2015-05-15 22:51:21 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Opening socket connection to
server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Socket connection established
to user1-HVM-domU.local/213.233.170.200:2181, initiating session
2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Session establishment complete
on server user1-HVM-domU.local/213.233.170.200:2181, sessionid =
0x14d572f6d9e08fc, negotiated timeout = 20000
2015-05-15 22:52:01 o.a.c.f.s.ConnectionStateManager [INFO] State change:
RECONNECTED
2015-05-15 22:52:01 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2015-05-15 22:52:03 o.a.c.ConnectionState [WARN] Connection attempt
unsuccessful after 140089 (greater than max timeout of 20000). Resetting
connection and trying again with a new connection.
2015-05-15 22:52:04 o.a.z.ZooKeeper [INFO] Session: 0x14d572f6d9e08fc closed
2015-05-15 22:52:04 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=213.233.170.200:2181/storm sessionTimeout=20000
watcher=org.apache.curator.ConnectionState@4be4c643
2015-05-15 22:52:04 o.a.z.ClientCnxn [INFO] EventThread shut down
2015-05-15 22:52:07 o.a.z.ClientCnxn [INFO] Opening socket connection to
server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-05-15 22:52:08 o.a.z.ClientCnxn [INFO] Socket connection established
to user1-HVM-domU.local/213.233.170.200:2181, initiating session
2015-05-15 22:52:08 o.a.z.ClientCnxn [INFO] Session establishment complete
on server user1-HVM-domU.local/213.233.170.200:2181, sessionid =
0x14d572f6d9e08fd, negotiated timeout = 20000
2015-05-15 22:52:38 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.io.IOException: Broken pipe
at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:119)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:68)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5573$fn__5588$fn__5617.invoke(executor.clj:563)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
Caused by: java.io.IOException: Broken pipe
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.6.0_35]
at java.io.FileOutputStream.write(FileOutputStream.java:300) ~[na:1.6.0_35]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[na:1.6.0_35]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[na:1.6.0_35]
at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.6.0_35]
at
com.yelp.pyleus.serializer.MessagePackSerializer.writeMessage(MessagePackSerializer.java:203)
~[stormjar.jar:na]
at
com.yelp.pyleus.serializer.MessagePackSerializer.writeTaskIds(MessagePackSerializer.java:194)
~[stormjar.jar:na]
at backtype.storm.utils.ShellProcess.writeTaskIds(ShellProcess.java:116)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:109)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 5 common frames omitted
2015-05-15 22:52:38 o.a.z.ClientCnxn [INFO] Client session timed out, have
not heard from server in 24784ms for sessionid 0x14d572f6d9e08fd, closing
socket connection and attempting reconnect
2015-05-15 22:52:48 b.s.d.executor [ERROR]
java.lang.RuntimeException: java.io.IOException: Broken pipe
at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:119)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:68)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5573$fn__5588$fn__5617.invoke(executor.clj:563)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
Caused by: java.io.IOException: Broken pipe
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.6.0_35]
at java.io.FileOutputStream.write(FileOutputStream.java:300) ~[na:1.6.0_35]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[na:1.6.0_35]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[na:1.6.0_35]
at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.6.0_35]
at
com.yelp.pyleus.serializer.MessagePackSerializer.writeMessage(MessagePackSerializer.java:203)
~[stormjar.jar:na]
at
com.yelp.pyleus.serializer.MessagePackSerializer.writeTaskIds(MessagePackSerializer.java:194)
~[stormjar.jar:na]
at backtype.storm.utils.ShellProcess.writeTaskIds(ShellProcess.java:116)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:109)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 5 common frames omitted
2015-05-15 22:52:57 o.a.c.f.s.ConnectionStateManager [INFO] State change:
SUSPENDED
2015-05-15 22:52:57 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2015-05-15 22:52:57 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: java.io.EOFException
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
Caused by: java.lang.RuntimeException: java.io.EOFException
at backtype.storm.task.ShellBolt.execute(ShellBolt.java:157)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted
Caused by: java.io.EOFException: null
at org.msgpack.io.StreamInput.readByte(StreamInput.java:60)
~[stormjar.jar:na]
at
org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66)
~[stormjar.jar:na]
at
org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396)
~[stormjar.jar:na]
at org.msgpack.template.MapTemplate.read(MapTemplate.java:59)
~[stormjar.jar:na]
at org.msgpack.template.MapTemplate.read(MapTemplate.java:27)
~[stormjar.jar:na]
at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31)
~[stormjar.jar:na]
at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na]
at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na]
at
com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198)
~[stormjar.jar:na]
at
com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74)
~[stormjar.jar:na]
at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:97)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.task.ShellBolt$1.run(ShellBolt.java:107)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 1 common frames omitted
2015-05-15 22:52:57 b.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.RuntimeException: java.io.EOFException
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
Caused by: java.lang.RuntimeException: java.io.EOFException
at backtype.storm.task.ShellBolt.execute(ShellBolt.java:157)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted
Caused by: java.io.EOFException: null
at org.msgpack.io.StreamInput.readByte(StreamInput.java:60)
~[stormjar.jar:na]
at
org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66)
~[stormjar.jar:na]
at
org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396)
~[stormjar.jar:na]
at org.msgpack.template.MapTemplate.read(MapTemplate.java:59)
~[stormjar.jar:na]
at org.msgpack.template.MapTemplate.read(MapTemplate.java:27)
~[stormjar.jar:na]
at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31)
~[stormjar.jar:na]
at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na]
at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na]
at
com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198)
~[stormjar.jar:na]
at
com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74)
~[stormjar.jar:na]
at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:97)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.task.ShellBolt$1.run(ShellBolt.java:107)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 1 common frames omitted
2015-05-15 22:52:57 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2015-05-15 22:53:04 o.a.z.ClientCnxn [INFO] Opening socket connection to
server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
authenticate using SASL (unknown error)
2015-05-15 22:53:04 o.a.z.ClientCnxn [INFO] Socket connection established
to user1-HVM-domU.local/213.233.170.200:2181, initiating session
2015-05-15 22:53:04 o.a.c.f.s.ConnectionStateManager [INFO] State change:
LOST
2015-05-15 22:53:04 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2015-05-15 22:53:11 o.a.z.ClientCnxn [INFO] Unable to reconnect to
ZooKeeper service, session 0x14d572f6d9e08fd has expired, closing socket
connection
2015-05-15 22:53:27 o.a.c.ConnectionState [ERROR] Connection timed out for
connection string (213.233.170.200:2181/storm) and timeout (15000) /
elapsed (15086)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
ConnectionLoss
at
org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
[curator-client-2.4.0.jar:na]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
[curator-client-2.4.0.jar:na]
at
org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113)
~[curator-client-2.4.0.jar:na]
at
org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:457)
~[curator-framework-2.4.0.jar:na]
at
org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172)
~[curator-framework-2.4.0.jar:na]
at
org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161)
~[curator-framework-2.4.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
~[curator-client-2.4.0.jar:na]
at
org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:157)
~[curator-framework-2.4.0.jar:na]
at
org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148)
~[curator-framework-2.4.0.jar:na]
at
org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36)
~[curator-framework-2.4.0.jar:na]
at
backtype.storm.zookeeper$exists_node_QMARK_$fn__1153.invoke(zookeeper.clj:101)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:98)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:114)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.cluster$mk_distributed_cluster_state$reify__1865.mkdirs(cluster.clj:109)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.cluster$mk_storm_cluster_state$reify__2284.report_error(cluster.clj:368)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$throttled_report_error_fn$fn__5421.invoke(executor.clj:178)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_executor_data$fn__5474$fn__5475.invoke(executor.clj:237)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:441)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
Thanks
Hadi
Re: error in the cluster
Posted by Jeffery Maass <ma...@gmail.com>.
Like I said, it's only a feeling I have.
Is the disk full?
df -h
How many open connections are there / grouped by state:
netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n
run top - see what the processor utilization is
are you monitoring the storm worker box? What does the monitoring say
about them?
are there other applications on the storm worker box? Are they having
problems?
Look though various system logs :
http://www.thegeekstuff.com/2011/08/linux-var-log-files/
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Fri, May 15, 2015 at 2:22 PM, Hadi Sotudeh <ha...@gmail.com>
wrote:
> can you explain more?
> What do you mean by look at the health of the underlying OS?
>
>
Re: error in the cluster
Posted by Hadi Sotudeh <ha...@gmail.com>.
can you explain more?
What do you mean by look at the health of the underlying OS?
Re: error in the cluster
Posted by Jeffery Maass <ma...@gmail.com>.
Boy, everything is fubar isn't it?
It note that the involved libraries are:
Apache Curator - A ZooKeeper keeper
backtype.storm.utils.DisruptorQueue - messaging - having nothing to do with
ZooKeeper
backtype.storm.spout.ShellSpout - looks like it wasn't working with
ZooKeeper either
org.apache.zookeeper.ClientCnxnSocketNIO - zookeeper
Something tells me to look at the health of the underlying OS, especially
the disk. I'm not sure why I feel this, I just do.
Sorry I couldn't be of more help.
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Fri, May 15, 2015 at 1:41 PM, Hadi Sotudeh <ha...@gmail.com>
wrote:
> Hi
> I've submitted my project to the cluster after a while I've got the
> following output:
> Any idea?
>
> 2015-05-15 22:47:33 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> SUSPENDED
> 2015-05-15 22:47:34 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-05-15 22:47:34 o.a.z.ClientCnxn [INFO] Socket connection established
> to user1-HVM-domU.local/213.233.170.200:2181, initiating session
> 2015-05-15 22:47:38 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
> 2015-05-15 22:47:49 b.s.cluster [WARN] Received event :disconnected::none:
> with disconnected Zookeeper.
> 2015-05-15 22:47:51 o.a.z.ClientCnxn [WARN] Session 0x14d572f6d9e08f7 for
> server user1-HVM-domU.local/213.233.170.200:2181, unexpected error,
> closing socket connection and attempting reconnect
> java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcher.write0(Native Method) ~[na:1.6.0_35]
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> ~[na:1.6.0_35]
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122) ~[na:1.6.0_35]
> at sun.nio.ch.IOUtil.write(IOUtil.java:93) ~[na:1.6.0_35]
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
> ~[na:1.6.0_35]
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
> ~[zookeeper-3.4.5.jar:3.4.5-1392090]
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
> ~[zookeeper-3.4.5.jar:3.4.5-1392090]
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> ~[zookeeper-3.4.5.jar:3.4.5-1392090]
> 2015-05-15 22:47:55 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-05-15 22:47:55 o.a.z.ClientCnxn [INFO] Socket connection established
> to user1-HVM-domU.local/213.233.170.200:2181, initiating session
> 2015-05-15 22:47:56 o.a.z.ClientCnxn [INFO] Unable to reconnect to
> ZooKeeper service, session 0x14d572f6d9e08f7 has expired, closing socket
> connection
> 2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> LOST
> 2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
> 2015-05-15 22:47:59 b.s.cluster [WARN] Received event :expired::none: with
> disconnected Zookeeper.
> 2015-05-15 22:47:59 o.a.c.ConnectionState [WARN] Session expired event
> received
> 2015-05-15 22:47:59 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=213.233.170.200:2181/storm sessionTimeout=20000
> watcher=org.apache.curator.ConnectionState@4be4c643
> 2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] EventThread shut down
> 2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Socket connection established
> to user1-HVM-domU.local/213.233.170.200:2181, initiating session
> 2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server user1-HVM-domU.local/213.233.170.200:2181, sessionid =
> 0x14d572f6d9e08fa, negotiated timeout = 20000
> 2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> RECONNECTED
> 2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
> 2015-05-15 22:48:56 o.a.z.ClientCnxn [INFO] Client session timed out, have
> not heard from server in 47980ms for sessionid 0x14d572f6d9e08fa, closing
> socket connection and attempting reconnect
> 2015-05-15 22:48:59 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> SUSPENDED
> 2015-05-15 22:48:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
> 2015-05-15 22:48:59 b.s.cluster [WARN] Received event :disconnected::none:
> with disconnected Zookeeper.
> 2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Socket connection established
> to user1-HVM-domU.local/213.233.170.200:2181, initiating session
> 2015-05-15 22:49:01 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> LOST
> 2015-05-15 22:49:01 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
> 2015-05-15 22:49:01 b.s.cluster [WARN] Received event :expired::none: with
> disconnected Zookeeper.
> 2015-05-15 22:49:01 o.a.c.ConnectionState [WARN] Session expired event
> received
> 2015-05-15 22:49:01 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=213.233.170.200:2181/storm sessionTimeout=20000
> watcher=org.apache.curator.ConnectionState@4be4c643
> 2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Unable to reconnect to
> ZooKeeper service, session 0x14d572f6d9e08fa has expired, closing socket
> connection
> 2015-05-15 22:51:21 o.a.z.ClientCnxn [INFO] EventThread shut down
> 2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Socket connection established
> to user1-HVM-domU.local/213.233.170.200:2181, initiating session
> 2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server user1-HVM-domU.local/213.233.170.200:2181, sessionid =
> 0x14d572f6d9e08fc, negotiated timeout = 20000
> 2015-05-15 22:52:01 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> RECONNECTED
> 2015-05-15 22:52:01 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
> 2015-05-15 22:52:03 o.a.c.ConnectionState [WARN] Connection attempt
> unsuccessful after 140089 (greater than max timeout of 20000). Resetting
> connection and trying again with a new connection.
> 2015-05-15 22:52:04 o.a.z.ZooKeeper [INFO] Session: 0x14d572f6d9e08fc
> closed
> 2015-05-15 22:52:04 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=213.233.170.200:2181/storm sessionTimeout=20000
> watcher=org.apache.curator.ConnectionState@4be4c643
> 2015-05-15 22:52:04 o.a.z.ClientCnxn [INFO] EventThread shut down
> 2015-05-15 22:52:07 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-05-15 22:52:08 o.a.z.ClientCnxn [INFO] Socket connection established
> to user1-HVM-domU.local/213.233.170.200:2181, initiating session
> 2015-05-15 22:52:08 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server user1-HVM-domU.local/213.233.170.200:2181, sessionid =
> 0x14d572f6d9e08fd, negotiated timeout = 20000
> 2015-05-15 22:52:38 b.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: java.io.IOException: Broken pipe
> at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:119)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:68)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5573$fn__5588$fn__5617.invoke(executor.clj:563)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
> Caused by: java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.6.0_35]
> at java.io.FileOutputStream.write(FileOutputStream.java:300) ~[na:1.6.0_35]
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.6.0_35]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.6.0_35]
> at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.6.0_35]
> at
> com.yelp.pyleus.serializer.MessagePackSerializer.writeMessage(MessagePackSerializer.java:203)
> ~[stormjar.jar:na]
> at
> com.yelp.pyleus.serializer.MessagePackSerializer.writeTaskIds(MessagePackSerializer.java:194)
> ~[stormjar.jar:na]
> at backtype.storm.utils.ShellProcess.writeTaskIds(ShellProcess.java:116)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:109)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 5 common frames omitted
> 2015-05-15 22:52:38 o.a.z.ClientCnxn [INFO] Client session timed out, have
> not heard from server in 24784ms for sessionid 0x14d572f6d9e08fd, closing
> socket connection and attempting reconnect
> 2015-05-15 22:52:48 b.s.d.executor [ERROR]
> java.lang.RuntimeException: java.io.IOException: Broken pipe
> at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:119)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:68)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5573$fn__5588$fn__5617.invoke(executor.clj:563)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
> Caused by: java.io.IOException: Broken pipe
> at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.6.0_35]
> at java.io.FileOutputStream.write(FileOutputStream.java:300) ~[na:1.6.0_35]
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> ~[na:1.6.0_35]
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> ~[na:1.6.0_35]
> at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.6.0_35]
> at
> com.yelp.pyleus.serializer.MessagePackSerializer.writeMessage(MessagePackSerializer.java:203)
> ~[stormjar.jar:na]
> at
> com.yelp.pyleus.serializer.MessagePackSerializer.writeTaskIds(MessagePackSerializer.java:194)
> ~[stormjar.jar:na]
> at backtype.storm.utils.ShellProcess.writeTaskIds(ShellProcess.java:116)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:109)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 5 common frames omitted
> 2015-05-15 22:52:57 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> SUSPENDED
> 2015-05-15 22:52:57 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
> 2015-05-15 22:52:57 b.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.EOFException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
> Caused by: java.lang.RuntimeException: java.io.EOFException
> at backtype.storm.task.ShellBolt.execute(ShellBolt.java:157)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted
> Caused by: java.io.EOFException: null
> at org.msgpack.io.StreamInput.readByte(StreamInput.java:60)
> ~[stormjar.jar:na]
> at
> org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66)
> ~[stormjar.jar:na]
> at
> org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396)
> ~[stormjar.jar:na]
> at org.msgpack.template.MapTemplate.read(MapTemplate.java:59)
> ~[stormjar.jar:na]
> at org.msgpack.template.MapTemplate.read(MapTemplate.java:27)
> ~[stormjar.jar:na]
> at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31)
> ~[stormjar.jar:na]
> at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na]
> at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na]
> at
> com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198)
> ~[stormjar.jar:na]
> at
> com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74)
> ~[stormjar.jar:na]
> at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:97)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.task.ShellBolt$1.run(ShellBolt.java:107)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 1 common frames omitted
> 2015-05-15 22:52:57 b.s.d.executor [ERROR]
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.io.EOFException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
> Caused by: java.lang.RuntimeException: java.io.EOFException
> at backtype.storm.task.ShellBolt.execute(ShellBolt.java:157)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted
> Caused by: java.io.EOFException: null
> at org.msgpack.io.StreamInput.readByte(StreamInput.java:60)
> ~[stormjar.jar:na]
> at
> org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66)
> ~[stormjar.jar:na]
> at
> org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396)
> ~[stormjar.jar:na]
> at org.msgpack.template.MapTemplate.read(MapTemplate.java:59)
> ~[stormjar.jar:na]
> at org.msgpack.template.MapTemplate.read(MapTemplate.java:27)
> ~[stormjar.jar:na]
> at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31)
> ~[stormjar.jar:na]
> at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na]
> at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na]
> at
> com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198)
> ~[stormjar.jar:na]
> at
> com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74)
> ~[stormjar.jar:na]
> at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:97)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.task.ShellBolt$1.run(ShellBolt.java:107)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 1 common frames omitted
> 2015-05-15 22:52:57 b.s.cluster [WARN] Received event :disconnected::none:
> with disconnected Zookeeper.
> 2015-05-15 22:53:04 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to
> authenticate using SASL (unknown error)
> 2015-05-15 22:53:04 o.a.z.ClientCnxn [INFO] Socket connection established
> to user1-HVM-domU.local/213.233.170.200:2181, initiating session
> 2015-05-15 22:53:04 o.a.c.f.s.ConnectionStateManager [INFO] State change:
> LOST
> 2015-05-15 22:53:04 o.a.c.f.s.ConnectionStateManager [WARN] There are no
> ConnectionStateListeners registered.
> 2015-05-15 22:53:11 o.a.z.ClientCnxn [INFO] Unable to reconnect to
> ZooKeeper service, session 0x14d572f6d9e08fd has expired, closing socket
> connection
> 2015-05-15 22:53:27 o.a.c.ConnectionState [ERROR] Connection timed out for
> connection string (213.233.170.200:2181/storm) and timeout (15000) /
> elapsed (15086)
> org.apache.curator.CuratorConnectionLossException: KeeperErrorCode =
> ConnectionLoss
> at
> org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
> [curator-client-2.4.0.jar:na]
> at
> org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
> [curator-client-2.4.0.jar:na]
> at
> org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113)
> ~[curator-client-2.4.0.jar:na]
> at
> org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:457)
> ~[curator-framework-2.4.0.jar:na]
> at
> org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172)
> ~[curator-framework-2.4.0.jar:na]
> at
> org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161)
> ~[curator-framework-2.4.0.jar:na]
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
> ~[curator-client-2.4.0.jar:na]
> at
> org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:157)
> ~[curator-framework-2.4.0.jar:na]
> at
> org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148)
> ~[curator-framework-2.4.0.jar:na]
> at
> org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36)
> ~[curator-framework-2.4.0.jar:na]
> at
> backtype.storm.zookeeper$exists_node_QMARK_$fn__1153.invoke(zookeeper.clj:101)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:98)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:114)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.cluster$mk_distributed_cluster_state$reify__1865.mkdirs(cluster.clj:109)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.cluster$mk_storm_cluster_state$reify__2284.report_error(cluster.clj:368)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$throttled_report_error_fn$fn__5421.invoke(executor.clj:178)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$mk_executor_data$fn__5474$fn__5475.invoke(executor.clj:237)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:441)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35]
>
>
> Thanks
>
> Hadi
>