You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Palak Shah <sp...@gmail.com> on 2014/09/04 17:24:21 UTC
Storm Worker process disconnects
I am running a Topology on Storm Cluster that uses the Kafka Spout. One of
my supervisor disconnects after a certain amount of time and is restarted
again. My supervisor logs for that node report the following -
Does anybody have some clue about this?
2014-09-04 19:10:11 b.s.d.supervisor [INFO] Launching worker with command:
'java' '-server' '-verbose:gc' '-XX:+PrintGCTimeStamps'
'-XX:+PrintGCDetails' '-Dcom.sun.management.jmxremote'
'-Dcom.sun.management.jmxremote.ssl=false'
'-Dcom.sun.management.jmxremote.authenticate=false'
'-Dcom.sun.management.jmxremote.port=16700'
'-Djava.library.path=/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources/Linux-amd64:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources:/usr/lib/jvm/java-7-oracle/lib'
'-Dlogfile.name=worker-6700.log'
'-Dstorm.home=/home/stormcluster/Storm/apache-storm-0.9.2-incubating'
'-Dlogback.configurationFile=/home/stormcluster/Storm/apache-storm-0.9.2-incubating/logback/cluster.xml'
'-Dstorm.id=palak-1-1409837530'
'-Dworker.id=88c6e738-cdc8-4b17-9f0d-849205dc9458' '-Dworker.port=6700'
'-cp'
'/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/conf:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/stormjar.jar'
'backtype.storm.daemon.worker' 'palak-1-1409837530'
'2350025d-9305-4481-8f99-23ef0cf3fdad' '6700'
'88c6e738-cdc8-4b17-9f0d-849205dc9458'
2014-09-04 19:10:13 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:13 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:14 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:14 b.s.d.supervisor [INFO] Removing code for storm id
palak-1-1409837530
2014-09-04 19:10:14 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:15 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:15 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:24 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:24 b.s.d.supervisor [INFO] Downloading code for storm id
palak-1-1409837530 from /tmp/stormtmp/nimbus/stormdist/palak-1-1409837530
2014-09-04 19:10:25 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:27 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:28 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:28 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:28 b.s.d.supervisor [INFO] Finished downloading code for
storm id palak-1-1409837530 from
/tmp/stormtmp/nimbus/stormdist/palak-1-1409837530
2014-09-04 19:10:29 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:29 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:30 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:35 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:36 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:36 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:21 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:21 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:22 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:22 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:23 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:23 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:24 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:24 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:25 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:25 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:12:10 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Worker
88c6e738-cdc8-4b17-9f0d-849205dc9458 failed to start
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shutting down and clearing
state for id 88c6e738-cdc8-4b17-9f0d-849205dc9458. Current supervisor time:
1409838131. State: :not-started, Heartbeat: nil
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shutting down
2350025d-9305-4481-8f99-23ef0cf3fdad:88c6e738-cdc8-4b17-9f0d-849205dc9458
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shut down
2350025d-9305-4481-8f99-23ef0cf3fdad:88c6e738-cdc8-4b17-9f0d-849205dc9458
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Launching worker with
assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"palak-1-1409837530", :executors ([5 5] [1 1])} for this supervisor
2350025d-9305-4481-8f99-23ef0cf3fdad on port 6700 with id
c391936c-8535-4040-9ad2-241cf46b5efa
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Launching worker with command:
'java' '-server' '-verbose:gc' '-XX:+PrintGCTimeStamps'
'-XX:+PrintGCDetails' '-Dcom.sun.management.jmxremote'
'-Dcom.sun.management.jmxremote.ssl=false'
'-Dcom.sun.management.jmxremote.authenticate=false'
'-Dcom.sun.management.jmxremote.port=16700'
'-Djava.library.path=/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources/Linux-amd64:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources:/usr/lib/jvm/java-7-oracle/lib'
'-Dlogfile.name=worker-6700.log'
'-Dstorm.home=/home/stormcluster/Storm/apache-storm-0.9.2-incubating'
'-Dlogback.configurationFile=/home/stormcluster/Storm/apache-storm-0.9.2-incubating/logback/cluster.xml'
'
.
.
.
Re: Storm Worker process disconnects
Posted by Palak Shah <sp...@gmail.com>.
Hi,
I increased the netty wait time in storm.yaml and it seemed like the issue
was solved.
storm.messaging.netty.max_retries: 300
storm.messaging.netty.max_wait_ms: 10000
storm.messaging.netty.min_wait_ms: 4000
But I got the same error again after a few hours. So the above fix just
delayed the error by a few hours. Can anyone suggest a permanent fix for
this?
Here are my worker logs at the time of failure -
2014-09-05 15:40:20 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=stormSlave1:2181, sessionTimeout=20000
watcher=org.apache.curator.ConnectionState@57a01ca7
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Opening socket connection to
server stormSlave1/192.168.1.57:2181. Will not attempt to authenticate
using SASL (unknown error)
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Socket connection established
to stormSlave1/192.168.1.57:2181, initiating session
2014-09-05 15:40:20 b.s.d.executor [INFO] Loading executor __system:[-1 -1]
2014-09-05 15:40:20 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-09-05 15:40:20 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=stormSlave1:2181 sessionTimeout=20000
watcher=org.apache.curator.ConnectionState@4d49e1ca
2014-09-05 15:40:20 b.s.d.executor [INFO] Loaded executor tasks
__system:[-1 -1]
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Opening socket connection to
server stormSlave1/192.168.1.57:2181. Will not attempt to authenticate
using SASL (unknown error)
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Socket connection established
to stormSlave1/192.168.1.57:2181, initiating session
2014-09-05 15:40:20 b.s.d.executor [INFO] Finished loading executor
__system:[-1 -1]
2014-09-05 15:40:20 b.s.d.worker [INFO] Launching receive-thread for
821e9e5c-f2de-4a5b-892a-b74ad1f6fae6:6700
2014-09-05 15:40:20 b.s.d.executor [INFO] Preparing bolt __system:(-1)
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Session establishment complete
on server stormSlave1/192.168.1.57:2181, sessionid = 0x148448b63260158,
negotiated timeout = 20000
2014-09-05 15:40:20 o.a.c.f.s.ConnectionStateManager [INFO] State change:
CONNECTED
2014-09-05 15:40:20 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2014-09-05 15:40:20 b.s.d.executor [INFO] Prepared bolt __system:(-1)
2014-09-05 15:40:20 b.s.m.n.Server [INFO] Create Netty Server
Netty-server-localhost-6700, buffer_size: 5242880, maxWorkers: 1
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Session establishment complete
on server stormSlave1/192.168.1.57:2181, sessionid = 0x148448b63260159,
negotiated timeout = 20000
2014-09-05 15:40:20 o.a.c.f.s.ConnectionStateManager [INFO] State change:
CONNECTED
2014-09-05 15:40:20 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2014-09-05 15:40:20 b.s.m.loader [INFO] Starting receive-thread: [stormId:
popeye-3-1409900360, port: 6700, thread-id: 0 ]
2014-09-05 15:40:20 s.k.DynamicBrokersReader [INFO] Read partition info
from zookeeper:
GlobalPartitionInformation{partitionMap={0=stormMaster:9093,
1=stormMaster:9094}}
2014-09-05 15:40:20 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-09-05 15:40:20 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=stormSlave1:2181 sessionTimeout=20000
watcher=org.apache.curator.ConnectionState@28ebe0a7
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Opening socket connection to
server stormSlave1/192.168.1.57:2181. Will not attempt to authenticate
using SASL (unknown error)
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Socket connection established
to stormSlave1/192.168.1.57:2181, initiating session
2014-09-05 15:40:20 b.s.d.executor [INFO] Opened spout kafkaSpout:(7)
2014-09-05 15:40:20 b.s.d.executor [INFO] Activating spout kafkaSpout:(7)
2014-09-05 15:40:20 s.k.ZkCoordinator [INFO] Task [1/1] Refreshing
partition manager connections
2014-09-05 15:40:20 o.a.z.ClientCnxn [INFO] Session establishment complete
on server stormSlave1/192.168.1.57:2181, sessionid = 0x148448b6326015a,
negotiated timeout = 20000
2014-09-05 15:40:20 o.a.c.f.s.ConnectionStateManager [INFO] State change:
CONNECTED
2014-09-05 15:40:20 o.a.c.f.s.ConnectionStateManager [WARN] There are no
ConnectionStateListeners registered.
2014-09-05 15:40:20 s.k.DynamicBrokersReader [INFO] Read partition info
from zookeeper:
GlobalPartitionInformation{partitionMap={0=stormMaster:9093,
1=stormMaster:9094}}
2014-09-05 15:40:20 s.k.KafkaUtils [INFO] Task [1/1] assigned
[Partition{host=stormMaster:9093, partition=0},
Partition{host=stormMaster:9094, partition=1}]
2014-09-05 15:40:20 s.k.ZkCoordinator [INFO] Task [1/1] Deleted partition
managers: []
2014-09-05 15:40:20 s.k.ZkCoordinator [INFO] Task [1/1] New partition
managers: [Partition{host=stormMaster:9094, partition=1},
Partition{host=stormMaster:9093, partition=0}]
On Thu, Sep 4, 2014 at 9:14 PM, Pavan Jakati G <Pa...@microland.com>
wrote:
> Check what worker log has to stay. May be storm user permission issue.
> Try running the command that's there in supervisor log and validate the
> output .
>
>
>
> -----Original Message-----
> From: Palak Shah [mailto:spalak12@gmail.com <sp...@gmail.com>]
> Sent: Thu 9/4/2014 8:54 PM
> To: user@storm.incubator.apache.org
> Subject: Storm Worker process disconnects
>
> I am running a Topology on Storm Cluster that uses the Kafka Spout. One of
> my supervisor disconnects after a certain amount of time and is restarted
> again. My supervisor logs for that node report the following -
>
> Does anybody have some clue about this?
>
> 2014-09-04 19:10:11 b.s.d.supervisor [INFO] Launching worker with command:
> 'java' '-server' '-verbose:gc' '-XX:+PrintGCTimeStamps'
> '-XX:+PrintGCDetails' '-Dcom.sun.management.jmxremote'
> '-Dcom.sun.management.jmxremote.ssl=false'
> '-Dcom.sun.management.jmxremote.authenticate=false'
> '-Dcom.sun.management.jmxremote.port=16700'
>
> '-Djava.library.path=/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources/Linux-amd64:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources:/usr/lib/jvm/java-7-oracle/lib'
> '-Dlogfile.name=worker-6700.log'
> '-Dstorm.home=/home/stormcluster/Storm/apache-storm-0.9.2-incubating'
>
> '-Dlogback.configurationFile=/home/stormcluster/Storm/apache-storm-0.9.2-incubating/logback/cluster.xml'
> '-Dstorm.id=palak-1-1409837530'
> '-Dworker.id=88c6e738-cdc8-4b17-9f0d-849205dc9458' '-Dworker.port=6700'
> '-cp'
>
> '/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/conf:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/stormjar.jar'
> 'backtype.storm.daemon.worker' 'palak-1-1409837530'
> '2350025d-9305-4481-8f99-23ef0cf3fdad' '6700'
> '88c6e738-cdc8-4b17-9f0d-849205dc9458'
> 2014-09-04 19:10:13 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:13 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:14 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:14 b.s.d.supervisor [INFO] Removing code for storm id
> palak-1-1409837530
> 2014-09-04 19:10:14 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:15 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:15 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:24 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:24 b.s.d.supervisor [INFO] Downloading code for storm id
> palak-1-1409837530 from /tmp/stormtmp/nimbus/stormdist/palak-1-1409837530
> 2014-09-04 19:10:25 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:27 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:28 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:28 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:28 b.s.d.supervisor [INFO] Finished downloading code for
> storm id palak-1-1409837530 from
> /tmp/stormtmp/nimbus/stormdist/palak-1-1409837530
> 2014-09-04 19:10:29 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:29 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:30 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:35 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:36 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:10:36 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:21 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:21 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:22 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:22 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:23 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:23 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:24 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:24 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:25 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:11:25 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:12:10 b.s.d.supervisor [INFO]
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
> 2014-09-04 19:12:11 b.s.d.supervisor [INFO] Worker
> 88c6e738-cdc8-4b17-9f0d-849205dc9458 failed to start
> 2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shutting down and clearing
> state for id 88c6e738-cdc8-4b17-9f0d-849205dc9458. Current supervisor time:
> 1409838131. State: :not-started, Heartbeat: nil
> 2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shutting down
> 2350025d-9305-4481-8f99-23ef0cf3fdad:88c6e738-cdc8-4b17-9f0d-849205dc9458
> 2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shut down
> 2350025d-9305-4481-8f99-23ef0cf3fdad:88c6e738-cdc8-4b17-9f0d-849205dc9458
> 2014-09-04 19:12:11 b.s.d.supervisor [INFO] Launching worker with
> assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
> "palak-1-1409837530", :executors ([5 5] [1 1])} for this supervisor
> 2350025d-9305-4481-8f99-23ef0cf3fdad on port 6700 with id
> c391936c-8535-4040-9ad2-241cf46b5efa
> 2014-09-04 19:12:11 b.s.d.supervisor [INFO] Launching worker with command:
> 'java' '-server' '-verbose:gc' '-XX:+PrintGCTimeStamps'
> '-XX:+PrintGCDetails' '-Dcom.sun.management.jmxremote'
> '-Dcom.sun.management.jmxremote.ssl=false'
> '-Dcom.sun.management.jmxremote.authenticate=false'
> '-Dcom.sun.management.jmxremote.port=16700'
>
> '-Djava.library.path=/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources/Linux-amd64:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources:/usr/lib/jvm/java-7-oracle/lib'
> '-Dlogfile.name=worker-6700.log'
> '-Dstorm.home=/home/stormcluster/Storm/apache-storm-0.9.2-incubating'
>
> '-Dlogback.configurationFile=/home/stormcluster/Storm/apache-storm-0.9.2-incubating/logback/cluster.xml'
> '
> .
> .
> .
>
> The information contained in this transmission may contain privileged
> and confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
>
RE: Storm Worker process disconnects
Posted by Pavan Jakati G <Pa...@microland.com>.
Check what worker log has to stay. May be storm user permission issue. Try running the command that's there in supervisor log and validate the output .
-----Original Message-----
From: Palak Shah [mailto:spalak12@gmail.com]
Sent: Thu 9/4/2014 8:54 PM
To: user@storm.incubator.apache.org
Subject: Storm Worker process disconnects
I am running a Topology on Storm Cluster that uses the Kafka Spout. One of
my supervisor disconnects after a certain amount of time and is restarted
again. My supervisor logs for that node report the following -
Does anybody have some clue about this?
2014-09-04 19:10:11 b.s.d.supervisor [INFO] Launching worker with command:
'java' '-server' '-verbose:gc' '-XX:+PrintGCTimeStamps'
'-XX:+PrintGCDetails' '-Dcom.sun.management.jmxremote'
'-Dcom.sun.management.jmxremote.ssl=false'
'-Dcom.sun.management.jmxremote.authenticate=false'
'-Dcom.sun.management.jmxremote.port=16700'
'-Djava.library.path=/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources/Linux-amd64:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources:/usr/lib/jvm/java-7-oracle/lib'
'-Dlogfile.name=worker-6700.log'
'-Dstorm.home=/home/stormcluster/Storm/apache-storm-0.9.2-incubating'
'-Dlogback.configurationFile=/home/stormcluster/Storm/apache-storm-0.9.2-incubating/logback/cluster.xml'
'-Dstorm.id=palak-1-1409837530'
'-Dworker.id=88c6e738-cdc8-4b17-9f0d-849205dc9458' '-Dworker.port=6700'
'-cp'
'/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/home/stormcluster/Storm/apache-storm-0.9.2-incubating/conf:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/stormjar.jar'
'backtype.storm.daemon.worker' 'palak-1-1409837530'
'2350025d-9305-4481-8f99-23ef0cf3fdad' '6700'
'88c6e738-cdc8-4b17-9f0d-849205dc9458'
2014-09-04 19:10:13 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:13 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:14 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:14 b.s.d.supervisor [INFO] Removing code for storm id
palak-1-1409837530
2014-09-04 19:10:14 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:15 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:15 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:24 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:24 b.s.d.supervisor [INFO] Downloading code for storm id
palak-1-1409837530 from /tmp/stormtmp/nimbus/stormdist/palak-1-1409837530
2014-09-04 19:10:25 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:27 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:28 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:28 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:28 b.s.d.supervisor [INFO] Finished downloading code for
storm id palak-1-1409837530 from
/tmp/stormtmp/nimbus/stormdist/palak-1-1409837530
2014-09-04 19:10:29 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:29 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:30 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:35 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:36 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:10:36 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:21 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:21 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:22 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:22 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:23 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:23 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:24 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:24 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:25 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:11:25 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:12:10 b.s.d.supervisor [INFO]
88c6e738-cdc8-4b17-9f0d-849205dc9458 still hasn't started
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Worker
88c6e738-cdc8-4b17-9f0d-849205dc9458 failed to start
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shutting down and clearing
state for id 88c6e738-cdc8-4b17-9f0d-849205dc9458. Current supervisor time:
1409838131. State: :not-started, Heartbeat: nil
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shutting down
2350025d-9305-4481-8f99-23ef0cf3fdad:88c6e738-cdc8-4b17-9f0d-849205dc9458
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Shut down
2350025d-9305-4481-8f99-23ef0cf3fdad:88c6e738-cdc8-4b17-9f0d-849205dc9458
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Launching worker with
assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"palak-1-1409837530", :executors ([5 5] [1 1])} for this supervisor
2350025d-9305-4481-8f99-23ef0cf3fdad on port 6700 with id
c391936c-8535-4040-9ad2-241cf46b5efa
2014-09-04 19:12:11 b.s.d.supervisor [INFO] Launching worker with command:
'java' '-server' '-verbose:gc' '-XX:+PrintGCTimeStamps'
'-XX:+PrintGCDetails' '-Dcom.sun.management.jmxremote'
'-Dcom.sun.management.jmxremote.ssl=false'
'-Dcom.sun.management.jmxremote.authenticate=false'
'-Dcom.sun.management.jmxremote.port=16700'
'-Djava.library.path=/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources/Linux-amd64:/tmp/stormtmp/supervisor/stormdist/palak-1-1409837530/resources:/usr/lib/jvm/java-7-oracle/lib'
'-Dlogfile.name=worker-6700.log'
'-Dstorm.home=/home/stormcluster/Storm/apache-storm-0.9.2-incubating'
'-Dlogback.configurationFile=/home/stormcluster/Storm/apache-storm-0.9.2-incubating/logback/cluster.xml'
'
.
.
.
The information contained in this transmission may contain privileged and confidential information of Microland Limited, including information protected by privacy laws. It is intended only for the use of Microland Limited. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, or duplication of this communication is strictly prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.