You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Muhammad Haseeb Javed <11...@seecs.edu.pk> on 2016/12/27 01:58:27 UTC
Storm topology does not start processing
I am launching a Storm v1.0.1 cluster with 1 nimbus and 4 supervisor nodes.
I have 2 zookeeper nodes. When I try to launch a sample Topology which
should be consuming from Kafka. However upon inspecting nimbus and
supervisor logs after the topolgy has been submitted, I discovered that
workers are not launching.
storm.yaml
storm.zookeeper.servers:
- "storage14"
- "storage16"
nimbus.host: "storage01"
nimbus.seeds : ["storage01"]
storm.local.dir: "/tmp/storm"
supervisor.slots.ports:
- 6700
worker.heap.memory.mb: 8192
supervisor.memory.capacity.mb: 10240
Nimbus log:
2016-12-26 03:28:47.921 o.a.s.s.EvenScheduler [INFO] Available slots:
(["c941fac0-be93-4ea3-8a39-c316a8430112" 6700]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700]
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700])
2016-12-26 03:28:47.972 o.a.s.d.nimbus [INFO] Setting new assignment
for topology id identity-1-1482740925:
#org.apache.storm.daemon.common.Assignment{:master-code-dir
"/tmp/storm", :node->host {"c941fac0-be93-4ea3-8a39-c316a8430112"
"storage02", "9edace71-d7b6-42d1-a7bd-964851669a17" "storage03",
"fc35bd75-dd24-49fc-af90-e6493cdb1e8f" "storage05",
"3363bfc3-4083-4a25-af82-0709f43e73cb" "storage04"},
:executor->node+port {[8 8] ["9edace71-d7b6-42d1-a7bd-964851669a17"
6700], [12 12] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [2 2]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [32 32]
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [7 7]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [22 22]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [3 3]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [33 33]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [24 24]
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [1 1]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [35 35]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [18 18]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [6 6]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [28 28]
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [20 20]
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [9 9]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [30 30]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [23 23]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [11 11]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [31 31]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [16 16]
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [13 13]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [19 19]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [21 21]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [5 5]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [27 27]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [29 29]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [26 26]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [10 10]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [14 14]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [4 4]
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [36 36]
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [15 15]
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [34 34]
["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [25 25]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [17 17]
["c941fac0-be93-4ea3-8a39-c316a8430112" 6700]},
:executor->start-time-secs {[8 8] 1482740927, [12 12] 1482740927, [2
2] 1482740927, [32 32] 1482740927, [7 7] 1482740927, [22 22]
1482740927, [3 3] 1482740927, [33 33] 1482740927, [24 24] 1482740927,
[1 1] 1482740927, [35 35] 1482740927, [18 18] 1482740927, [6 6]
1482740927, [28 28] 1482740927, [20 20] 1482740927, [9 9] 1482740927,
[30 30] 1482740927, [23 23] 1482740927, [11 11] 1482740927, [31 31]
1482740927, [16 16] 1482740927, [13 13] 1482740927, [19 19]
1482740927, [21 21] 1482740927, [5 5] 1482740927, [27 27] 1482740927,
[29 29] 1482740927, [26 26] 1482740927, [10 10] 1482740927, [14 14]
1482740927, [4 4] 1482740927, [36 36] 1482740927, [15 15] 1482740927,
[34 34] 1482740927, [25 25] 1482740927, [17 17] 1482740927},
:worker->resources {["c941fac0-be93-4ea3-8a39-c316a8430112" 6700] [0.0
0.0 0.0], ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700] [0.0 0.0 0.0],
["9edace71-d7b6-42d1-a7bd-964851669a17" 6700] [0.0 0.0 0.0],
["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700] [0.0 0.0 0.0]}}
2016-12-26 03:28:48.123 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormjar.jar with id
880ba380-ace1-4622-8765-f1bdd977b57d
2016-12-26 03:28:48.127 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormjar.jar with id
6f0ccc65-6968-4bda-8cc6-13b719e44128
2016-12-26 03:28:48.132 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormjar.jar with id
14cb58cd-45b2-4f75-bbd1-3aa20a6cc76f
2016-12-26 03:28:48.142 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormjar.jar with id
3b7100e6-9a4a-44b2-a5ec-2637da91453f
2016-12-26 03:28:48.762 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormcode.ser with id
1e0546d8-43e7-4422-8290-b00d6e8856da
2016-12-26 03:28:48.767 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormcode.ser with id
16137c26-eb4d-4000-bda0-658197b30e2f
2016-12-26 03:28:48.772 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormconf.ser with id
a939e4b1-0f52-4abc-8456-0fb504307221
2016-12-26 03:28:48.777 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormcode.ser with id
2d8d4769-14a8-490c-9737-5c4cf34d0e2d
2016-12-26 03:28:48.783 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormconf.ser with id
49728932-320c-4a05-acbf-cae872d10651
2016-12-26 03:28:48.789 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormconf.ser with id
13513809-a15c-43e0-8cbc-c4ac3b43f23e
2016-12-26 03:28:48.873 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormcode.ser with id
07f54afe-695e-4253-bb3b-04c9701d0578
2016-12-26 03:28:48.881 o.a.s.d.nimbus [INFO] Created download session
for identity-1-1482740925-stormconf.ser with id
1bc067a0-f88d-459e-bb92-9d92535d5bb8
2016-12-26 03:30:49.418 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[8 8] not alive
2016-12-26 03:30:49.418 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[12 12] not alive
2016-12-26 03:30:49.418 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[2 2] not alive
2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[32 32] not alive
2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[7 7] not alive
2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[22 22] not alive
2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[3 3] not alive
2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[33 33] not alive
2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[24 24] not alive
2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[1 1] not alive
2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[35 35] not alive
2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[18 18] not alive
2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[6 6] not alive
2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[28 28] not alive
2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[20 20] not alive
2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[9 9] not alive
2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[30 30] not alive
2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[23 23] not alive
2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[11 11] not alive
2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[31 31] not alive
2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[16 16] not alive
2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[13 13] not alive
2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[19 19] not alive
2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[21 21] not alive
2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[5 5] not alive
2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[27 27] not alive
2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[29 29] not alive
2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[26 26] not alive
2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[10 10] not alive
2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[14 14] not alive
2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[4 4] not alive
2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[36 36] not alive
2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[15 15] not alive
2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[34 34] not alive
2016-12-26 03:30:49.423 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[25 25] not alive
2016-12-26 03:30:49.423 o.a.s.d.nimbus [INFO] Executor
identity-1-1482740925:[17 17] not alive
2016-12-26 03:30:49.447 o.a.s.d.nimbus [INFO] Setting new assignment
for topology id identity-1-1482740925:
#org.apache.storm.daemon.common.Assignment{:master-code-dir
"/tmp/storm", :node->host {}, :executor->node+port {},
:executor->start-time-secs {[8 8] 1482740927, [12 12] 1482740927, [2
2] 1482740927, [32 32] 1482740927, [7 7] 1482740927, [22 22]
1482740927, [3 3] 1482740927, [33 33] 1482740927, [24 24] 1482740927,
[1 1] 1482740927, [35 35] 1482740927, [18 18] 1482740927, [6 6]
1482740927, [28 28] 1482740927, [20 20] 1482740927, [9 9] 1482740927,
[30 30] 1482740927, [23 23] 1482740927, [11 11] 1482740927, [31 31]
1482740927, [16 16] 1482740927, [13 13] 1482740927, [19 19]
1482740927, [21 21] 1482740927, [5 5] 1482740927, [27 27] 1482740927,
[29 29] 1482740927, [26 26] 1482740927, [10 10] 1482740927, [14 14]
1482740927, [4 4] 1482740927, [36 36] 1482740927, [15 15] 1482740927,
[34 34] 1482740927, [25 25] 1482740927, [17 17] 1482740927},
:worker->resources {}}
Supervisor log:
2016-12-26 03:30:49.362 o.a.s.config [INFO] GET worker-user
ace3c157-ec78-49f9-af52-0f78d62bd29f
2016-12-26 03:30:49.369 o.a.s.config [INFO] REMOVE worker-user
ace3c157-ec78-49f9-af52-0f78d62bd29f
2016-12-26 03:30:49.369 o.a.s.d.supervisor [INFO] Shut down
9edace71-d7b6-42d1-a7bd-964851669a17:ace3c157-ec78-49f9-af52-0f78d62bd29f
2016-12-26 03:30:49.370 o.a.s.d.supervisor [INFO] Launching worker
with assignment {:storm-id "identity-1-1482740925", :executors [[8 8]
[12 12] [32 32] [24 24] [28 28] [20 20] [16 16] [4 4] [36 36]],
:resources #object[org.apache.storm.generated.WorkerResources
0x7b1ae274 "WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0,
cpu:0.0)"]} for this supervisor 9edace71-d7b6-42d1-a7bd-964851669a17
on port 6700 with id 14ce240d-5ef3-43be-9cb8-752ef73333cf
2016-12-26 03:30:49.385 o.a.s.d.supervisor [INFO] Launching worker
with command: '/opt/java/1.7.0//bin/java' '-cp'
'/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/reflectasm-1.10.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/clojure-1.7.0.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/minlog-1.3.0.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/storm-rename-hack-1.0.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/objenesis-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/disruptor-3.3.2.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/storm-core-1.0.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/asm-5.0.3.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-core-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-over-slf4j-1.6.6.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/servlet-api-2.5.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/kryo-3.0.3.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/slf4j-api-1.7.7.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-api-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-slf4j-impl-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/conf:/tmp/storm/supervisor/stormdist/identity-1-1482740925/stormjar.jar'
'-Xmx64m' '-Dlogfile.name=worker.log'
'-Dstorm.home=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1'
'-Dworkers.artifacts=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/logs/workers-artifacts'
'-Dstorm.id=identity-1-1482740925'
'-Dworker.id=14ce240d-5ef3-43be-9cb8-752ef73333cf'
'-Dworker.port=6700'
'-Dstorm.log.dir=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/logs'
'-Dlog4j.configurationFile=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/log4j2/worker.xml'
'-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector'
'org.apache.storm.LogWriter' '/opt/java/1.7.0//bin/java' '-server'
'-Xmx8192m' '-XX:+PrintGCDetails' '-Xloggc:artifacts/gc.log'
'-XX:+PrintGCDateStamps' '-XX:+PrintGCTimeStamps'
'-XX:+UseGCLogFileRotation' '-XX:NumberOfGCLogFiles=10'
'-XX:GCLogFileSize=1M' '-XX:+HeapDumpOnOutOfMemoryError'
'-XX:HeapDumpPath=artifacts/heapdump'
'-Djava.library.path=/tmp/storm/supervisor/stormdist/identity-1-1482740925/resources/Linux-amd64:/tmp/storm/supervisor/stormdist/identity-1-1482740925/resources:/usr/local/lib:/opt/local/lib:/usr/lib'
'-Dlogfile.name=worker.log'
'-Dstorm.home=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1'
'-Dworkers.artifacts=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/logs/workers-artifacts'
'-Dstorm.conf.file=' '-Dstorm.options='
'-Dstorm.log.dir=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/logs'
'-Djava.io.tmpdir=/tmp/storm/workers/14ce240d-5ef3-43be-9cb8-752ef73333cf/tmp'
'-Dlogging.sensitivity=S3'
'-Dlog4j.configurationFile=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/log4j2/worker.xml'
'-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector'
'-Dstorm.id=identity-1-1482740925'
'-Dworker.id=14ce240d-5ef3-43be-9cb8-752ef73333cf'
'-Dworker.port=6700' '-cp'
'/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/reflectasm-1.10.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/clojure-1.7.0.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/minlog-1.3.0.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/storm-rename-hack-1.0.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/objenesis-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/disruptor-3.3.2.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/storm-core-1.0.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/asm-5.0.3.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/listreaming-benchmarks/apache-storm-1.0.1/lib/log4j-over-slf4j-1.6.6.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/servlet-api-2.5.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/kryo-3.0.3.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/slf4j-api-1.7.7.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-api-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-slf4j-impl-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/conf:/tmp/storm/supervisor/stormdist/identity-1-1482740925/stormjar.jar'
'org.apache.storm.daemon.worker' 'identity-1-1482740925'
'3363bfc3-4083-4a25-af82-0709f43e73cb' '6700'
'5fbae825-35ee-45f6-bced-f91a73037e10'
2016-12-26 03:30:49.316 o.a.s.config [INFO] SET worker-user
5fbae825-35ee-45f6-bced-f91a73037e10 javed.19
2016-12-26 03:30:49.316 o.a.s.d.supervisor [INFO] Creating symlinks
for worker-id: 5fbae825-35ee-45f6-bced-f91a73037e10 storm-id:
identity-1-1482740925 to its port artifacts directory
2016-12-26 03:30:49.325 o.a.s.d.supervisor [INFO] Creating symlinks
for worker-id: 5fbae825-35ee-45f6-bced-f91a73037e10 storm-id:
identity-1-1482740925 for files(1): ("resources")
2016-12-26 03:30:49.329 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:49.829 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:49.849 o.a.s.d.supervisor [INFO] Worker Process
5fbae825-35ee-45f6-bced-f91a73037e10 exited with code: 1
2016-12-26 03:30:50.329 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:50.830 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:51.330 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:51.830 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:52.331 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:52.831 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:53.331 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:53.832 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:54.332 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:54.832 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:55.333 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:55.833 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:56.333 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:56.834 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:57.334 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:57.834 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:58.335 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:58.835 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:59.335 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:30:59.836 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:00.336 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:00.837 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:01.337 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:01.837 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:02.338 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:02.838 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:03.338 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:03.839 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:04.339 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:04.839 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:05.340 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:05.840 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:06.340 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:06.841 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:07.341 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:07.841 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:08.342 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:08.842 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:09.342 o.a.s.d.supervisor [INFO]
5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
2016-12-26 03:31:09.813 o.a.s.u.NimbusClient [WARN] Using deprecated
config nimbus.host for backward compatibility. Please update your
storm.yaml so it only has config nimbus.seeds
Also I have confirmed that nimbus and supervisors are reachable from each
other. What else could be causing this issue ?
P.S: You might wonder why I have both nimbus.host and nimbus.seeds property
in my storm.yaml. If I only use numbs.seeds, Storm UI does not detect any
nimbus saying : Found nimbuses [] none of which is elected as leader,
please try again after some t
Re: Storm topology does not start processing
Posted by Mostafa Gomaa <mg...@trendak.com>.
You should use only nimbus.seeds, nimbus.host is deprecated. Make sure that
zookeeper is up and running, and that the nimbus machine can reach the
zookeeper machine. Also are the machines resolvable from DNS?, if not then
make sure that /etc/hosts is properly configured.
On Tue, Dec 27, 2016 at 3:58 AM, Muhammad Haseeb Javed <
11besemjaved@seecs.edu.pk> wrote:
> I am launching a Storm v1.0.1 cluster with 1 nimbus and 4 supervisor
> nodes. I have 2 zookeeper nodes. When I try to launch a sample Topology
> which should be consuming from Kafka. However upon inspecting nimbus and
> supervisor logs after the topolgy has been submitted, I discovered that
> workers are not launching.
>
> storm.yaml
>
> storm.zookeeper.servers:
> - "storage14"
> - "storage16"
>
> nimbus.host: "storage01"
> nimbus.seeds : ["storage01"]
>
> storm.local.dir: "/tmp/storm"
>
> supervisor.slots.ports:
> - 6700
>
> worker.heap.memory.mb: 8192
> supervisor.memory.capacity.mb: 10240
>
> Nimbus log:
>
> 2016-12-26 03:28:47.921 o.a.s.s.EvenScheduler [INFO] Available slots: (["c941fac0-be93-4ea3-8a39-c316a8430112" 6700] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700])
> 2016-12-26 03:28:47.972 o.a.s.d.nimbus [INFO] Setting new assignment for topology id identity-1-1482740925: #org.apache.storm.daemon.common.Assignment{:master-code-dir "/tmp/storm", :node->host {"c941fac0-be93-4ea3-8a39-c316a8430112" "storage02", "9edace71-d7b6-42d1-a7bd-964851669a17" "storage03", "fc35bd75-dd24-49fc-af90-e6493cdb1e8f" "storage05", "3363bfc3-4083-4a25-af82-0709f43e73cb" "storage04"}, :executor->node+port {[8 8] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [12 12] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [2 2] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [32 32] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [7 7] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [22 22] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [3 3] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [33 33] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [24 24] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [1 1] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [35 35] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [18 18] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [6 6] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [28 28] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [20 20] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [9 9] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [30 30] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [23 23] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [11 11] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [31 31] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [16 16] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [13 13] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [19 19] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [21 21] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [5 5] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [27 27] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [29 29] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [26 26] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [10 10] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [14 14] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [4 4] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [36 36] ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700], [15 15] ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700], [34 34] ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700], [25 25] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700], [17 17] ["c941fac0-be93-4ea3-8a39-c316a8430112" 6700]}, :executor->start-time-secs {[8 8] 1482740927, [12 12] 1482740927, [2 2] 1482740927, [32 32] 1482740927, [7 7] 1482740927, [22 22] 1482740927, [3 3] 1482740927, [33 33] 1482740927, [24 24] 1482740927, [1 1] 1482740927, [35 35] 1482740927, [18 18] 1482740927, [6 6] 1482740927, [28 28] 1482740927, [20 20] 1482740927, [9 9] 1482740927, [30 30] 1482740927, [23 23] 1482740927, [11 11] 1482740927, [31 31] 1482740927, [16 16] 1482740927, [13 13] 1482740927, [19 19] 1482740927, [21 21] 1482740927, [5 5] 1482740927, [27 27] 1482740927, [29 29] 1482740927, [26 26] 1482740927, [10 10] 1482740927, [14 14] 1482740927, [4 4] 1482740927, [36 36] 1482740927, [15 15] 1482740927, [34 34] 1482740927, [25 25] 1482740927, [17 17] 1482740927}, :worker->resources {["c941fac0-be93-4ea3-8a39-c316a8430112" 6700] [0.0 0.0 0.0], ["fc35bd75-dd24-49fc-af90-e6493cdb1e8f" 6700] [0.0 0.0 0.0], ["9edace71-d7b6-42d1-a7bd-964851669a17" 6700] [0.0 0.0 0.0], ["3363bfc3-4083-4a25-af82-0709f43e73cb" 6700] [0.0 0.0 0.0]}}
> 2016-12-26 03:28:48.123 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormjar.jar with id 880ba380-ace1-4622-8765-f1bdd977b57d
> 2016-12-26 03:28:48.127 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormjar.jar with id 6f0ccc65-6968-4bda-8cc6-13b719e44128
> 2016-12-26 03:28:48.132 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormjar.jar with id 14cb58cd-45b2-4f75-bbd1-3aa20a6cc76f
> 2016-12-26 03:28:48.142 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormjar.jar with id 3b7100e6-9a4a-44b2-a5ec-2637da91453f
> 2016-12-26 03:28:48.762 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormcode.ser with id 1e0546d8-43e7-4422-8290-b00d6e8856da
> 2016-12-26 03:28:48.767 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormcode.ser with id 16137c26-eb4d-4000-bda0-658197b30e2f
> 2016-12-26 03:28:48.772 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormconf.ser with id a939e4b1-0f52-4abc-8456-0fb504307221
> 2016-12-26 03:28:48.777 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormcode.ser with id 2d8d4769-14a8-490c-9737-5c4cf34d0e2d
> 2016-12-26 03:28:48.783 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormconf.ser with id 49728932-320c-4a05-acbf-cae872d10651
> 2016-12-26 03:28:48.789 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormconf.ser with id 13513809-a15c-43e0-8cbc-c4ac3b43f23e
> 2016-12-26 03:28:48.873 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormcode.ser with id 07f54afe-695e-4253-bb3b-04c9701d0578
> 2016-12-26 03:28:48.881 o.a.s.d.nimbus [INFO] Created download session for identity-1-1482740925-stormconf.ser with id 1bc067a0-f88d-459e-bb92-9d92535d5bb8
> 2016-12-26 03:30:49.418 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[8 8] not alive
> 2016-12-26 03:30:49.418 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[12 12] not alive
> 2016-12-26 03:30:49.418 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[2 2] not alive
> 2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[32 32] not alive
> 2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[7 7] not alive
> 2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[22 22] not alive
> 2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[3 3] not alive
> 2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[33 33] not alive
> 2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[24 24] not alive
> 2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[1 1] not alive
> 2016-12-26 03:30:49.419 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[35 35] not alive
> 2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[18 18] not alive
> 2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[6 6] not alive
> 2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[28 28] not alive
> 2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[20 20] not alive
> 2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[9 9] not alive
> 2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[30 30] not alive
> 2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[23 23] not alive
> 2016-12-26 03:30:49.420 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[11 11] not alive
> 2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[31 31] not alive
> 2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[16 16] not alive
> 2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[13 13] not alive
> 2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[19 19] not alive
> 2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[21 21] not alive
> 2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[5 5] not alive
> 2016-12-26 03:30:49.421 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[27 27] not alive
> 2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[29 29] not alive
> 2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[26 26] not alive
> 2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[10 10] not alive
> 2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[14 14] not alive
> 2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[4 4] not alive
> 2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[36 36] not alive
> 2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[15 15] not alive
> 2016-12-26 03:30:49.422 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[34 34] not alive
> 2016-12-26 03:30:49.423 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[25 25] not alive
> 2016-12-26 03:30:49.423 o.a.s.d.nimbus [INFO] Executor identity-1-1482740925:[17 17] not alive
> 2016-12-26 03:30:49.447 o.a.s.d.nimbus [INFO] Setting new assignment for topology id identity-1-1482740925: #org.apache.storm.daemon.common.Assignment{:master-code-dir "/tmp/storm", :node->host {}, :executor->node+port {}, :executor->start-time-secs {[8 8] 1482740927, [12 12] 1482740927, [2 2] 1482740927, [32 32] 1482740927, [7 7] 1482740927, [22 22] 1482740927, [3 3] 1482740927, [33 33] 1482740927, [24 24] 1482740927, [1 1] 1482740927, [35 35] 1482740927, [18 18] 1482740927, [6 6] 1482740927, [28 28] 1482740927, [20 20] 1482740927, [9 9] 1482740927, [30 30] 1482740927, [23 23] 1482740927, [11 11] 1482740927, [31 31] 1482740927, [16 16] 1482740927, [13 13] 1482740927, [19 19] 1482740927, [21 21] 1482740927, [5 5] 1482740927, [27 27] 1482740927, [29 29] 1482740927, [26 26] 1482740927, [10 10] 1482740927, [14 14] 1482740927, [4 4] 1482740927, [36 36] 1482740927, [15 15] 1482740927, [34 34] 1482740927, [25 25] 1482740927, [17 17] 1482740927}, :worker->resources {}}
>
> Supervisor log:
>
> 2016-12-26 03:30:49.362 o.a.s.config [INFO] GET worker-user ace3c157-ec78-49f9-af52-0f78d62bd29f
> 2016-12-26 03:30:49.369 o.a.s.config [INFO] REMOVE worker-user ace3c157-ec78-49f9-af52-0f78d62bd29f
> 2016-12-26 03:30:49.369 o.a.s.d.supervisor [INFO] Shut down 9edace71-d7b6-42d1-a7bd-964851669a17:ace3c157-ec78-49f9-af52-0f78d62bd29f
> 2016-12-26 03:30:49.370 o.a.s.d.supervisor [INFO] Launching worker with assignment {:storm-id "identity-1-1482740925", :executors [[8 8] [12 12] [32 32] [24 24] [28 28] [20 20] [16 16] [4 4] [36 36]], :resources #object[org.apache.storm.generated.WorkerResources 0x7b1ae274 "WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0)"]} for this supervisor 9edace71-d7b6-42d1-a7bd-964851669a17 on port 6700 with id 14ce240d-5ef3-43be-9cb8-752ef73333cf
> 2016-12-26 03:30:49.385 o.a.s.d.supervisor [INFO] Launching worker with command: '/opt/java/1.7.0//bin/java' '-cp' '/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/reflectasm-1.10.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/clojure-1.7.0.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/minlog-1.3.0.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/storm-rename-hack-1.0.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/objenesis-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/disruptor-3.3.2.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/storm-core-1.0.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/asm-5.0.3.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-core-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-over-slf4j-1.6.6.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/servlet-api-2.5.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/kryo-3.0.3.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/slf4j-api-1.7.7.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-api-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-slf4j-impl-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/conf:/tmp/storm/supervisor/stormdist/identity-1-1482740925/stormjar.jar' '-Xmx64m' '-Dlogfile.name=worker.log' '-Dstorm.home=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1' '-Dworkers.artifacts=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/logs/workers-artifacts' '-Dstorm.id=identity-1-1482740925' '-Dworker.id=14ce240d-5ef3-43be-9cb8-752ef73333cf' '-Dworker.port=6700' '-Dstorm.log.dir=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/logs' '-Dlog4j.configurationFile=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/log4j2/worker.xml' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' 'org.apache.storm.LogWriter' '/opt/java/1.7.0//bin/java' '-server' '-Xmx8192m' '-XX:+PrintGCDetails' '-Xloggc:artifacts/gc.log' '-XX:+PrintGCDateStamps' '-XX:+PrintGCTimeStamps' '-XX:+UseGCLogFileRotation' '-XX:NumberOfGCLogFiles=10' '-XX:GCLogFileSize=1M' '-XX:+HeapDumpOnOutOfMemoryError' '-XX:HeapDumpPath=artifacts/heapdump' '-Djava.library.path=/tmp/storm/supervisor/stormdist/identity-1-1482740925/resources/Linux-amd64:/tmp/storm/supervisor/stormdist/identity-1-1482740925/resources:/usr/local/lib:/opt/local/lib:/usr/lib' '-Dlogfile.name=worker.log' '-Dstorm.home=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1' '-Dworkers.artifacts=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/logs/workers-artifacts' '-Dstorm.conf.file=' '-Dstorm.options=' '-Dstorm.log.dir=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/logs' '-Djava.io.tmpdir=/tmp/storm/workers/14ce240d-5ef3-43be-9cb8-752ef73333cf/tmp' '-Dlogging.sensitivity=S3' '-Dlog4j.configurationFile=/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/log4j2/worker.xml' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' '-Dstorm.id=identity-1-1482740925' '-Dworker.id=14ce240d-5ef3-43be-9cb8-752ef73333cf' '-Dworker.port=6700' '-cp' '/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/reflectasm-1.10.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/clojure-1.7.0.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/minlog-1.3.0.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/storm-rename-hack-1.0.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/objenesis-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/disruptor-3.3.2.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/storm-core-1.0.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/asm-5.0.3.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/listreaming-benchmarks/apache-storm-1.0.1/lib/log4j-over-slf4j-1.6.6.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/servlet-api-2.5.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/kryo-3.0.3.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/slf4j-api-1.7.7.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-api-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/lib/log4j-slf4j-impl-2.1.jar:/home/javed.19/git-pull/finished/streaming-benchmarks/apache-storm-1.0.1/conf:/tmp/storm/supervisor/stormdist/identity-1-1482740925/stormjar.jar' 'org.apache.storm.daemon.worker' 'identity-1-1482740925' '3363bfc3-4083-4a25-af82-0709f43e73cb' '6700' '5fbae825-35ee-45f6-bced-f91a73037e10'
> 2016-12-26 03:30:49.316 o.a.s.config [INFO] SET worker-user 5fbae825-35ee-45f6-bced-f91a73037e10 javed.19
> 2016-12-26 03:30:49.316 o.a.s.d.supervisor [INFO] Creating symlinks for worker-id: 5fbae825-35ee-45f6-bced-f91a73037e10 storm-id: identity-1-1482740925 to its port artifacts directory
> 2016-12-26 03:30:49.325 o.a.s.d.supervisor [INFO] Creating symlinks for worker-id: 5fbae825-35ee-45f6-bced-f91a73037e10 storm-id: identity-1-1482740925 for files(1): ("resources")
> 2016-12-26 03:30:49.329 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:49.829 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:49.849 o.a.s.d.supervisor [INFO] Worker Process 5fbae825-35ee-45f6-bced-f91a73037e10 exited with code: 1
> 2016-12-26 03:30:50.329 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:50.830 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:51.330 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:51.830 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:52.331 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:52.831 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:53.331 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:53.832 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:54.332 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:54.832 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:55.333 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:55.833 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:56.333 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:56.834 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:57.334 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:57.834 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:58.335 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:58.835 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:59.335 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:30:59.836 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:00.336 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:00.837 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:01.337 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:01.837 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:02.338 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:02.838 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:03.338 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:03.839 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:04.339 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:04.839 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:05.340 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:05.840 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:06.340 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:06.841 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:07.341 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:07.841 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:08.342 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:08.842 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:09.342 o.a.s.d.supervisor [INFO] 5fbae825-35ee-45f6-bced-f91a73037e10 still hasn't started
> 2016-12-26 03:31:09.813 o.a.s.u.NimbusClient [WARN] Using deprecated config nimbus.host for backward compatibility. Please update your storm.yaml so it only has config nimbus.seeds
>
> Also I have confirmed that nimbus and supervisors are reachable from each
> other. What else could be causing this issue ?
>
> P.S: You might wonder why I have both nimbus.host and nimbus.seeds property
> in my storm.yaml. If I only use numbs.seeds, Storm UI does not detect any
> nimbus saying : Found nimbuses [] none of which is elected as leader,
> please try again after some t
>