Storm-starter not executing on centos

[ravi kiran puttaswamy <ra...@live.com>]

Hello all,

    I am currently trying to setup a storm local cluster using the master branch (2.0.0 snapshot) from github.

    I am able to start the zookeeper,  nimbus, supervisor and ui services without any errors.

    I wanted to execute WordCountTopology to verify the setup. The topology gets submitted without any errors, but the worker process exit abruptly. The following are the snippets of output from submit command.
<snip>
centos@centos-base ~/s/bin> ./storm jar ../examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar  org.apache.storm.starter.WordCountTopology ww6
Running: java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/home/centos/storm -Dstorm.log.dir=/home/centos/storm/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 -Dstorm.conf.file= -cp /home/centos/storm/*:/home/centos/storm/lib/*:/home/centos/storm/extlib/*:../examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar:/home/centos/storm/conf:/home/centos/storm/bin -Dstorm.jar=../examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.starter.WordCountTopology ww6
21:30:11.814 [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -4869737168860327630:-9077312875146305878
21:30:11.897 [main] WARN  o.a.s.v.ConfigValidation - storm.messaging.netty.max_retries is a deprecated config please see class org.apache.storm.Config.STORM_MESSAGING_NETTY_MAX_RETRIES for more information.
21:30:11.964 [main] INFO  o.a.s.u.NimbusClient - Found leader nimbus : centos-base.novalocal:6627
21:30:11.965 [main] INFO  o.a.s.s.a.AuthUtils - Got AutoCreds []
21:30:12.003 [main] INFO  o.a.s.StormSubmitter - Uploading dependencies - jars...
21:30:12.003 [main] INFO  o.a.s.StormSubmitter - Uploading dependencies - artifacts...
21:30:12.003 [main] INFO  o.a.s.StormSubmitter - Dependency Blob keys - jars : [] / artifacts : []
21:30:12.014 [main] INFO  o.a.s.StormSubmitter - Uploading topology jar ../examples/storm-starter/target/storm-starter-2.0.0-SNAPSHOT.jar to assigned location: /tmp/storm/nimbus/inbox/stormjar-dc6e6f5c-7d5c-4360-8aa3-3a7653296788.jar
21:30:12.410 [main] INFO  o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: /tmp/storm/nimbus/inbox/stormjar-dc6e6f5c-7d5c-4360-8aa3-3a7653296788.jar
21:30:12.411 [main] INFO  o.a.s.StormSubmitter - Submitting topology ww6 in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-4869737168860327630:-9077312875146305878","topology.workers":3,"topology.debug":true}
21:30:13.705 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: ww6
</snip>

This is the snippet from the supervisor.log
<snip>
2018-03-29 21:30:19.261 o.a.s.d.s.BasicContainer SLOT_6701 [INFO] Launching worker with command: 'java' '-cp' '/home/centos/storm/lib-worker/*:/home/centos/storm/extlib/*:/home/centos/storm/conf:/tmp/storm/supervisor/stormdist/ww6-4-1522377012/stormjar.jar' '-Xmx64m' '-Dlogging.sensitivity=S3' '-Dlogfile.name=worker.log' '-Dstorm.home=/home/centos/storm' '-Dworkers.artifacts=/home/centos/storm/logs/workers-artifacts' '-Dstorm.id=ww6-4-1522377012' '-Dworker.id=a8755d1f-400c-4683-9d19-4bba2076e603' '-Dworker.port=6701' '-Dstorm.log.dir=/home/centos/storm/logs' '-Dlog4j.configurationFile=/home/centos/storm/log4j2/worker.xml' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' '-Dstorm.local.dir=/tmp/storm' '-Dworker.memory_limit_mb=1152' 'org.apache.storm.LogWriter' 'java' '-server' '-Dlogging.sensitivity=S3' '-Dlogfile.name=worker.log' '-Dstorm.home=/home/centos/storm' '-Dworkers.artifacts=/home/centos/storm/logs/workers-artifacts' '-Dstorm.id=ww6-4-1522377012' '-Dworker.id=a8755d1f-400c-4683-9d19-4bba2076e603' '-Dworker.port=6701' '-Dstorm.log.dir=/home/centos/storm/logs' '-Dlog4j.configurationFile=/home/centos/storm/log4j2/worker.xml' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' '-Dstorm.local.dir=/tmp/storm' '-Dworker.memory_limit_mb=1152' '-Xmx1152m' '-XX:+PrintGCDetails' '-Xloggc:artifacts/gc.log' '-XX:+PrintGCDateStamps' '-XX:+PrintGCTimeStamps' '-XX:+UseGCLogFileRotation' '-XX:NumberOfGCLogFiles=10' '-XX:GCLogFileSize=1M' '-XX:+HeapDumpOnOutOfMemoryError' '-XX:HeapDumpPath=artifacts/heapdump' '-Djava.library.path=/tmp/storm/supervisor/stormdist/ww6-4-1522377012/resources/Linux-amd64:/tmp/storm/supervisor/stormdist/ww6-4-1522377012/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64' '-Dstorm.conf.file=' '-Dstorm.options=' '-Djava.io.tmpdir=/tmp/storm/workers/a8755d1f-400c-4683-9d19-4bba2076e603/tmp' '-cp' '/home/centos/storm/lib-worker/*:/home/centos/storm/extlib/*:/home/centos/storm/conf:/tmp/storm/supervisor/stormdist/ww6-4-1522377012/stormjar.jar' 'org.apache.storm.daemon.worker.Worker' 'ww6-4-1522377012' '21e05477-6017-4062-bf0f-342b9f2cceca-127.0.0.1' '6701' 'a8755d1f-400c-4683-9d19-4bba2076e603'.
2018-03-29 21:30:19.262 o.a.s.d.s.Slot SLOT_6700 [INFO] STATE WAITING_FOR_BLOB_LOCALIZATION msInState: 12 -> WAITING_FOR_WORKER_START msInState: 0 topo:ww6-4-1522377012 worker:159f603a-f552-43fd-b363-d2e744cc2131
2018-03-29 21:30:19.262 o.a.s.d.s.Slot SLOT_6700 [INFO] SLOT 6700: Changing current assignment from null to LocalAssignment(topology_id:ww6-4-1522377012, executors:[ExecutorInfo(task_start:22, task_end:22), ExecutorInfo(task_start:16, task_end:16), ExecutorInfo(task_start:28, task_end:28), ExecutorInfo(task_start:10, task_end:10), ExecutorInfo(task_start:4, task_end:4), ExecutorInfo(task_start:19, task_end:19), ExecutorInfo(task_start:13, task_end:13), ExecutorInfo(task_start:7, task_end:7), ExecutorInfo(task_start:1, task_end:1), ExecutorInfo(task_start:25, task_end:25)], resources:WorkerResources(mem_on_heap:1280.0, mem_off_heap:0.0, cpu:100.0, shared_mem_on_heap:0.0, shared_mem_off_heap:0.0, resources:{offheap.memory.mb=0.0, onheap.memory.mb=1280.0, cpu.pcore.percent=100.0}, shared_resources:{}), owner:centos)
2018-03-29 21:30:19.281 o.a.s.d.s.Slot SLOT_6701 [INFO] STATE WAITING_FOR_BLOB_LOCALIZATION msInState: 22 -> WAITING_FOR_WORKER_START msInState: 0 topo:ww6-4-1522377012 worker:a8755d1f-400c-4683-9d19-4bba2076e603
2018-03-29 21:30:19.281 o.a.s.d.s.Slot SLOT_6701 [INFO] SLOT 6701: Changing current assignment from null to LocalAssignment(topology_id:ww6-4-1522377012, executors:[ExecutorInfo(task_start:14, task_end:14), ExecutorInfo(task_start:20, task_end:20), ExecutorInfo(task_start:26, task_end:26), ExecutorInfo(task_start:8, task_end:8), ExecutorInfo(task_start:2, task_end:2), ExecutorInfo(task_start:17, task_end:17), ExecutorInfo(task_start:11, task_end:11), ExecutorInfo(task_start:5, task_end:5), ExecutorInfo(task_start:23, task_end:23)], resources:WorkerResources(mem_on_heap:1152.0, mem_off_heap:0.0, cpu:90.0, shared_mem_on_heap:0.0, shared_mem_off_heap:0.0, resources:{offheap.memory.mb=0.0, onheap.memory.mb=1152.0, cpu.pcore.percent=90.0}, shared_resources:{}), owner:centos)
2018-03-29 21:30:19.310 o.a.s.d.s.Slot SLOT_6702 [INFO] STATE WAITING_FOR_BLOB_LOCALIZATION msInState: 58 -> WAITING_FOR_WORKER_START msInState: 0 topo:ww6-4-1522377012 worker:a410536b-56ef-4eb3-83d5-4810e92c2c6c
2018-03-29 21:30:19.310 o.a.s.d.s.Slot SLOT_6702 [INFO] SLOT 6702: Changing current assignment from null to LocalAssignment(topology_id:ww6-4-1522377012, executors:[ExecutorInfo(task_start:18, task_end:18), ExecutorInfo(task_start:24, task_end:24), ExecutorInfo(task_start:12, task_end:12), ExecutorInfo(task_start:6, task_end:6), ExecutorInfo(task_start:15, task_end:15), ExecutorInfo(task_start:3, task_end:3), ExecutorInfo(task_start:9, task_end:9), ExecutorInfo(task_start:21, task_end:21), ExecutorInfo(task_start:27, task_end:27)], resources:WorkerResources(mem_on_heap:1152.0, mem_off_heap:0.0, cpu:90.0, shared_mem_on_heap:0.0, shared_mem_off_heap:0.0, resources:{offheap.memory.mb=0.0, onheap.memory.mb=1152.0, cpu.pcore.percent=90.0}, shared_resources:{}), owner:centos)
2018-03-29 21:30:22.548 o.a.s.d.s.BasicContainer Thread-211 [INFO] Worker Process a410536b-56ef-4eb3-83d5-4810e92c2c6c exited with code: 1
2018-03-29 21:30:22.551 o.a.s.d.s.BasicContainer Thread-209 [INFO] Worker Process 159f603a-f552-43fd-b363-d2e744cc2131 exited with code: 1
2018-03-29 21:30:22.725 o.a.s.d.s.BasicContainer Thread-210 [INFO] Worker Process a8755d1f-400c-4683-9d19-4bba2076e603 exited with code: 1
2018-03-29 21:30:35.795 o.a.s.d.s.t.SupervisorHealthCheck timer [INFO] Running supervisor healthchecks...
2018-03-29 21:30:35.795 o.a.s.h.HealthChecker timer [INFO] The supervisor healthchecks succeeded.
</snip>

I am currently using a CentOS Linux release 7.4.1708 (Core). I installed storm version 1.2 and it works fine. I tried the same configuration (for storm v2.0) on an ubuntu machine, and storm-starter works fine there.

I am not sure what I'm missing. Can you let me know if redhat needs any additional configuration steps, or any steps to help debug my setup further?

Please let me know if you need any additional data/files from me.

thanks for your time,

regards,
ravi