You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Fang Chen <fc...@gmail.com> on 2015/06/02 08:25:24 UTC
Re: Supervisor believes worker has not started.
I ran into this problem and could not find a workaround. In my case, the
worker does not commit suicide, just that the supervisor believes the
worker is dead. Here is the output from one of the supervisor:
2015-06-02T05:56:47.355+0000 b.s.d.supervisor [INFO] Downloading code for
storm id asyncVarGenTopology-2-1433224607 from
/usr/local/storm/storm-local/nimbus/stormdist/asyncVarGenTopology-2-1433224607
2015-06-02T05:56:47.710+0000 b.s.d.supervisor [INFO] Finished downloading
code for storm id asyncVarGenTopology-2-1433224607 from
/usr/local/storm/storm-local/nimbus/stormdist/asyncVarGenTopology-2-1433224607
2015-06-02T05:56:47.713+0000 b.s.d.supervisor [INFO] Launching worker with
assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"asyncVarGenTopology-2-1433224607", :executors ([3 3] [38 38] [8 8] [43 43]
[13 13] [48 48] [18 18] [23 23] [28 28] [33 33])} for this supervisor
5dbc466f-24f0-45ba-8880-3d93771a79c2 on port 6703 with id
fe2e3a1f-51e2-4b94-8a31-79773ac8e641
2015-06-02T05:56:47.716+0000 b.s.d.supervisor [INFO] Launching worker with
command: 'java' '-server' '-Xms12G' '-Xmx12G' '-XX:NewSize=6G'
'-XX:+UseParNewGC' '-XX:+UseConcMarkSweepGC'
'-XX:+CMSParallelRemarkEnabled' '-XX:SurvivorRatio=6'
'-XX:MaxTenuringThreshold=2' '-XX:CMSInitiatingOccupancyFraction=70'
'-XX:+UseCMSInitiatingOccupancyOnly' '-XX:+UseTLAB' '-XX:+UseCondCardMark'
'-XX:CMSWaitDuration=5000' '-XX:+CMSScavengeBeforeRemark'
'-XX:+UnlockDiagnosticVMOptions' '-XX:ParGCCardsPerStrideChunk=4096'
'-XX:+ExplicitGCInvokesConcurrent' '-XX:+PrintGCDetails'
'-XX:+PrintGCDateStamps' '-XX:+PrintTenuringDistribution'
'-XX:PrintFLSStatistics=1' '-XX:+PrintPromotionFailure'
'-XX:+PrintGCApplicationStoppedTime' '-XX:+PrintHeapAtGC'
'-XX:+PrintSafepointStatistics' '-Xloggc:/usr/local/storm/logs/gc.log'
'-XX:+UseGCLogFileRotation' '-XX:NumberOfGCLogFiles=10'
'-XX:GCLogFileSize=100M' '-Dcom.sun.management.jmxremote.port=9999'
'-Dcom.sun.management.jmxremote.ssl=false'
'-Dcom.sun.management.jmxremote.authenticate=false'
'-XX:+CMSClassUnloadingEnabled'
'-Djava.library.path=/usr/local/storm/storm-local/supervisor/stormdist/asyncVarGenTopology-2-1433224607/resources/Linux-amd64:/usr/local/storm/storm-local/supervisor/stormdist/asyncVarGenTopology-2-1433224607/resources:/usr/local/lib:/opt/local/lib:/usr/lib'
'-Dlogfile.name=worker-6703.log'
'-Dstorm.home=/usr/local/apache-storm-0.9.4' '-Dstorm.conf.file='
'-Dstorm.options=' '-Dstorm.log.dir=/usr/local/apache-storm-0.9.4/logs'
'-Dlogback.configurationFile=/usr/local/apache-storm-0.9.4/logback/cluster.xml'
'-Dstorm.id=asyncVarGenTopology-2-1433224607'
'-Dworker.id=fe2e3a1f-51e2-4b94-8a31-79773ac8e641' '-Dworker.port=6703'
'-cp'
'/usr/local/apache-storm-0.9.4/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.4/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.4/lib/logback-classic-1.0.13.jar:/usr/local/apache-storm-0.9.4/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.4/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.4/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.4/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.4/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.4/lib/json-simple-1.1.jar:/usr/local/apache-storm-0.9.4/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.4/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.4/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.4/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.4/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.4/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.4/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.4/lib/logback-core-1.0.13.jar:/usr/local/apache-storm-0.9.4/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.4/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.4/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.4/lib/slf4j-api-1.7.5.jar:/usr/local/apache-storm-0.9.4/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.4/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.4/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.4/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.4/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.4/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.4/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.4/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.4/lib/clj-stacktrace-0.2.2.jar:/usr/local/apache-storm-0.9.4/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.4/lib/storm-core-0.9.4.jar:/usr/local/apache-storm-0.9.4/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.4/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.4/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.4/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.4/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.4/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.4/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.4/conf:/usr/local/storm/storm-local/supervisor/stormdist/asyncVarGenTopology-2-1433224607/stormjar.jar'
'backtype.storm.daemon.worker' 'asyncVarGenTopology-2-1433224607'
'5dbc466f-24f0-45ba-8880-3d93771a79c2' '6703'
'fe2e3a1f-51e2-4b94-8a31-79773ac8e641'
2015-06-02T05:56:47.717+0000 b.s.d.supervisor [INFO]
fe2e3a1f-51e2-4b94-8a31-79773ac8e641 still hasn't started
2015-06-02T05:56:48.217+0000 b.s.d.supervisor [INFO]
fe2e3a1f-51e2-4b94-8a31-79773ac8e641 still hasn't started
2015-06-02T05:56:48.718+0000 b.s.d.supervisor [INFO]
fe2e3a1f-51e2-4b94-8a31-79773ac8e641 still hasn't started
2015-06-02T05:56:49.218+0000 b.s.d.supervisor [INFO]
fe2e3a1f-51e2-4b94-8a31-79773ac8e641 still hasn't started
2015-06-02T05:56:49.719+0000 b.s.d.supervisor [INFO]
fe2e3a1f-51e2-4b94-8a31-79773ac8e641 still hasn't started
2015-06-02T05:56:50.219+0000 b.s.d.supervisor [INFO]
fe2e3a1f-51e2-4b94-8a31-79773ac8e641 still hasn't started
2015-06-02T05:56:50.719+0000 b.s.d.supervisor [INFO]
fe2e3a1f-51e2-4b94-8a31-79773ac8e641 still hasn't started
2015-06-02T06:09:23.423+0000 b.s.d.supervisor [INFO] Removing code for
storm id asyncVarGenTopology-2-1433224607
2015-06-02T06:09:23.425+0000 b.s.d.supervisor [INFO] Shutting down and
clearing state for id fe2e3a1f-51e2-4b94-8a31-79773ac8e641. Current
supervisor time: 1433225363. State: :disallowed, Heartbeat:
#backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1433225130,
:storm-id "asyncVarGenTopology-2-1433224607", :executors #{[3 3] [38 38] [8
8] [43 43] [13 13] [48 48] [18 18] [23 23] [28 28] [-1 -1] [33 33]}, :port
6703}
2015-06-02T06:09:23.425+0000 b.s.d.supervisor [INFO] Shutting down
5dbc466f-24f0-45ba-8880-3d93771a79c2:fe2e3a1f-51e2-4b94-8a31-79773ac8e641
2015-06-02T06:09:24.433+0000 b.s.d.supervisor [INFO] Shut down
5dbc466f-24f0-45ba-8880-3d93771a79c2:fe2e3a1f-51e2-4b94-8a31-79773ac8e641
2015-06-02T06:11:23.425+0000 b.s.d.supervisor [INFO] Downloading code for
storm id asyncVarGenTopology-2-1433224607 from
/usr/local/storm/storm-local/nimbus/stormdist/asyncVarGenTopology-2-1433224607
2015-06-02T06:11:23.665+0000 b.s.d.supervisor [INFO] Finished downloading
code for storm id asyncVarGenTopology-2-1433224607 from
/usr/local/storm/storm-local/nimbus/stormdist/asyncVarGenTopology-2-1433224607
2015-06-02T06:11:23.668+0000 b.s.d.supervisor [INFO] Launching worker with
assignment #backtype.storm.daemon.supervisor.LocalAssignment{:storm-id
"asyncVarGenTopology-2-1433224607", :executors ([3 3] [38 38] [8 8] [43 43]
[13 13] [48 48] [18 18] [23 23] [28 28] [33 33])} for this supervisor
5dbc466f-24f0-45ba-8880-3d93771a79c2 on port 6703 with id
c98c6803-84d4-493b-a5aa-cc2d24f040ac
2015-06-02T06:11:23.670+0000 b.s.d.supervisor [INFO] Launching worker with
command: 'java' '-server' '-Xms12G' '-Xmx12G' '-XX:NewSize=6G'
'-XX:+UseParNewGC' '-XX:+UseConcMarkSweepGC'
'-XX:+CMSParallelRemarkEnabled' '-XX:SurvivorRatio=6'
'-XX:MaxTenuringThreshold=2' '-XX:CMSInitiatingOccupancyFraction=70'
'-XX:+UseCMSInitiatingOccupancyOnly' '-XX:+UseTLAB' '-XX:+UseCondCardMark'
'-XX:CMSWaitDuration=5000' '-XX:+CMSScavengeBeforeRemark'
'-XX:+UnlockDiagnosticVMOptions' '-XX:ParGCCardsPerStrideChunk=4096'
'-XX:+ExplicitGCInvokesConcurrent' '-XX:+PrintGCDetails'
'-XX:+PrintGCDateStamps' '-XX:+PrintTenuringDistribution'
'-XX:PrintFLSStatistics=1' '-XX:+PrintPromotionFailure'
'-XX:+PrintGCApplicationStoppedTime' '-XX:+PrintHeapAtGC'
'-XX:+PrintSafepointStatistics' '-Xloggc:/usr/local/storm/logs/gc.log'
'-XX:+UseGCLogFileRotation' '-XX:NumberOfGCLogFiles=10'
'-XX:GCLogFileSize=100M' '-Dcom.sun.management.jmxremote.port=9999'
'-Dcom.sun.management.jmxremote.ssl=false'
'-Dcom.sun.management.jmxremote.authenticate=false'
'-XX:+CMSClassUnloadingEnabled'
'-Djava.library.path=/usr/local/storm/storm-local/supervisor/stormdist/asyncVarGenTopology-2-1433224607/resources/Linux-amd64:/usr/local/storm/storm-local/supervisor/stormdist/asyncVarGenTopology-2-1433224607/resources:/usr/local/lib:/opt/local/lib:/usr/lib'
'-Dlogfile.name=worker-6703.log'
'-Dstorm.home=/usr/local/apache-storm-0.9.4' '-Dstorm.conf.file='
'-Dstorm.options=' '-Dstorm.log.dir=/usr/local/apache-storm-0.9.4/logs'
'-Dlogback.configurationFile=/usr/local/apache-storm-0.9.4/logback/cluster.xml'
'-Dstorm.id=asyncVarGenTopology-2-1433224607'
'-Dworker.id=c98c6803-84d4-493b-a5aa-cc2d24f040ac' '-Dworker.port=6703'
'-cp'
'/usr/local/apache-storm-0.9.4/lib/clj-time-0.4.1.jar:/usr/local/apache-storm-0.9.4/lib/clojure-1.5.1.jar:/usr/local/apache-storm-0.9.4/lib/logback-classic-1.0.13.jar:/usr/local/apache-storm-0.9.4/lib/ring-servlet-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/objenesis-1.2.jar:/usr/local/apache-storm-0.9.4/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/apache-storm-0.9.4/lib/ring-core-1.1.5.jar:/usr/local/apache-storm-0.9.4/lib/minlog-1.2.jar:/usr/local/apache-storm-0.9.4/lib/commons-lang-2.5.jar:/usr/local/apache-storm-0.9.4/lib/json-simple-1.1.jar:/usr/local/apache-storm-0.9.4/lib/joda-time-2.0.jar:/usr/local/apache-storm-0.9.4/lib/servlet-api-2.5.jar:/usr/local/apache-storm-0.9.4/lib/commons-logging-1.1.3.jar:/usr/local/apache-storm-0.9.4/lib/jetty-6.1.26.jar:/usr/local/apache-storm-0.9.4/lib/hiccup-0.3.6.jar:/usr/local/apache-storm-0.9.4/lib/commons-fileupload-1.2.1.jar:/usr/local/apache-storm-0.9.4/lib/carbonite-1.4.0.jar:/usr/local/apache-storm-0.9.4/lib/logback-core-1.0.13.jar:/usr/local/apache-storm-0.9.4/lib/compojure-1.1.3.jar:/usr/local/apache-storm-0.9.4/lib/tools.logging-0.2.3.jar:/usr/local/apache-storm-0.9.4/lib/chill-java-0.3.5.jar:/usr/local/apache-storm-0.9.4/lib/slf4j-api-1.7.5.jar:/usr/local/apache-storm-0.9.4/lib/reflectasm-1.07-shaded.jar:/usr/local/apache-storm-0.9.4/lib/snakeyaml-1.11.jar:/usr/local/apache-storm-0.9.4/lib/commons-exec-1.1.jar:/usr/local/apache-storm-0.9.4/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/math.numeric-tower-0.0.1.jar:/usr/local/apache-storm-0.9.4/lib/core.incubator-0.1.0.jar:/usr/local/apache-storm-0.9.4/lib/tools.macro-0.1.0.jar:/usr/local/apache-storm-0.9.4/lib/commons-codec-1.6.jar:/usr/local/apache-storm-0.9.4/lib/ring-devel-0.3.11.jar:/usr/local/apache-storm-0.9.4/lib/asm-4.0.jar:/usr/local/apache-storm-0.9.4/lib/clj-stacktrace-0.2.2.jar:/usr/local/apache-storm-0.9.4/lib/jetty-util-6.1.26.jar:/usr/local/apache-storm-0.9.4/lib/storm-core-0.9.4.jar:/usr/local/apache-storm-0.9.4/lib/clout-1.0.1.jar:/usr/local/apache-storm-0.9.4/lib/kryo-2.21.jar:/usr/local/apache-storm-0.9.4/lib/jgrapht-core-0.9.0.jar:/usr/local/apache-storm-0.9.4/lib/jline-2.11.jar:/usr/local/apache-storm-0.9.4/lib/commons-io-2.4.jar:/usr/local/apache-storm-0.9.4/lib/disruptor-2.10.1.jar:/usr/local/apache-storm-0.9.4/lib/tools.cli-0.2.4.jar:/usr/local/apache-storm-0.9.4/conf:/usr/local/storm/storm-local/supervisor/stormdist/asyncVarGenTopology-2-1433224607/stormjar.jar'
'backtype.storm.daemon.worker' 'asyncVarGenTopology-2-1433224607'
'5dbc466f-24f0-45ba-8880-3d93771a79c2' '6703'
'c98c6803-84d4-493b-a5aa-cc2d24f040ac'
2015-06-02T06:11:23.671+0000 b.s.d.supervisor [INFO]
c98c6803-84d4-493b-a5aa-cc2d24f040ac still hasn't started
2015-06-02T06:11:24.172+0000 b.s.d.supervisor [INFO]
c98c6803-84d4-493b-a5aa-cc2d24f040ac still hasn't started
2015-06-02T06:11:24.672+0000 b.s.d.supervisor [INFO]
c98c6803-84d4-493b-a5aa-cc2d24f040ac still hasn't started
2015-06-02T06:11:25.172+0000 b.s.d.supervisor [INFO]
c98c6803-84d4-493b-a5aa-cc2d24f040ac still hasn't started
2015-06-02T06:11:25.673+0000 b.s.d.supervisor [INFO]
c98c6803-84d4-493b-a5aa-cc2d24f040ac still hasn't started
2015-06-02T06:11:26.173+0000 b.s.d.supervisor [INFO]
c98c6803-84d4-493b-a5aa-cc2d24f040ac still hasn't started
2015-06-02T06:11:26.673+0000 b.s.d.supervisor [INFO]
c98c6803-84d4-493b-a5aa-cc2d24f040ac still hasn't started
On Fri, May 29, 2015 at 3:09 PM, Jeffery Maass <ma...@gmail.com> wrote:
> When you look at the worker logs, do some of the workers sometimes kill
> themselves because there is a missing stormconf.ser file? If so, grab that
> error message and have fun googling.
>
> Some will say those problems went away with the latest release.
> Apparently, it is complicated.
>
> My best advice is to avoid resource contention issues ( failed heartbeats
> ) or unhandled exceptions ( worker commits suicide ) that result in workers
> dying or being killed. This may lead to a state that I describe as the
> fubar worker loop. I haven't figured it out, but basically, a worker gets
> up, the supervisor is looking for the wrong worker, the supervisor decides
> the worker is dead, so cleans up, the clean up means the worker no longer
> sees its stormconf.ser, so it commits suicide. That's one version of
> events.
>
> Good luck.
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
> On Fri, May 29, 2015 at 12:36 PM, Grant Overby (groverby) <
> groverby@cisco.com> wrote:
>
>> Supervisor is reporting that the worker “still hasn’t started” and
>> eventually kills and restarts the worker. However; the worker has started
>> and is processing tuples. This repeats indefinitely.
>>
>> Debugging steps?
>>
>>
>> *Grant Overby*
>> Software Engineer
>> Cisco.com <http://www.cisco.com/>
>> groverby@cisco.com
>> Mobile: *865 724 4910 <865%20724%204910>*
>>
>>
>>
>> Think before you print.
>>
>> This email may contain confidential and privileged material for the sole
>> use of the intended recipient. Any review, use, distribution or disclosure
>> by others is strictly prohibited. If you are not the intended recipient (or
>> authorized to receive for the recipient), please contact the sender by
>> reply email and delete all copies of this message.
>>
>> Please click here
>> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
>> Company Registration Information.
>>
>>
>>
>>
>
Re: Supervisor believes worker has not started.
Posted by Fang Chen <fc...@gmail.com>.
Indeed they kill was commanded by nimbus. I see messages like the following
before supervisor kills worker. Now the question is why nimbus believe
worker executors are not alive?
Fang
=====
2015-06-04T00:35:52.229+0000 b.s.d.nimbus [INFO] Executor
asyncVarGenTopology-2-1433377984:[29 29] not alive
2015-06-04T00:35:52.229+0000 b.s.d.nimbus [INFO] Executor
asyncVarGenTopology-2-1433377984:[30 30] not alive
2015-06-04T00:35:52.229+0000 b.s.d.nimbus [INFO] Executor
asyncVarGenTopology-2-1433377984:[31 31] not alive
2015-06-04T00:35:52.229+0000 b.s.d.nimbus [INFO] Executor
asyncVarGenTopology-2-1433377984:[32 32] not alive
2015-06-04T00:35:52.229+0000 b.s.d.nimbus [INFO] Executor
asyncVarGenTopology-2-1433377984:[1 1] not alive
2015-06-04T00:35:52.229+0000 b.s.d.nimbus [INFO] Executor
asyncVarGenTopology-2-1433377984:[33 33] not alive
2015-06-04T00:35:52.242+0000 b.s.s.EvenScheduler [INFO] Available slots:
(["1fee065f-699e-427d-b626-7ae51132db12" 6702]
["1fee065f-699e-427d-b626-7ae51132db12" 6701]
["1fee065f-699e-427d-b626-7ae51132db12" 6700]
["029d3393-8bd8-4ea7-9f79-3b23f46aa289" 6702]
["029d3393-8bd8-4ea7-9f79-3b23f46aa289" 6701]
["029d3393-8bd8-4ea7-9f79-3b23f46aa289" 6700]
["87d03069-baec-4c9c-ba92-fc548cd07b7d" 6702]
["87d03069-baec-4c9c-ba92-fc548cd07b7d" 6701]
["87d03069-baec-4c9c-ba92-fc548cd07b7d" 6700]
["06d1b8d8-eb1c-473f-93cb-444d3c69b24a" 6702]
["06d1b8d8-eb1c-473f-93cb-444d3c69b24a" 6701]
["06d1b8d8-eb1c-473f-93cb-444d3c69b24a" 6700]
["8f2db495-c860-4ff2-99f9-11c74b6cf258" 6702]
["8f2db495-c860-4ff2-99f9-11c74b6cf258" 6701]
["8f2db495-c860-4ff2-99f9-11c74b6cf258" 6700])
2015-06-04T00:35:52.242+0000 b.s.d.nimbus [INFO] Reassigning
asyncVarGenTopology-2-1433377984 to 5 slots
2015-06-04T00:35:52.243+0000 b.s.d.nimbus [INFO] Reassign executors: [[2 2]
[34 34] [3 3] [35 35] [4 4] [36 36] [5 5] [37 37] [6 6] [38 38] [7 7] [39
39] [8 8] [40 40] [9 9] [41 41] [10 10] [42 42] [11 11] [43 43] [12 12] [44
44] [13 13] [45 45] [14 14] [46 46] [15 15] [47 47] [16 16] [48 48] [17 17]
[49 49] [18 18] [50 50] [19 19] [51 51] [20 20] [21 21] [22 22] [23 23] [24
24] [25 25] [26 26] [27 27] [28 28] [29 29] [30 30] [31 31] [32 32] [1 1]
[33 33]]
On Tue, Jun 2, 2015 at 3:08 PM, Jeffery Maass <ma...@gmail.com> wrote:
> disallowed = the request to shut down came from nimbus.
>
> It takes a village. Also, it takes all of the logs.....
>
> Get them together to see a better picture:
> Nimbus, Supervisor, Worker
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <ma...@gmail.com>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
> On Tue, Jun 2, 2015 at 10:29 AM, Grant Overby (groverby) <
> groverby@cisco.com> wrote:
>
>> 0.9.3.2.2 (hdp 2.2.4.2-2)
>>
>> *Grant Overby*
>> Software Engineer
>> Cisco.com <http://www.cisco.com/>
>> groverby@cisco.com
>> Mobile: *865 724 4910 <865%20724%204910>*
>>
>>
>>
>> Think before you print.
>>
>> This email may contain confidential and privileged material for the sole
>> use of the intended recipient. Any review, use, distribution or disclosure
>> by others is strictly prohibited. If you are not the intended recipient (or
>> authorized to receive for the recipient), please contact the sender by
>> reply email and delete all copies of this message.
>>
>> Please click here
>> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
>> Company Registration Information.
>>
>>
>>
>>
>> From: Harsha <st...@harsha.io>
>> Date: Tuesday, June 2, 2015 at 11:24 AM
>> To: "Grant Overby (groverby)" <gr...@cisco.com>, "
>> user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Supervisor believes worker has not started.
>>
>> Which version of storm are you using?
>>
>> --
>> Harsha
>>
>>
>> On June 2, 2015 at 8:04:48 AM, Grant Overby (groverby) (
>> groverby@cisco.com) wrote:
>>
>> Same here. The worker isn’t committing suicide; its being murdered by
>> the supervisor.
>>
>> *Grant Overby*
>> Software Engineer
>> Cisco.com <http://www.cisco.com/>
>> groverby@cisco.com
>> Mobile: *865 724 4910 <865%20724%204910>*
>>
>>
>>
>> Think before you print.
>>
>> This email may contain confidential and privileged material for the sole
>> use of the intended recipient. Any review, use, distribution or disclosure
>> by others is strictly prohibited. If you are not the intended recipient (or
>> authorized to receive for the recipient), please contact the sender by
>> reply email and delete all copies of this message.
>>
>> Please click here
>> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
>> Company Registration Information.
>>
>>
>>
>> From: Fang Chen <fc...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: Tuesday, June 2, 2015 at 2:25 AM
>> To: "user@storm.apache.org" <us...@storm.apache.org>
>> Subject: Re: Supervisor believes worker has not started.
>>
>> ld not find a workaround. In my case, the w
>>
>>
>
Re: Supervisor believes worker has not started.
Posted by Jeffery Maass <ma...@gmail.com>.
disallowed = the request to shut down came from nimbus.
It takes a village. Also, it takes all of the logs.....
Get them together to see a better picture:
Nimbus, Supervisor, Worker
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <ma...@gmail.com>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Tue, Jun 2, 2015 at 10:29 AM, Grant Overby (groverby) <groverby@cisco.com
> wrote:
> 0.9.3.2.2 (hdp 2.2.4.2-2)
>
> *Grant Overby*
> Software Engineer
> Cisco.com <http://www.cisco.com/>
> groverby@cisco.com
> Mobile: *865 724 4910 <865%20724%204910>*
>
>
>
> Think before you print.
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> Please click here
> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
> Company Registration Information.
>
>
>
>
> From: Harsha <st...@harsha.io>
> Date: Tuesday, June 2, 2015 at 11:24 AM
> To: "Grant Overby (groverby)" <gr...@cisco.com>, "user@storm.apache.org"
> <us...@storm.apache.org>
> Subject: Re: Supervisor believes worker has not started.
>
> Which version of storm are you using?
>
> --
> Harsha
>
>
> On June 2, 2015 at 8:04:48 AM, Grant Overby (groverby) (groverby@cisco.com)
> wrote:
>
> Same here. The worker isn’t committing suicide; its being murdered by
> the supervisor.
>
> *Grant Overby*
> Software Engineer
> Cisco.com <http://www.cisco.com/>
> groverby@cisco.com
> Mobile: *865 724 4910 <865%20724%204910>*
>
>
>
> Think before you print.
>
> This email may contain confidential and privileged material for the sole
> use of the intended recipient. Any review, use, distribution or disclosure
> by others is strictly prohibited. If you are not the intended recipient (or
> authorized to receive for the recipient), please contact the sender by
> reply email and delete all copies of this message.
>
> Please click here
> <http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for
> Company Registration Information.
>
>
>
> From: Fang Chen <fc...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: Tuesday, June 2, 2015 at 2:25 AM
> To: "user@storm.apache.org" <us...@storm.apache.org>
> Subject: Re: Supervisor believes worker has not started.
>
> ld not find a workaround. In my case, the w
>
>
Re: Supervisor believes worker has not started.
Posted by "Grant Overby (groverby)" <gr...@cisco.com>.
0.9.3.2.2 (hdp 2.2.4.2-2)
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: Harsha <st...@harsha.io>>
Date: Tuesday, June 2, 2015 at 11:24 AM
To: "Grant Overby (groverby)" <gr...@cisco.com>>, "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Supervisor believes worker has not started.
Which version of storm are you using?
--
Harsha
On June 2, 2015 at 8:04:48 AM, Grant Overby (groverby) (groverby@cisco.com<ma...@cisco.com>) wrote:
Same here. The worker isn’t committing suicide; its being murdered by the supervisor.
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: Fang Chen <fc...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, June 2, 2015 at 2:25 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Supervisor believes worker has not started.
ld not find a workaround. In my case, the w
Re: Supervisor believes worker has not started.
Posted by Harsha <st...@harsha.io>.
Which version of storm are you using?
--
Harsha
On June 2, 2015 at 8:04:48 AM, Grant Overby (groverby) (groverby@cisco.com) wrote:
Same here. The worker isn’t committing suicide; its being murdered by the supervisor.
Grant Overby
Software Engineer
Cisco.com
groverby@cisco.com
Mobile: 865 724 4910
Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here for Company Registration Information.
From: Fang Chen <fc...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: Tuesday, June 2, 2015 at 2:25 AM
To: "user@storm.apache.org" <us...@storm.apache.org>
Subject: Re: Supervisor believes worker has not started.
ld not find a workaround. In my case, the w
Re: Supervisor believes worker has not started.
Posted by "Grant Overby (groverby)" <gr...@cisco.com>.
Same here. The worker isn’t committing suicide; its being murdered by the supervisor.
[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]
Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<ma...@cisco.com>
Mobile: 865 724 4910
[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.
This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message.
Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information.
From: Fang Chen <fc...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: Tuesday, June 2, 2015 at 2:25 AM
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Subject: Re: Supervisor believes worker has not started.
ld not find a workaround. In my case, the w