You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "P. Taylor Goetz" <pt...@gmail.com> on 2014/02/14 03:34:46 UTC
Re: Storm 0.9.0.1 - topology halts processing after a while
Yeah, we had to disable issues on nathanmarz/storm in order to get everyone to start using the Apache infrastructure. An unfortunate side effect of that was that now links to old github issues return a 404.
All the github issues before the move have been migrated to JIRA now (https://issues.apache.org/jira/browse/STORM). You may have to do some searching to find them, but James Xu did a lot of work to ensure that most of the comments got transferred to JIRA.
I’ll consolidate them under a single JIRA when I get the chance, but the relevant issues are:
STORM-202, STORM-105, STORM-29, STORM-143, and STORM-95
Or use this query: https://issues.apache.org/jira/browse/STORM-202?jql=project%20%3D%20STORM%20AND%20text%20~%20%22stdout%22
It would be great if we could come up with a proper solution. For those who have been bitten by this it’s painful — zero indication of what the issue is.
At the very least, it should be well documented. Heisenbugs suck. ;)
- Taylor
On Feb 13, 2014, at 11:10 AM, Marc Vaillant <va...@animetrics.com> wrote:
> I was going to mention something similar. The stdout/stderr issue is
> insidious. Only happens in cluster mode and while it was fixed for some
> stdout/stderr pathways, others have not been fixed AFAIK. Taylor has
> mentioned one, another is stdout/stderr coming from native code via JNI
> (i.e. printf, std::cout, std::cerr from C and C++). Unfortunately, I
> seem to be unable to access the original issue thread from github that I
> tried to reopen about 9 months ago.
>
> Marc
>
> On Wed, Feb 12, 2014 at 03:50:52PM -0500, P. Taylor Goetz wrote:
>> It looks like you have GC logging turned on, but you haven t specified an
>> output file:
>>
>> "worker.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
>> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Dcom.sun.management.jmxremote.port=1%ID% -Xmx1200m
>>
>> I ve run into this before:
>>
>> "The root cause was that GC logging for workers had been turned on without
>> specifying an output file. As a result the GC logging went to standard output
>> without being redirected to logback. Eventually the buffer filled and the JVM
>> would hang (and thus stop heartbeating) and be killed. The amount of time the
>> worker would last depended on memory pressure and allocated heap size
>> (obviously, in hindsight the more GC, the more GC logging, the faster the
>> buffer fills).
>>
>> The symptoms were workers timing out and being killed for no apparent reason.
>>
>> I would try with GC logging turned off or redirected to a file.
>>
>> - Taylor
>>
>> On Feb 12, 2014, at 3:12 PM, Mark Hu <ma...@gmail.com> wrote:
>>
>>
>> Hi guys,
>>
>>
>> Just an update.
>>
>> 1. After investigating the hang, it looked like the Spout was properly
>> sending tuples to a bolt, but the bolt would never acknowledge the incoming
>> tuple. After switching from using the Netty transport layer back to 0MQ,
>> recovery and reliability guarantees were restored. Please note that we're
>> using jruby-1.7.3 on top of storm java - could this cause issues with
>> netty?
>> 2. Can anyone confirm if the above worker restart behaviour is normal?
>>
>> Thanks,
>> Mark
>>
>>
>>
>> On Feb 6, 2014, at 10:20 PM, Mark Hu <ma...@gmail.com> wrote:
>>
>>
>> HISTORY:
>> I have past experience with Storm 0.8.2 and encountered silent worker death
>> during development. After experimenting with various parallelism, worker,
>> and max_spout configurations, we were able to find one that worked well
>> with high reliability despite this issue.
>>
>> The silent worker death basically involves a worker not responding and
>> waiting 30 seconds before starting to process work again. A full load test
>> reveals no data loss here.
>>
>> I have this topology running well for the past 3 months and the only errors
>> were due to the user.
>>
>> PRESENT:
>> I am now testing Storm 0.9.0.1 and it appears the silent worker death issue
>> is still present, but now the topology hangs. Here are the details:
>>
>> Type and description for my test topology
>> 1. One spout - makes web calls to a rest service to retrieve ids.
>> 2. One bolt - matchRetrieveBolt - retrieves blobs for those ids.
>> 3. 50 bolts - matchProcessBolt - does some processing on the blobs.
>>
>> Storm performed swimmingly for about 63,000 blobs before it hung. After
>> examining the logs, it appears that one worker was processing both bolt
>> type 2 and bolt type 3. For some reason, a reset occurs and the bolts
>> reinitialize but become unresponsive. The spout continues to send work but
>> times out and retries.
>>
>> Here is the data for worker-6710.log:
>>
>>
>> 2014-02-07 01:31:26 MatchProcessBolt [INFO] [BOLT-MatchProcessBolt] bitfield already allocated, ttl: -1
>> 2014-02-07 01:31:26 MatchProcessBolt [INFO] [BOLT-MatchProcessBolt] All done. Acking now
>> 2014-02-07 01:31:26 MatchRetrieveBolt [INFO] [BOLT-1] process: first = 66300, last = 66350
>> 2014-02-07 01:31:26 MatchRetrieveBolt [INFO] [BOLT-1] url: SOMEURL
>>
>>
>> As per the above, this worker is performing bolt1 and bolt2 work.
>>
>> Suddenly, a restart occurs as per the below:
>>
>>
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:host.name=dev-shared2-supervisor-i-e67015c7
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.version=1.6.0_27
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Sun Microsystems Inc.
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/opt/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/opt/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/opt/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/opt/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/opt/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/opt/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/opt/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/opt/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/opt/storm/storm-0.9.0.1/lib/guava-13.0.jar:/opt/storm/storm-0.9.0.1/lib/commo> ns-fileupload-1.2.1.jar:/opt/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/opt/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/opt/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/opt/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/asm-4.0.jar:/opt/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/opt/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/opt/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/opt/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/opt/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/opt/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/opt/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/o> pt/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/opt/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/opt/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/opt/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/opt/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/opt/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/opt/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/opt/storm/storm-0.9.0.1/lib/json-simple-1.1.jar:/opt/storm/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/opt/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/opt/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:/opt/storm/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/opt/storm/storm-0.9.0.1/conf:/mnt/storm/supervisor/stormdist/match_parser-12-1391735866/stormjar.jar
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:os.version=3.2.0-48-virtual
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:user.name=storm
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
>> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:user.dir=/opt/storm/storm-0.9.0.1
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:host.name=dev-shared2-supervisor-i-e67015c7
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.6.0_27
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Sun Microsystems Inc.
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/opt/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/opt/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/opt/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/opt/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/opt/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/opt/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/opt/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/opt/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/opt/storm/storm-0.9.0.1/lib/guava-13.0.jar:/opt/storm/storm-0.9.0.1/l> ib/commons-fileupload-1.2.1.jar:/opt/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/opt/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/opt/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/opt/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/asm-4.0.jar:/opt/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/opt/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/opt/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/opt/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/opt/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/opt/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/opt/storm/storm-0.9.0.1/lib/ring-devel-0.3.1> 1.jar:/opt/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/opt/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/opt/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/opt/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/opt/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/opt/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/opt/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/opt/storm/storm-0.9.0.1/lib/json-simple-1.1.jar:/opt/storm/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/opt/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/opt/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:/opt/storm/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/opt/storm/storm-0.9.0.1/conf:/mnt/storm/supervisor/stormdist/match_parser-12-1391735866/stormjar.jar
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.2.0-48-virtual
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
>> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/opt/storm/storm-0.9.0.1
>> 2014-02-07 01:32:01 b.s.d.worker [INFO] Launching worker for match_parser-12-1391735866 on f1b66213-7237-483e-a5db-e543bb36bcd9:6710 with id 80483ed6-2c51-4f86-9d06-9d2bed705748 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/mnt/storm", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nim> bus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "10.0.5.50", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["10.0.6.131" "10.0.6.130" "10.0.6.55"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1%ID% -Xmx1200m", "sup> ervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 100, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703 6704 6705 6706 6707 6708 6709 6710 6711 6712 6713 6714 6715 6716 6717 6718 6719 6720 6721], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.> sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8091", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs> " 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8090", "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil}
>> 2014-02-07 01:32:01 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>> 2014-02-07 01:32:01 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=10.0.6.131:2181,10.0.6.130:2181,10.0.6.55:2181 sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@25de152f
>> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Opening socket connection to server /10.0.6.131:2181
>> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Socket connection established to ip-10-0-6-131.ec2.internal/10.0.6.131:2181, initiating session
>> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Session establishment complete on server ip-10-0-6-131.ec2.internal/10.0.6.131:2181, sessionid = 0xf343fa0ee4d600f0, negotiated timeout = 20000
>> 2014-02-07 01:32:01 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
>> 2014-02-07 01:32:01 o.a.z.ZooKeeper [INFO] Session: 0xf343fa0ee4d600f0 closed
>> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] EventThread shut down
>> 2014-02-07 01:32:01 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
>> 2014-02-07 01:32:01 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=10.0.6.131:2181,10.0.6.130:2181,10.0.6.55:2181/storm sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@31930611
>> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Opening socket connection to server /10.0.6.131:2181
>> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Socket connection established to ip-10-0-6-131.ec2.internal/10.0.6.131:2181, initiating session
>> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Session establishment complete on server ip-10-0-6-131.ec2.internal/10.0.6.131:2181, sessionid = 0xf343fa0ee4d600f1, negotiated timeout = 20000
>> 2014-02-07 01:32:01 b.s.m.TransportFactory [INFO] Storm peer transport plugin:backtype.storm.messaging.netty.Context
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loading executor 2:[2 2]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loaded executor tasks 2:[2 2]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Finished loading executor 2:[2 2]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loading executor __acker:[102 102]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Preparing bolt 2:(2)
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loaded executor tasks __acker:[102 102]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Timeouts disabled for executor __acker:[102 102]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Preparing bolt __acker:(102)
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Finished loading executor __acker:[102 102]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Prepared bolt __acker:(102)
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loading executor 3:[52 52]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loaded executor tasks 3:[52 52]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Finished loading executor 3:[52 52]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Preparing bolt 3:(52)
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loading executor __system:[-1 -1]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loaded executor tasks __system:[-1 -1]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Finished loading executor __system:[-1 -1]
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Preparing bolt __system:(-1)
>> 2014-02-07 01:32:01 b.s.d.worker [INFO] Launching receive-thread for f1b66213-7237-483e-a5db-e543bb36bcd9:6710
>> 2014-02-07 01:32:01 b.s.d.executor [INFO] Prepared bolt __system:(-1)
>> 2014-02-07 01:32:02 b.s.d.worker [INFO] Worker has topology config {"storm.id" "match_parser-12-1391735866", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "executor.send.buffer.size" 16384, "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/mnt/storm", "iwparser_socket" "/tmp/iwparser2Sockets/iwparserClients.ipc", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.t> imeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "batch_size" 50, "nimbus.inbox.jar.expiration.secs" 3600, "aws_creds.aws_secret_access_key" "wmGLQ4ItvdhniRFMJe3yDJvBwTJUTqbuLYAS5ZPU", "drpc.worker.threads" 64, "redis.processed.match.log.size" 100000, "redis.key.match.ok" "MATCH-OK", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "10.0.5.50", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "aws_creds.aws_access_key_id" "AKIAJP4O6MTXON3MLV2Q", "executor.receive.buffer.size" 16384, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "aws.s3.match_bucket_name" "prod-matches", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["10.0.6.131" "10.0.6.130" "10.0.6.55"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" 50, "topology.kryo.decorators" > (), "topology.name" "match_parser", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" "-Djruby.compat.version=RUBY1_9", "drpc.queue.size" 128, "worker.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1%ID% -Xmx1200m", "supervisor.heartbeat.frequency.secs" 5, "transfer.buffer.size" 32, "topology.error.throttle.interval.secs" 10, "throttle.spout" 1, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 100, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "receiver.buffer.size" 8, "debug.logging.redis" true, "topology.max.spout.pending" 1, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1> , "platform" "xbox", "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703 6704 6705 6706 6707 6708 6709 6710 6711 6712 6713 6714 6715 6716 6717 6718 6719 6720 6721], "iwparser_port" 12330, "redis.port" 6379, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "debug.logging.stdout" true, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "dw_retries.max_attempts" 6, "topology.workers" 50, "supervisor.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8091", "nimbus.thrift.port" 6627, "redis_prefix" "prod", "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.w> ait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "http_proxy" "proxy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "debug.logging.kibana" false, "iwparser_host" "10.200.6.145", "drpc.request.timeout.secs" 600, "demonware.endpoint" "http://10.0.1.78:9000/matches/", "redis_simple_prefix" "prod", "redis.host" "storm090-redis.uvvujo.0001.use1.cache.amazonaws.com", "dw_retries.wait_interval" 2, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts> " "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8090", "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil, "statsd.endpoint" "statsd", "redis.log.size" 100000}
>> 2014-02-07 01:32:02 b.s.d.worker [INFO] Worker 80483ed6-2c51-4f86-9d06-9d2bed705748 for storm match_parser-12-1391735866 on f1b66213-7237-483e-a5db-e543bb36bcd9:6710 has finished loading
>> 2014-02-07 01:32:08 MatchRetrieveBolt [INFO] [BOLT-1] on_init
>> 2014-02-07 01:32:08 MatchProcessBolt [INFO] [BOlT-MatchProcessBolt] on_init
>> 2014-02-07 01:32:08 b.s.d.executor [INFO] Prepared bolt 2:(2)
>> 2014-02-07 01:32:08 b.s.d.executor [INFO] Prepared bolt 3:(52)
>>
>>
>> The bolts have reinitialized. The Storm UI is showing that bolt1 is still being handled by this specific worker.
>>
>>
>>
>> The spout still sends tuples to matchRetrieveBolt but never gets a response. The failure is caught and retries continue.
>>
>>
>> Can anyone comment or provide any thoughts on the above issue?
>>
>> Thanks,
>> Mark
>>
>>
>
>
Re: Storm 0.9.0.1 - topology halts processing after a while
Posted by Marc Vaillant <va...@animetrics.com>.
On Thu, Feb 13, 2014 at 09:34:46PM -0500, P. Taylor Goetz wrote:
> Yeah, we had to disable issues on nathanmarz/storm in order to get
> everyone to start using the Apache infrastructure. An unfortunate side
> effect of that was that now links to old github issues return a 404.
>
> All the github issues before the move have been migrated to JIRA now
> (https://issues.apache.org/jira/browse/STORM). You may have to do some
> searching to find them, but James Xu did a lot of work to ensure that
> most of the comments got transferred to JIRA.
Including issues that were closed before the move? The specific one I
am looking for was
https://github.com/nathanmarz/storm/issues/31
titled "Redirect stdout/stderr of workers to a file" which I can't find
at JIRA.
>
> I’ll consolidate them under a single JIRA when I get the chance, but the relevant issues are:
>
> STORM-202, STORM-105, STORM-29, STORM-143, and STORM-95
>
> Or use this query: https://issues.apache.org/jira/browse/STORM-202?jql=project%20%3D%20STORM%20AND%20text%20~%20%22stdout%22
>
> It would be great if we could come up with a proper solution. For those who have been bitten by this it’s painful — zero indication of what the issue is.
I did find the following for the JNI case which I have not tried yet:
http://tabbott.com/2010/09/capturing-native-code-output/
Marc
>
> At the very least, it should be well documented. Heisenbugs suck. ;)
>
> - Taylor
>
> On Feb 13, 2014, at 11:10 AM, Marc Vaillant <va...@animetrics.com> wrote:
>
> > I was going to mention something similar. The stdout/stderr issue is
> > insidious. Only happens in cluster mode and while it was fixed for some
> > stdout/stderr pathways, others have not been fixed AFAIK. Taylor has
> > mentioned one, another is stdout/stderr coming from native code via JNI
> > (i.e. printf, std::cout, std::cerr from C and C++). Unfortunately, I
> > seem to be unable to access the original issue thread from github that I
> > tried to reopen about 9 months ago.
> >
> > Marc
> >
> > On Wed, Feb 12, 2014 at 03:50:52PM -0500, P. Taylor Goetz wrote:
> >> It looks like you have GC logging turned on, but you haven t specified an
> >> output file:
> >>
> >> "worker.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
> >> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false
> >> -Dcom.sun.management.jmxremote.authenticate=false
> >> -Dcom.sun.management.jmxremote.port=1%ID% -Xmx1200m
> >>
> >> I ve run into this before:
> >>
> >> "The root cause was that GC logging for workers had been turned on without
> >> specifying an output file. As a result the GC logging went to standard output
> >> without being redirected to logback. Eventually the buffer filled and the JVM
> >> would hang (and thus stop heartbeating) and be killed. The amount of time the
> >> worker would last depended on memory pressure and allocated heap size
> >> (obviously, in hindsight the more GC, the more GC logging, the faster the
> >> buffer fills).
> >>
> >> The symptoms were workers timing out and being killed for no apparent reason.
> >>
> >> I would try with GC logging turned off or redirected to a file.
> >>
> >> - Taylor
> >>
> >> On Feb 12, 2014, at 3:12 PM, Mark Hu <ma...@gmail.com> wrote:
> >>
> >>
> >> Hi guys,
> >>
> >>
> >> Just an update.
> >>
> >> 1. After investigating the hang, it looked like the Spout was properly
> >> sending tuples to a bolt, but the bolt would never acknowledge the incoming
> >> tuple. After switching from using the Netty transport layer back to 0MQ,
> >> recovery and reliability guarantees were restored. Please note that we're
> >> using jruby-1.7.3 on top of storm java - could this cause issues with
> >> netty?
> >> 2. Can anyone confirm if the above worker restart behaviour is normal?
> >>
> >> Thanks,
> >> Mark
> >>
> >>
> >>
> >> On Feb 6, 2014, at 10:20 PM, Mark Hu <ma...@gmail.com> wrote:
> >>
> >>
> >> HISTORY:
> >> I have past experience with Storm 0.8.2 and encountered silent worker death
> >> during development. After experimenting with various parallelism, worker,
> >> and max_spout configurations, we were able to find one that worked well
> >> with high reliability despite this issue.
> >>
> >> The silent worker death basically involves a worker not responding and
> >> waiting 30 seconds before starting to process work again. A full load test
> >> reveals no data loss here.
> >>
> >> I have this topology running well for the past 3 months and the only errors
> >> were due to the user.
> >>
> >> PRESENT:
> >> I am now testing Storm 0.9.0.1 and it appears the silent worker death issue
> >> is still present, but now the topology hangs. Here are the details:
> >>
> >> Type and description for my test topology
> >> 1. One spout - makes web calls to a rest service to retrieve ids.
> >> 2. One bolt - matchRetrieveBolt - retrieves blobs for those ids.
> >> 3. 50 bolts - matchProcessBolt - does some processing on the blobs.
> >>
> >> Storm performed swimmingly for about 63,000 blobs before it hung. After
> >> examining the logs, it appears that one worker was processing both bolt
> >> type 2 and bolt type 3. For some reason, a reset occurs and the bolts
> >> reinitialize but become unresponsive. The spout continues to send work but
> >> times out and retries.
> >>
> >> Here is the data for worker-6710.log:
> >>
> >>
> >> 2014-02-07 01:31:26 MatchProcessBolt [INFO] [BOLT-MatchProcessBolt] bitfield already allocated, ttl: -1
> >> 2014-02-07 01:31:26 MatchProcessBolt [INFO] [BOLT-MatchProcessBolt] All done. Acking now
> >> 2014-02-07 01:31:26 MatchRetrieveBolt [INFO] [BOLT-1] process: first = 66300, last = 66350
> >> 2014-02-07 01:31:26 MatchRetrieveBolt [INFO] [BOLT-1] url: SOMEURL
> >>
> >>
> >> As per the above, this worker is performing bolt1 and bolt2 work.
> >>
> >> Suddenly, a restart occurs as per the below:
> >>
> >>
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:host.name=dev-shared2-supervisor-i-e67015c7
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.version=1.6.0_27
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Sun Microsystems Inc.
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/opt/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/opt/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/opt/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/opt/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/opt/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/opt/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/opt/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/opt/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/opt/storm/storm-0.9.0.1/lib/guava-13.0.jar:/opt/storm/storm-0.9.0.1/lib/commo> ns-fileupload-1.2.1.jar:/opt/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/opt/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/opt/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/opt/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/asm-4.0.jar:/opt/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/opt/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/opt/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/opt/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/opt/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/opt/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/opt/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/o> pt/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/opt/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/opt/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/opt/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/opt/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/opt/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/opt/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/opt/storm/storm-0.9.0.1/lib/json-simple-1.1.jar:/opt/storm/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/opt/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/opt/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:/opt/storm/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/opt/storm/storm-0.9.0.1/conf:/mnt/storm/supervisor/stormdist/match_parser-12-1391735866/stormjar.jar
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:os.version=3.2.0-48-virtual
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:user.name=storm
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:user.home=/home/storm
> >> 2014-02-07 01:32:00 o.a.z.ZooKeeper [INFO] Client environment:user.dir=/opt/storm/storm-0.9.0.1
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:host.name=dev-shared2-supervisor-i-e67015c7
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.6.0_27
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Sun Microsystems Inc.
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/opt/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/opt/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/opt/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/opt/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/opt/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/opt/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/opt/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/opt/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/opt/storm/storm-0.9.0.1/lib/guava-13.0.jar:/opt/storm/storm-0.9.0.1/l> ib/commons-fileupload-1.2.1.jar:/opt/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/opt/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/opt/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/opt/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/asm-4.0.jar:/opt/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/opt/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/opt/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/opt/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/opt/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:/opt/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/opt/storm/storm-0.9.0.1/lib/ring-devel-0.3.1> 1.jar:/opt/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/opt/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/opt/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/opt/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/opt/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/opt/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/opt/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/opt/storm/storm-0.9.0.1/lib/json-simple-1.1.jar:/opt/storm/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/opt/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/opt/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:/opt/storm/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/opt/storm/storm-0.9.0.1/conf:/mnt/storm/supervisor/stormdist/match_parser-12-1391735866/stormjar.jar
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:os.version=3.2.0-48-virtual
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.home=/home/storm
> >> 2014-02-07 01:32:00 o.a.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/opt/storm/storm-0.9.0.1
> >> 2014-02-07 01:32:01 b.s.d.worker [INFO] Launching worker for match_parser-12-1391735866 on f1b66213-7237-483e-a5db-e543bb36bcd9:6710 with id 80483ed6-2c51-4f86-9d06-9d2bed705748 and conf {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/mnt/storm", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.timeouts" true, "nim> bus.cleanup.inbox.freq.secs" 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64, "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "10.0.5.50", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["10.0.6.131" "10.0.6.130" "10.0.6.55"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil, "drpc.queue.size" 128, "worker.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1%ID% -Xmx1200m", "sup> ervisor.heartbeat.frequency.secs" 5, "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 100, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending" nil, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703 6704 6705 6706 6707 6708 6709 6710 6711 6712 6713 6714 6715 6716 6717 6718 6719 6720 6721], "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "topology.workers" 1, "supervisor.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.> sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8091", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.wait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "drpc.request.timeout.secs> " 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8090", "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil}
> >> 2014-02-07 01:32:01 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
> >> 2014-02-07 01:32:01 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=10.0.6.131:2181,10.0.6.130:2181,10.0.6.55:2181 sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@25de152f
> >> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Opening socket connection to server /10.0.6.131:2181
> >> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Socket connection established to ip-10-0-6-131.ec2.internal/10.0.6.131:2181, initiating session
> >> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Session establishment complete on server ip-10-0-6-131.ec2.internal/10.0.6.131:2181, sessionid = 0xf343fa0ee4d600f0, negotiated timeout = 20000
> >> 2014-02-07 01:32:01 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
> >> 2014-02-07 01:32:01 o.a.z.ZooKeeper [INFO] Session: 0xf343fa0ee4d600f0 closed
> >> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] EventThread shut down
> >> 2014-02-07 01:32:01 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
> >> 2014-02-07 01:32:01 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=10.0.6.131:2181,10.0.6.130:2181,10.0.6.55:2181/storm sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@31930611
> >> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Opening socket connection to server /10.0.6.131:2181
> >> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Socket connection established to ip-10-0-6-131.ec2.internal/10.0.6.131:2181, initiating session
> >> 2014-02-07 01:32:01 o.a.z.ClientCnxn [INFO] Session establishment complete on server ip-10-0-6-131.ec2.internal/10.0.6.131:2181, sessionid = 0xf343fa0ee4d600f1, negotiated timeout = 20000
> >> 2014-02-07 01:32:01 b.s.m.TransportFactory [INFO] Storm peer transport plugin:backtype.storm.messaging.netty.Context
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loading executor 2:[2 2]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loaded executor tasks 2:[2 2]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Finished loading executor 2:[2 2]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loading executor __acker:[102 102]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Preparing bolt 2:(2)
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loaded executor tasks __acker:[102 102]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Timeouts disabled for executor __acker:[102 102]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Preparing bolt __acker:(102)
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Finished loading executor __acker:[102 102]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Prepared bolt __acker:(102)
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loading executor 3:[52 52]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loaded executor tasks 3:[52 52]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Finished loading executor 3:[52 52]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Preparing bolt 3:(52)
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loading executor __system:[-1 -1]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Loaded executor tasks __system:[-1 -1]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Finished loading executor __system:[-1 -1]
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Preparing bolt __system:(-1)
> >> 2014-02-07 01:32:01 b.s.d.worker [INFO] Launching receive-thread for f1b66213-7237-483e-a5db-e543bb36bcd9:6710
> >> 2014-02-07 01:32:01 b.s.d.executor [INFO] Prepared bolt __system:(-1)
> >> 2014-02-07 01:32:02 b.s.d.worker [INFO] Worker has topology config {"storm.id" "match_parser-12-1391735866", "dev.zookeeper.path" "/tmp/dev-storm-zookeeper", "topology.tick.tuple.freq.secs" nil, "topology.builtin.metrics.bucket.size.secs" 60, "topology.fall.back.on.java.serialization" true, "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xmx768m", "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 500, "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx128m", "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib", "executor.send.buffer.size" 16384, "topology.executor.send.buffer.size" 1024, "storm.local.dir" "/mnt/storm", "iwparser_socket" "/tmp/iwparser2Sockets/iwparserClients.ipc", "storm.messaging.netty.buffer_size" 5242880, "supervisor.worker.start.timeout.secs" 120, "topology.enable.message.t> imeouts" true, "nimbus.cleanup.inbox.freq.secs" 600, "batch_size" 50, "nimbus.inbox.jar.expiration.secs" 3600, "aws_creds.aws_secret_access_key" "wmGLQ4ItvdhniRFMJe3yDJvBwTJUTqbuLYAS5ZPU", "drpc.worker.threads" 64, "redis.processed.match.log.size" 100000, "redis.key.match.ok" "MATCH-OK", "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "10.0.5.50", "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181, "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size" 1024, "aws_creds.aws_access_key_id" "AKIAJP4O6MTXON3MLV2Q", "executor.receive.buffer.size" 16384, "transactional.zookeeper.servers" nil, "storm.zookeeper.root" "/storm", "aws.s3.match_bucket_name" "prod-matches", "storm.zookeeper.retry.intervalceiling.millis" 30000, "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1, "storm.zookeeper.servers" ["10.0.6.131" "10.0.6.130" "10.0.6.55"], "transactional.zookeeper.root" "/transactional", "topology.acker.executors" 50, "topology.kryo.decorators" > (), "topology.name" "match_parser", "topology.transfer.buffer.size" 1024, "topology.worker.childopts" "-Djruby.compat.version=RUBY1_9", "drpc.queue.size" 128, "worker.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1%ID% -Xmx1200m", "supervisor.heartbeat.frequency.secs" 5, "transfer.buffer.size" 32, "topology.error.throttle.interval.secs" 10, "throttle.spout" 1, "zmq.hwm" 0, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m", "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3, "topology.tasks" nil, "storm.messaging.netty.max_retries" 100, "topology.spout.wait.strategy" "backtype.storm.spout.SleepSpoutWaitStrategy", "receiver.buffer.size" 8, "debug.logging.redis" true, "topology.max.spout.pending" 1, "storm.zookeeper.retry.interval" 1000, "topology.sleep.spout.wait.strategy.time.ms" 1> , "platform" "xbox", "nimbus.topology.validator" "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports" [6700 6701 6702 6703 6704 6705 6706 6707 6708 6709 6710 6711 6712 6713 6714 6715 6716 6717 6718 6719 6720 6721], "iwparser_port" 12330, "redis.port" 6379, "topology.debug" false, "nimbus.task.launch.secs" 120, "nimbus.supervisor.timeout.secs" 60, "debug.logging.stdout" true, "topology.kryo.register" nil, "topology.message.timeout.secs" 30, "task.refresh.poll.secs" 10, "dw_retries.max_attempts" 6, "topology.workers" 50, "supervisor.childopts" "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8091", "nimbus.thrift.port" 6627, "redis_prefix" "prod", "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer" "backtype.storm.serialization.types.ListDelegateSerializer", "topology.disruptor.w> ait.strategy" "com.lmax.disruptor.BlockingWaitStrategy", "http_proxy" "proxy", "nimbus.task.timeout.secs" 30, "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory" "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port" 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times" 5, "storm.thrift.transport" "backtype.storm.security.auth.SimpleTransportPlugin", "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.messaging.transport" "backtype.storm.messaging.netty.Context", "logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000, "debug.logging.kibana" false, "iwparser_host" "10.200.6.145", "drpc.request.timeout.secs" 600, "demonware.endpoint" "http://10.0.1.78:9000/matches/", "redis_simple_prefix" "prod", "redis.host" "storm090-redis.uvvujo.0001.use1.cache.amazonaws.com", "dw_retries.wait_interval" 2, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts> " "-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8090", "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil, "statsd.endpoint" "statsd", "redis.log.size" 100000}
> >> 2014-02-07 01:32:02 b.s.d.worker [INFO] Worker 80483ed6-2c51-4f86-9d06-9d2bed705748 for storm match_parser-12-1391735866 on f1b66213-7237-483e-a5db-e543bb36bcd9:6710 has finished loading
> >> 2014-02-07 01:32:08 MatchRetrieveBolt [INFO] [BOLT-1] on_init
> >> 2014-02-07 01:32:08 MatchProcessBolt [INFO] [BOlT-MatchProcessBolt] on_init
> >> 2014-02-07 01:32:08 b.s.d.executor [INFO] Prepared bolt 2:(2)
> >> 2014-02-07 01:32:08 b.s.d.executor [INFO] Prepared bolt 3:(52)
> >>
> >>
> >> The bolts have reinitialized. The Storm UI is showing that bolt1 is still being handled by this specific worker.
> >>
> >>
> >>
> >> The spout still sends tuples to matchRetrieveBolt but never gets a response. The failure is caught and retries continue.
> >>
> >>
> >> Can anyone comment or provide any thoughts on the above issue?
> >>
> >> Thanks,
> >> Mark
> >>
> >>
> >
> >
>