[jira] [Commented] (STORM-95) Topology hangs with worker processor threads TIMED_WAITING. Edit

["P. Taylor Goetz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-95?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866165#comment-13866165 ] 

P. Taylor Goetz commented on STORM-95:
--------------------------------------

This behavior is typically caused by doing something with STDOUT like enabling GC logging without directing it to a file. It can also happen with some logging configurations that log to CONSOLE.

I would suggest turning off GC logging or making sure that it is properly directed to a file. If you have altered the logging settings, I would try reverting to the default settings that Storm comes with.

> Topology hangs with worker processor threads TIMED_WAITING. Edit
> ----------------------------------------------------------------
>
>                 Key: STORM-95
>                 URL: https://issues.apache.org/jira/browse/STORM-95
>             Project: Apache Storm (Incubating)
>          Issue Type: Improvement
>            Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/763
> Hi Nathan,
> We are this issue very frequently now while using Storm 0.8.2. There are no errors in any worker logs/ supervisor.log/nimbus.log . However the topology stops processing the tuples.
> On collesting the thread dump of the worker processor we can see all the threads are going into TIMED_WAITING states and toplogy hangs.
> The following is the brief on our toplogy.
> We are using BaseIRich Spout and bolts.
> We have one file reader spout and three processing bolts.(24, 48 and 24 executors)
> Each tuple will contain 100 messages of size 10kb each totaling 1mb.
> We aim to process 30 mil such records within 6 hrs.
> We are running it on SUSE Linux 11 entreprise server.
> We are using all the recomended versions (Storm 0.8.2,Java 1.7, Zookeeper 3.4.5, ZeroMQ - 2.1.7, JZMQ-)
> Below are the list of variuos combination of the storm configuration we tried.
> Conf -3
> worker.childopts: "-Xmx3072m"
> topology.acker.executors: 20
> topology.max.spout.pending: 50
> topology.message.timeout.secs: 300
> topology.executor.receive.buffer.size: 16384 #batched
> topology.executor.send.buffer.size: 16384
> Conf-2
> worker.childopts: "-Xmx3072m"
> topology.acker.executors: 20
> topology.max.spout.pending: 300
> topology.message.timeout.secs: 300
> topology.executor.receive.buffer.size: 16384 #batched
> topology.executor.send.buffer.size: 16384
> Conf-1
> worker.childopts: "-Xmx3072m"
> topology.acker.executors: 20
> topology.max.spout.pending: 1000
> topology.message.timeout.secs: 300
> Also attaching the thread dumps for your reference.
> We desperately need your help to resolve this issue as we are looking to go live soon.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)