[jira] [Commented] (STORM-323) Unacknowledged __tick and __metrics_tick tuples hangs worker processes

["Srinath (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STORM-323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000615#comment-14000615 ] 

Srinath commented on STORM-323:
-------------------------------

The discussions over mailing list also provide some information:
User list:
http://mail-archives.apache.org/mod_mbox/storm-user/201405.mbox/%3cCAHhx-GLJPrs-h2wUBE-aNT33UeS2-sUkAiuqR=2Kg2+jusfvjQ@mail.gmail.com%3e

Dev list:
http://mail-archives.apache.org/mod_mbox/storm-dev/201405.mbox/ajax/%3CCAHhx-GLzaKeYZbD1B6YQQkxRfPOPxvrZUVf1v85ch3u82xriLw%40mail.gmail.com%3E

> Unacknowledged __tick and __metrics_tick tuples hangs worker processes
> ----------------------------------------------------------------------
>
>                 Key: STORM-323
>                 URL: https://issues.apache.org/jira/browse/STORM-323
>             Project: Apache Storm (Incubating)
>          Issue Type: Bug
>    Affects Versions: 0.9.1-incubating
>         Environment: Storm:
> Nimbus, Supervisor and Zookeeper running on Centos 6.2 over m1.small instances (1.7G mem, 1 CPU, 1 core)
> Netty as the transport
> Topology:
> 2 worker processes on the same supervisor instance each allocated 512 Mb of heap
> Each of the worker processes have around 30 executors running around 112 tasks.
>            Reporter: Srinath
>            Priority: Critical
>
> Symptoms observed:
> 1. One of the bolts not getting executed after about 5 days of run
> 2. Spout gradually slows down and finally stops calling nextTuple()
> 3. Topology is non-functional since there is no exchange of tuples across worker processes
> Notes from troubleshooting:
> 1. There is a transfer of data across worker processes but the bolt is not receiving the tuples
> 2. backtype.storm.messaging.netty.Server#message_queue is not getting consumed.
> 3. Later on found that there are several __tick and __metrics_tick tuples piling up in memory over a period of time. This piling up is gradual and probably the reason why it takes so long for it to cause any visible problems.
> I have shared access to thread dumps and topology layout at https://drive.google.com/folderview?id=0B2F_3UACQZNESXpwZlA4MFlqSVU&usp=drive_web



--
This message was sent by Atlassian JIRA
(v6.2#6252)