[jira] [Created] (STORM-2017) ShellBolt stops reporting task ids

["Lasse Kiviluoto (JIRA)" <ji...@apache.org>]

Lasse Kiviluoto created STORM-2017:
--------------------------------------

             Summary: ShellBolt stops reporting task ids
                 Key: STORM-2017
                 URL: https://issues.apache.org/jira/browse/STORM-2017
             Project: Apache Storm
          Issue Type: Bug
          Components: storm-core
    Affects Versions: 1.0.1, 1.0.3
            Reporter: Lasse Kiviluoto


After running enough flow throw ShellBolt in some cases after tens of minutes ShellBolt stopped reporting task ids. After this error condition no new task ids where reported back. When acking of the tuples processed by the bolt where set in callback related to arrival of the task ids all tuple trees going through the bolt would fail after reporting stopped. ShellBolt will continue to operate new tuples and respond to heartbeats.

After running some tests and making some changes to the code. I have following hypothesis for the reason:

org.apache.storm.utils.ShellBoltMessageQueue has two queues one being for taskIds and the other for bolt messages.
taskIds queue is implemented by LinkedList and bolt msg queue LinkedBlockingQueue. Both of the queues are operated similarly.
One major difference between the structures is that LinkedList is not synchronized.

In the code:

ShellBoltMessageQueue.java:58 add method is used without holding the lock. Where as ShellBoltMessageQueue.java:110 uses the poll method with the lock. 
As in ShellBolt BoltReaderRunnable and BoltWriterRunnable are run concurrently this can lead to race condition.

If I move the ShellBoltMessageQueue.java:58 inside the lock and run the test in similar fashion it seems to solve the issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)