[jira] [Created] (STORM-171) Add "progress" method to OutputCollector

["James Xu (JIRA)" <ji...@apache.org>]

James Xu created STORM-171:
------------------------------

             Summary: Add "progress" method to OutputCollector
                 Key: STORM-171
                 URL: https://issues.apache.org/jira/browse/STORM-171
             Project: Apache Storm (Incubating)
          Issue Type: New Feature
            Reporter: James Xu
            Priority: Minor


https://github.com/nathanmarz/storm/issues/168

void progress(Tuple input)

This would send a message back to the spout (through the acker) to increase the timeout for the roots of the tuple. Would be useful if the processing times of a tuple is highly variable. The timeout should reset to TOPOLOGY_MESSAGE_TIMEOUT plus the current time.

-----------
xumingming: what's the content of the tuple passed to the method progress?

-----------
nathanmarz: It would be the tuple that was passed to "execute". For example, in pseudocode:

execute(Tuple input):
while(...):
// do some processing
_collector.progress(input)
_collector.ack(input)

So the progress method would extend the timeout for the spout tuples at the root of "input"

Just a note – a "progress" method could be a key to implementing MapReduce on top of Storm (along with more powerful grouping/scheduling capabilities and the ability to make use of local disk).

-----------
rohitprasad15: I wanted to take up this issue. I learnt Clojure and Storm a week back, but want to get deep into both.
I can briefly describe my approach - 
1. Modify task.clj. Implement ^void progress [this ^Tuple tuple] as part of output-collector. Send a message to the acker, in a way similar to how ack method does it.
2. Modify acker.clj. Change execute() of IBolt implementation. Need to add ACKER-PROGRESS-STREAM-ID, to distinguish that its a progress message, and then finally reset the timeout to maxTopologyMessageTimeout.

Is this approach in the right direction?




--
This message was sent by Atlassian JIRA
(v6.1.4#6159)