You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@storm.apache.org by "Stig Rohde Døssing (JIRA)" <ji...@apache.org> on 2017/03/17 20:03:41 UTC
[jira] [Resolved] (STORM-171) Add "progress" method to
OutputCollector
[ https://issues.apache.org/jira/browse/STORM-171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stig Rohde Døssing resolved STORM-171.
--------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
1.0.0
This sounds similar to https://issues.apache.org/jira/browse/STORM-1549. I'm marking this one as resolved. Feel free to reopen if https://issues.apache.org/jira/browse/STORM-1549 doesn't cover this issue.
> Add "progress" method to OutputCollector
> ----------------------------------------
>
> Key: STORM-171
> URL: https://issues.apache.org/jira/browse/STORM-171
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-core
> Reporter: James Xu
> Priority: Minor
> Fix For: 1.0.0, 2.0.0
>
>
> https://github.com/nathanmarz/storm/issues/168
> void progress(Tuple input)
> This would send a message back to the spout (through the acker) to increase the timeout for the roots of the tuple. Would be useful if the processing times of a tuple is highly variable. The timeout should reset to TOPOLOGY_MESSAGE_TIMEOUT plus the current time.
> -----------
> xumingming: what's the content of the tuple passed to the method progress?
> -----------
> nathanmarz: It would be the tuple that was passed to "execute". For example, in pseudocode:
> execute(Tuple input):
> while(...):
> // do some processing
> _collector.progress(input)
> _collector.ack(input)
> So the progress method would extend the timeout for the spout tuples at the root of "input"
> Just a note – a "progress" method could be a key to implementing MapReduce on top of Storm (along with more powerful grouping/scheduling capabilities and the ability to make use of local disk).
> -----------
> rohitprasad15: I wanted to take up this issue. I learnt Clojure and Storm a week back, but want to get deep into both.
> I can briefly describe my approach -
> 1. Modify task.clj. Implement ^void progress [this ^Tuple tuple] as part of output-collector. Send a message to the acker, in a way similar to how ack method does it.
> 2. Modify acker.clj. Change execute() of IBolt implementation. Need to add ACKER-PROGRESS-STREAM-ID, to distinguish that its a progress message, and then finally reset the timeout to maxTopologyMessageTimeout.
> Is this approach in the right direction?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)