[jira] [Commented] (FLINK-951) Reworking of Iteration Synchronization, Accumulators and Aggregators

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040622#comment-14040622 ] 

ASF GitHub Bot commented on FLINK-951:
--------------------------------------

GitHub user markus-h opened a pull request:

    https://github.com/apache/incubator-flink/pull/36

    Reworking of Iteration Synchronization, Accumulators and Aggregators

    This pullrequest includes:
    Iteration superstep synchronization through RPC and JobManager
    Unification of Accumulators and Aggregators (removal of former Aggregators)
    Adjusted Testcases accordingly
    
    A detailed description can also be found on Jira: https://issues.apache.org/jira/browse/FLINK-951
    
    Currently the testcase "KMeansIterativeNepheleITCase" sometimes fails. I think this is connected to this issue: https://issues.apache.org/jira/browse/FLINK-909 and needs to be resolved in a following pull request.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/markus-h/incubator-flink aggregatorsReworkToAccumulatorsRebase4

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-flink/pull/36.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #36
    
----
commit 9da8aa6442861be17c7658c1dd2c2d9a6943d507
Author: Markus Holzemer <ma...@gmx.de>
Date:   2014-06-16T12:56:36Z

    Iteration superstep synchronization through RPC and JobManager
    Unification of Accumulators and Aggregators (removal of former Aggregators)
    Adjusted Testcases accordingly

----


> Reworking of Iteration Synchronization, Accumulators and Aggregators
> --------------------------------------------------------------------
>
>                 Key: FLINK-951
>                 URL: https://issues.apache.org/jira/browse/FLINK-951
>             Project: Flink
>          Issue Type: Improvement
>          Components: Optimizer
>    Affects Versions: 0.6-incubating
>            Reporter: Markus Holzemer
>            Assignee: Markus Holzemer
>              Labels: refactoring
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> I just realized that there is no real Jira issue for the task I am currently working on. 
> I am currently reworking a few things regarding Iteration Synchronization, Accumulators and Aggregators. Currently the synchronization at the end of one superstep is done through channel events. That makes it hard to track the current status of iterations. That is why I am changing this synchronization to use RPC calls with the JobManager, so that the JobManager manages the current status of all iterations.
> Currently we use Accumulators outside of iterations and Aggregators inside of iterations. Both have a similiar function, but a bit different interfaces and handling. I want to unify these two concepts. I propose that we stick in the future to Accumulators only. Aggregators therefore are removed and Accumulators are extended to cover the usecases Aggregators were used fore before. The switch to RPC for iterations makes it possible to also send the current Accumulator values at the end of each superstep, so that the JobManager (and thereby the webinterface) will be able to print intermediate accumulation results.



--
This message was sent by Atlassian JIRA
(v6.2#6252)