You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nemo.apache.org by GitBox <gi...@apache.org> on 2018/11/04 14:31:36 UTC

[GitHub] johnyangk opened a new pull request #148: [NEMO-260] Beam Accumulator-based Partial Aggregation

johnyangk opened a new pull request #148: [NEMO-260] Beam Accumulator-based Partial Aggregation
URL: https://github.com/apache/incubator-nemo/pull/148
 
 
   JIRA: [NEMO-260: Beam Accumulator-based Partial Aggregation](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-260)
   
   **Major changes:**
   - Accumulator-based combining for Combine.PerKey PTransforms
   - Direct translation from the Beam DAG to the Nemo IR DAG, using the CompositeBehavior (similar to the Spark/Flink runners)
   - CombineFnPartialTransform, and CombineFnFinalTransform
   
   **Minor changes to note:**
   - A bit more clearer separation of concerns between the Beam frontend components: NemoPipelineRunner, PipelineTranslationContext, PipelineTranslator, PipelineVisitor
   - Reduces the number of vertices to examine in optimizationpass tests, to make it easier to change them
   
   **Tests for the changes:**
   - Existing tests that use Per.Key continue to pass
   
   **Other comments:**
   - This change applies to batch operations only (for now)
   
   Closes #GITHUB_PR_NUMBER
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services