You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Aviem Zur (JIRA)" <ji...@apache.org> on 2017/03/21 04:04:41 UTC
[jira] [Comment Edited] (BEAM-1765) Remove Aggregators from Spark
runner
[ https://issues.apache.org/jira/browse/BEAM-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934063#comment-15934063 ]
Aviem Zur edited comment on BEAM-1765 at 3/21/17 4:04 AM:
----------------------------------------------------------
Awesome to hear we are moving forward on removing aggregators!
How can this be removed from the runners independently from Java SDK?
Most uses of aggregators in Spark runner is to pass these aggregators to SDK methods and constructors which require them. For example: {{ReduceFnRunner}}.
Another use are the {{PAssert}} success and counter aggregators which allow runners to ensure that all assertions in tests actually happened and not pass tests that should have failed.
The only other use is the support of aggregators in {{SparkPipelineResult}} which can be removed independently.
was (Author: aviemzur):
How can this be removed from the runners independently from Java SDK?
Most uses of aggregators in Spark runner is to pass these aggregators to SDK methods and constructors which require them. For example: {{ReduceFnRunner}}.
Another use are the {{PAssert}} success and counter aggregators which allow runners to ensure that all assertions in tests actually happened and not pass tests that should have failed.
The only other use is the support of aggregators in {{SparkPipelineResult}} which can be removed independently.
> Remove Aggregators from Spark runner
> ------------------------------------
>
> Key: BEAM-1765
> URL: https://issues.apache.org/jira/browse/BEAM-1765
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Reporter: Pablo Estrada
> Assignee: Amit Sela
>
> I have started removing aggregators from the Java SDK, but runners use them in different ways that I can't figure out well. This is to track the independent effort in Spark.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)