You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Ismaël Mejía (JIRA)" <ji...@apache.org> on 2016/11/23 10:10:01 UTC
[jira] [Assigned] (BEAM-797) A PipelineVisitor that creates a
Spark-native pipeline.
[ https://issues.apache.org/jira/browse/BEAM-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ismaël Mejía reassigned BEAM-797:
---------------------------------
Assignee: Ismaël Mejía
> A PipelineVisitor that creates a Spark-native pipeline.
> --------------------------------------------------------
>
> Key: BEAM-797
> URL: https://issues.apache.org/jira/browse/BEAM-797
> Project: Beam
> Issue Type: Wish
> Components: runner-spark
> Reporter: Amit Sela
> Assignee: Ismaël Mejía
> Priority: Minor
>
> It could be very useful for debugging purposes to have a custom PipelineVisitor that can tell what's the underlying Spark code that is being called.
> One idea:
> This could be called with a flag in SparkPipelineOptions and instead of executing the pipeline, it would print the underlying Spark DAG.
> Clearly, DoFn internals would be obfuscated, but the Spark code could note {{mapPartitions("ExtractWords")}}
> Another difference would be Sources as they are a custom implementation for Beam.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)