[jira] [Resolved] (BEAM-797) A PipelineVisitor that creates a Spark-native pipeline.

["Amit Sela (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit Sela resolved BEAM-797.
----------------------------
       Resolution: Fixed
    Fix Version/s: First stable release

> A PipelineVisitor that creates a Spark-native pipeline. 
> --------------------------------------------------------
>
>                 Key: BEAM-797
>                 URL: https://issues.apache.org/jira/browse/BEAM-797
>             Project: Beam
>          Issue Type: Wish
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Aviem Zur
>            Priority: Minor
>             Fix For: First stable release
>
>
> It could be very useful for debugging purposes to have a custom PipelineVisitor that can tell what's the underlying Spark code that is being called.
> One idea:
> This could be called with a flag in SparkPipelineOptions and instead of executing the pipeline, it would print the underlying Spark DAG.
> Clearly, DoFn internals would be obfuscated, but the Spark code could note {{mapPartitions("ExtractWords")}}
> Another difference would be Sources as they are a custom implementation for Beam.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)