[jira] [Created] (BEAM-797) A PipelineVisitor that creates a Spark-native pipeline.

["Amit Sela (JIRA)" <ji...@apache.org>]

Amit Sela created BEAM-797:
------------------------------

             Summary: A PipelineVisitor that creates a Spark-native pipeline. 
                 Key: BEAM-797
                 URL: https://issues.apache.org/jira/browse/BEAM-797
             Project: Beam
          Issue Type: Wish
          Components: runner-spark
            Reporter: Amit Sela
            Assignee: Amit Sela
            Priority: Minor


It could be very useful for debugging purposes to have a custom PipelineVisitor that can tell what's the underlying Spark code that is being called.

One idea:
This could be called with a flag in SparkPipelineOptions and instead of executing the pipeline, it would print the underlying Spark DAG.
Clearly, DoFn internals would be obfuscated, but the Spark code could note {{mapPartitions("ExtractWords")}}
Another difference would be Sources as they are a custom implementation for Beam.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)