You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pig.apache.org by "liyunzhang_intel (JIRA)" <ji...@apache.org> on 2016/04/05 03:44:25 UTC

[jira] [Created] (PIG-4863) Re-design Spark plan to optimize the RDD pipeline

liyunzhang_intel created PIG-4863:
-------------------------------------

             Summary: Re-design Spark plan to optimize the RDD pipeline
                 Key: PIG-4863
                 URL: https://issues.apache.org/jira/browse/PIG-4863
             Project: Pig
          Issue Type: Sub-task
          Components: spark
            Reporter: liyunzhang_intel


Currently the SparkLaucher converts the SparkPlan to RDD pipeline and immediately executes it,There is no intermediate step that allows optimization of the RDD pipeline, if so deemed necessary,before execution, This will need re-working of sparkPlanToRDD(), perhaps by introduction of a RDDPlan of RDDOperators.   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)