[jira] [Created] (PIG-4549) Set CROSS operation parallelism for Spark engine

["Mohit Sabharwal (JIRA)" <ji...@apache.org>]

Mohit Sabharwal created PIG-4549:
------------------------------------

             Summary: Set CROSS operation parallelism for Spark engine
                 Key: PIG-4549
                 URL: https://issues.apache.org/jira/browse/PIG-4549
             Project: Pig
          Issue Type: Sub-task
          Components: spark
    Affects Versions: spark-branch
            Reporter: Mohit Sabharwal
            Assignee: Mohit Sabharwal
             Fix For: spark-branch


Spark engine should set parallelism to be used for CROSS operation by GFCross UDF.

If not set, GFCross throws an exception:
{code}
                String s = cfg.get(PigImplConstants.PIG_CROSS_PARALLELISM + "." + crossKey);
                if (s == null) {
                    throw new IOException("Unable to get parallelism hint from job conf");
                }
{code}

Estimating parallelism for Spark engine is a TBD item. Until that is done, for CROSS to work, we should use the default parallelism value in GFCross.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)