You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "liyunzhang_intel (JIRA)" <ji...@apache.org> on 2016/12/08 06:26:58 UTC
[jira] [Comment Edited] (PIG-4952) Calculate the value of parallism
for spark mode
[ https://issues.apache.org/jira/browse/PIG-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731254#comment-15731254 ]
liyunzhang_intel edited comment on PIG-4952 at 12/8/16 6:26 AM:
----------------------------------------------------------------
[~kexianda]:
"spark.default.parallism" is the configuration item to set the parallism([see|http://spark.apache.org/docs/latest/configuration.html]). before we use "spark.reducers", i guess "spark.reducers" is not official configuration name. will change the name to "spark.default.parallism".
[~nkollar]:
{quote}
Does it make sense to set the parallelism level for these operators too,
{quote}
Make sense, will accept the modification in PIG-4952_1.patch about SkewedJoinConverter.java,SortConverter.java and RankConverter.java.
[~kexianda] and [~nkollar]: please help review PIG-4952_2.patch in [review board|https://reviews.apache.org/r/54523/]
was (Author: kellyzly):
[~kexianda]:
"spark.default.parallism" is the configuration item to set the parallism([see|http://spark.apache.org/docs/latest/configuration.html]). before we use "spark.reducers", i guess "spark.reducers" is not official configuration name. will change the name to "spark.default.parallism".
[~nkollar]:
{quote}
Does it make sense to set the parallelism level for these operators too,
{quote}
Make sense, will accept the modification in PIG-4952_1.patch about SkewedJoinConverter.java,SortConverter.java and RankConverter.java.
[~kexianda] and [~nkollar]: please help review PIG-4952_2.patch.
> Calculate the value of parallism for spark mode
> -----------------------------------------------
>
> Key: PIG-4952
> URL: https://issues.apache.org/jira/browse/PIG-4952
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4952.patch, PIG-4952_1.patch, PIG-4952_2.patch
>
>
> Calculate the value of parallism for spark mode like what org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.ParallelismSetter does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)