[jira] [Commented] (PIG-4846) Use pigmix to test the performance of pig on spark

["liyunzhang_intel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347613#comment-15347613 ] 

liyunzhang_intel commented on PIG-4846:
---------------------------------------

[~pallavi.rao],[~mohitsabharwal]:
Following is the result of pigmix between spark and mr mode. I use patch of PIG-4797, PIG-4903, PIG-4893.
The test env: 3 nodes cluster( i assign 28 cpu core and 56g memery on each node)
yarn configuration
{code}
yarn.nodemanager.resource.memory-mb=56G
 yarn.nodemanger.resource.cpu-vcores=28
{code}

Spark configurations in pig.properties
{code}
spark.executor.cores=4
#6553m=6.4g
spark.executor.memory=6553m  
#1638=1.6g
spark.yarn.executor.memoryOverhead=1638  
#2048m=2g
spark.driver.memory=2048m
spark.executor.instances=7
{code}

||Script||mr||spark||
|L_1|41|33|
|L_2|37|28|
|L_3|67|30|
|L_4|36|29|
|L_5|36|25|
|L_6|36|27|
|L_7|36|24|
|L_8|25|24|
|L_9|260|148|
|L_10|298|148|
|L_11|154|36|
|L_12|41|32|
|L_13|36|24|
|L_14|53|22|
|L_15|35|24|
|L_16|35|25|
|L_17|76|33|


It shows that pig on spark runs faster than pig on mr.

> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>
>                 Key: PIG-4846
>                 URL: https://issues.apache.org/jira/browse/PIG-4846
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>             Fix For: spark-branch
>
>         Attachments: PIG-4846.patch, PIG-4846_1.patch
>
>
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)