[jira] [Commented] (PIG-4846) Use pigmix to test the performance of pig on spark

["liyunzhang_intel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483372#comment-15483372 ] 

liyunzhang_intel commented on PIG-4846:
---------------------------------------

[~rohini],[~xuefuz], [~mohitsabharwal],[~pallavi.rao]:  Here is the comparision of Pig on Spark and Pig on MR in 1 TB pigmix test after using patch of PIG-4920 and PIG-5029.

||Script||mr||spark| (mr-spark)/mr*100|
|L_1|8089|7934|1.91|
|L_2|6540|6183|5.45|
|L_3|6956|6972|-0.23|
|L_4|6730|6843|-1.67|
|L_5|6752|6454|4.41|
|L_6|7046|7425|-5.37|
|L_7|6916|7250|-4.82|
|L_8|6697|6878|-2.70|
|L_9|52648|30240|42.56|
|L_10|48641|Fail| |
|L_11|6916|6346|8.24|
|L_12|6922|Fail| |
|L_13|7598|7569|0.38|
|L_14|8454|7089|16.14|
|L_15|6647|Fail| |
|L_16|6921|Fail| |
|L_17|24242|Fail | |

From the result we can see following points:
1. there are some failures in the whole test in spark mode because of different reasons. I will continue to investigate why they fail
2. Pig On Spark runs as faster as Pig On MR L1,2,3,4,5,6,7,8,13( i think they are same if (mr-spark)/mr*100% is  from-5%~ 5%)
3. Pig On Spark runs faster than Pig On MR on L9 by 42.5%,L11 by 8.24%, L14 by 16.14%


> Use pigmix to test the performance of pig on spark
> --------------------------------------------------
>
>                 Key: PIG-4846
>                 URL: https://issues.apache.org/jira/browse/PIG-4846
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>             Fix For: spark-branch
>
>         Attachments: PIG-4846.patch, PIG-4846_1.patch
>
>
> We can compare the performance between mr and spark mode by pigmix.
> The introduction of pigmix is https://cwiki.apache.org/confluence/display/PIG/PigMix.
> PIG-4846.patch is to make pigmix run by specied exectype.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)