[jira] [Commented] (PIG-4893) Task deserialization time is too long for spark on yarn mode

["liyunzhang_intel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282531#comment-15282531 ] 

liyunzhang_intel commented on PIG-4893:
---------------------------------------

[~mohitsabharwal] and [~xuefuz]: Can you help to view this issue?
It shows that in some stage the task deserialization is a bit long in attached picture.
It is reasonable that task deserialization is as long as about 10s?
If not, the problem is because we use many big objects in our code or we load many jars in the [SPARK_JARS|https://github.com/apache/pig/blob/spark/bin/pig#L391]? 


> Task deserialization time is too long for spark on yarn mode
> ------------------------------------------------------------
>
>                 Key: PIG-4893
>                 URL: https://issues.apache.org/jira/browse/PIG-4893
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: liyunzhang_intel
>             Fix For: spark-branch
>
>         Attachments: time.PNG
>
>
> I found the task deserialization time is a bit long when i run any scripts of pigmix in spark on yarn mode.  see the attachment picture.  The duration time is 3s but the task deserialization is 13s.  
> My env is hadoop2.6+spark1.6.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)