[jira] [Commented] (HIVE-15259) The deserialization time of HOS20 is longer than what in HOS16

["Rui Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15686282#comment-15686282 ] 

Rui Li commented on HIVE-15259:
-------------------------------

With Spark 2.0, you don't have to copy all the jars to Hive lib. Please refer to our wiki https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started.

The log you posted is from Spark. I guess it uploads the required jars to prepare the classpath for containers. Not sure if it's related to the deserialization time. And on Hive side we don't have much control on it - we basically just specify that hive-exec is needed. Others are up to Spark.

> The deserialization time of HOS20 is longer than what in  HOS16
> ---------------------------------------------------------------
>
>                 Key: HIVE-15259
>                 URL: https://issues.apache.org/jira/browse/HIVE-15259
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: liyunzhang_intel
>         Attachments: Deserialization_HOS16.PNG, Deserialization_HOS20.PNG
>
>
> deploy Hive on Spark on spark 1.6 version and spark 2.0 version.
> run query and in latest code(with spark2.0) the deserialization time of a task is 4 sec while the deserialization time of spark1.6 is 1 sec. The detail is in attached picture.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)