You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Yin Huai (JIRA)" <ji...@apache.org> on 2015/04/15 02:51:58 UTC

[jira] [Updated] (SPARK-6368) Build a specialized serializer for Exchange operator.

     [ https://issues.apache.org/jira/browse/SPARK-6368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yin Huai updated SPARK-6368:
----------------------------
    Attachment: SchemaBased.nps
                Kryo.nps

SchemaBased.nps is the profiling result of the new serializer. You can take a look at Executor thread.

> Build a specialized serializer for Exchange operator. 
> ------------------------------------------------------
>
>                 Key: SPARK-6368
>                 URL: https://issues.apache.org/jira/browse/SPARK-6368
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Yin Huai
>            Assignee: Yin Huai
>            Priority: Critical
>         Attachments: Kryo.nps, SchemaBased.nps
>
>
> Kryo is still pretty slow because it works on individual objects and relative expensive to allocate. For Exchange operator, because the schema for key and value are already defined, we can create a specialized serializer to handle the specific schemas of key and value. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org