You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 05:37:31 UTC

[jira] [Resolved] (SPARK-4112) Have a reserved copy of Sorter/SortDataFormat

     [ https://issues.apache.org/jira/browse/SPARK-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-4112.
---------------------------------
    Resolution: Incomplete

> Have a reserved copy of Sorter/SortDataFormat
> ---------------------------------------------
>
>                 Key: SPARK-4112
>                 URL: https://issues.apache.org/jira/browse/SPARK-4112
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle, Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Xiangrui Meng
>            Priority: Major
>              Labels: bulk-closed
>
> After SPARK-4084, developers can use Sorter with their own SortDataFormat. However, if there are multiple subclasses of SortDataFormat instantiated, JIT won't inline the methods in SortDataFormat and virtual method table lookup is slow. One solution could be making two copies of the code and reserve one for shuffle only, and expose the other to developers.
> Before we do that, we should compare the performance with/without JIT and check whether it is worth the extra code complexity.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org