[jira] [Created] (SPARK-7542) Use LongArray for sort buffer in UnsafeExternalSorter

["Josh Rosen (JIRA)" <ji...@apache.org>]

Josh Rosen created SPARK-7542:
---------------------------------

             Summary: Use LongArray for sort buffer in UnsafeExternalSorter
                 Key: SPARK-7542
                 URL: https://issues.apache.org/jira/browse/SPARK-7542
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 1.4.0
            Reporter: Josh Rosen


{{UnsafeExternalSorter}}, introduced in SPARK-7081, uses on-heap {{long[]}} arrays as its sort buffers.  When records are small, the sorting array might be as large as the data pages, so it would be useful to be able to allocate this array off-heap (using our unsafe LongArray).  Unfortunately, we can't currently do this because TimSort calls {{allocate()}} to create data buffers but doesn't call any corresponding cleanup methods to free them.

We should look into extending TimSort with buffer freeing methods, then consider switching to LongArray in UnsafeShuffleSortDataFormat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org