You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "dzcxzl (Jira)" <ji...@apache.org> on 2019/11/01 03:22:00 UTC
[jira] [Updated] (SPARK-29657) Iterator spill supporting radix sort
with null prefix
[ https://issues.apache.org/jira/browse/SPARK-29657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dzcxzl updated SPARK-29657:
---------------------------
Issue Type: Bug (was: Improvement)
> Iterator spill supporting radix sort with null prefix
> -----------------------------------------------------
>
> Key: SPARK-29657
> URL: https://issues.apache.org/jira/browse/SPARK-29657
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: dzcxzl
> Priority: Trivial
>
> In the case of radix sort, when the insertRecord part of the keyPrefix is null, the iterator type returned by getSortedIterator is ChainedIterator.
> Currently ChainedIterator does not support spill, causing UnsafeExternalSorter to take up a lot of execution memory, allocatePage fails, throw SparkOutOfMemoryError Unable to acquire xxx bytes of memory, got 0
> The following is a log of an error we encountered in the production environment.
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: Memory used in task 66055
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@39dd866e: 64.0 KB
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@74d17927: 4.6 GB
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@31478f9c: 61.0 MB
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: 0 bytes of memory were used by task 66055 but are not associated with specific consumers
> [Executor task launch worker for task 66055] INFO TaskMemoryManager: 4962998749 bytes of memory are used for execution and 2218326 bytes of memory are used for storage
> [Executor task launch worker for task 66055] ERROR Executor: Exception in task 42.3 in stage 29.0 (TID 66055)
> SparkOutOfMemoryError: Unable to acquire 3436 bytes of memory, got 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org