You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "dzcxzl (Jira)" <ji...@apache.org> on 2019/10/30 11:13:00 UTC
[jira] [Created] (SPARK-29657) Iterator spill supporting radix sort
with null prefix
dzcxzl created SPARK-29657:
------------------------------
Summary: Iterator spill supporting radix sort with null prefix
Key: SPARK-29657
URL: https://issues.apache.org/jira/browse/SPARK-29657
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.4.0
Reporter: dzcxzl
In the case of radix sort, when the insertRecord part of the keyPrefix is null, the iterator type returned by getSortedIterator is ChainedIterator.
Currently ChainedIterator does not support spill, causing UnsafeExternalSorter to take up a lot of execution memory, allocatePage fails, throw SparkOutOfMemoryError Unable to acquire xxx bytes of memory, got 0
The following is a log of an error we encountered in the production environment.
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Memory used in task 66055
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@39dd866e: 64.0 KB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@74d17927: 4.6 GB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: Acquired by org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@31478f9c: 61.0 MB
[Executor task launch worker for task 66055] INFO TaskMemoryManager: 0 bytes of memory were used by task 66055 but are not associated with specific consumers
[Executor task launch worker for task 66055] INFO TaskMemoryManager: 4962998749 bytes of memory are used for execution and 2218326 bytes of memory are used for storage
[Executor task launch worker for task 66055] ERROR Executor: Exception in task 42.3 in stage 29.0 (TID 66055)
SparkOutOfMemoryError: Unable to acquire 3436 bytes of memory, got 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org