[jira] [Created] (SPARK-27431) move HashedRelation to global UnifiedMemoryManager and enable offheap

["Xiaoju Wu (JIRA)" <ji...@apache.org>]

Xiaoju Wu created SPARK-27431:
---------------------------------

             Summary: move HashedRelation to global UnifiedMemoryManager and enable offheap
                 Key: SPARK-27431
                 URL: https://issues.apache.org/jira/browse/SPARK-27431
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.3.0
            Reporter: Xiaoju Wu


Why is HashedRelation currently managed by a newly created MemoryManager and disabled with offheap? Can we improve this part?

/**
 * Create a HashedRelation from an Iterator of InternalRow.
 */
def apply(
 input: Iterator[InternalRow],
 key: Seq[Expression],
 sizeEstimate: Int = 64,
 taskMemoryManager: TaskMemoryManager = null): HashedRelation = {
 val mm = Option(taskMemoryManager).getOrElse {
 new TaskMemoryManager(
 new UnifiedMemoryManager(
 new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
 Long.MaxValue,
 Long.MaxValue / 2,
 1),
 0)
 }

 if (key.length == 1 && key.head.dataType == LongType) {
 LongHashedRelation(input, key, sizeEstimate, mm)
 } else {
 UnsafeHashedRelation(input, key, sizeEstimate, mm)
 }
}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org