[jira] [Resolved] (SPARK-27431) move HashedRelation to global UnifiedMemoryManager and enable offheap

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-27431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-27431.
----------------------------------
    Resolution: Invalid

> move HashedRelation to global UnifiedMemoryManager and enable offheap
> ---------------------------------------------------------------------
>
>                 Key: SPARK-27431
>                 URL: https://issues.apache.org/jira/browse/SPARK-27431
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.3.0
>            Reporter: Xiaoju Wu
>            Priority: Major
>
> Why is HashedRelation currently managed by a newly created MemoryManager and disabled with offheap? Can we improve this part?
> /**
>  * Create a HashedRelation from an Iterator of InternalRow.
>  */
> def apply(
>  input: Iterator[InternalRow],
>  key: Seq[Expression],
>  sizeEstimate: Int = 64,
>  taskMemoryManager: TaskMemoryManager = null): HashedRelation = {
>  val mm = Option(taskMemoryManager).getOrElse {
>  new TaskMemoryManager(
>  new UnifiedMemoryManager(
>  new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
>  Long.MaxValue,
>  Long.MaxValue / 2,
>  1),
>  0)
>  }
>  if (key.length == 1 && key.head.dataType == LongType) {
>  LongHashedRelation(input, key, sizeEstimate, mm)
>  } else {
>  UnsafeHashedRelation(input, key, sizeEstimate, mm)
>  }
> }



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org