You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/04/11 18:39:00 UTC
[jira] [Commented] (SPARK-27431) move HashedRelation to global
UnifiedMemoryManager and enable offheap
[ https://issues.apache.org/jira/browse/SPARK-27431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815699#comment-16815699 ]
Hyukjin Kwon commented on SPARK-27431:
--------------------------------------
Sounds like a question. If you're not sure, ask it to mailing list first before filing as an issue.
> move HashedRelation to global UnifiedMemoryManager and enable offheap
> ---------------------------------------------------------------------
>
> Key: SPARK-27431
> URL: https://issues.apache.org/jira/browse/SPARK-27431
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Xiaoju Wu
> Priority: Major
>
> Why is HashedRelation currently managed by a newly created MemoryManager and disabled with offheap? Can we improve this part?
> /**
> * Create a HashedRelation from an Iterator of InternalRow.
> */
> def apply(
> input: Iterator[InternalRow],
> key: Seq[Expression],
> sizeEstimate: Int = 64,
> taskMemoryManager: TaskMemoryManager = null): HashedRelation = {
> val mm = Option(taskMemoryManager).getOrElse {
> new TaskMemoryManager(
> new UnifiedMemoryManager(
> new SparkConf().set(MEMORY_OFFHEAP_ENABLED.key, "false"),
> Long.MaxValue,
> Long.MaxValue / 2,
> 1),
> 0)
> }
> if (key.length == 1 && key.head.dataType == LongType) {
> LongHashedRelation(input, key, sizeEstimate, mm)
> } else {
> UnsafeHashedRelation(input, key, sizeEstimate, mm)
> }
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org