You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2018/11/16 16:25:00 UTC
[jira] [Updated] (SPARK-24985) Executing SQL with "Full Outer Join"
on top of large tables when there is data skew met OOM
[ https://issues.apache.org/jira/browse/SPARK-24985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated SPARK-24985:
------------------------------
Priority: Major (was: Critical)
> Executing SQL with "Full Outer Join" on top of large tables when there is data skew met OOM
> -------------------------------------------------------------------------------------------
>
> Key: SPARK-24985
> URL: https://issues.apache.org/jira/browse/SPARK-24985
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: sheperd huang
> Priority: Major
>
> When we run SQL with "Full Outer Join" on large tables when there is data skew, we found it's quite easy to hit OOM. We once thought we hit https://issues.apache.org/jira/browse/SPARK-13450. But taking a look at fix in [https://github.com/apache/spark/pull/16909,] we found that PR hasn't handled the "Full Outer Join" case.
> The root cause of the OOM is there are a lot of rows with the same key.
> See below code:
> {code:java}
> private def findMatchingRows(matchingKey: InternalRow): Unit = {
> leftMatches.clear()
> rightMatches.clear()
> leftIndex = 0
> rightIndex = 0
> while (leftRowKey != null && keyOrdering.compare(leftRowKey, matchingKey) == 0) {
> leftMatches += leftRow.copy()
> advancedLeft()
> }
> while (rightRowKey != null && keyOrdering.compare(rightRowKey, matchingKey) == 0) {
> rightMatches += rightRow.copy()
> advancedRight()
> }
> {code}
> It seems we haven't limited the data added to leftMatches and rightMatches.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org