You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wan Kun (Jira)" <ji...@apache.org> on 2022/09/01 15:03:00 UTC
[jira] [Updated] (SPARK-40306) Support more than Integer.MAX_VALUE of the same join key
[ https://issues.apache.org/jira/browse/SPARK-40306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wan Kun updated SPARK-40306:
----------------------------
Description:
For SMJ, the number of the same join key records of the right table is greater than Integer.MAX_VALUE, the result will be incorrect.
Before SMJ JOIN, we will put the records of the same join key into the ExternalAppendOnlyUnsafeRowArray. ExternalAppendOnlyUnsafeRowArray.numRows overflow may cause OOM. During SMJ JOIN, SpillableArrayIterator.startIndex overflow may cause incorrect result.
For example, one of our production table:
!image-2022-09-01-23-02-15-955.png!
was:
For SMJ, the number of the same join key records of the right table is greater than Integer.MAX_VALUE, the result will be incorrect.
Before SMJ JOIN, we will put the records of the same join key into the ExternalAppendOnlyUnsafeRowArray. ExternalAppendOnlyUnsafeRowArray.numRows overflow may cause OOM. During SMJ JOIN, SpillableArrayIterator.startIndex overflow may cause incorrect result.
For example, one of our production table:
!image-2022-09-01-23-01-46-282.png!
> Support more than Integer.MAX_VALUE of the same join key
> --------------------------------------------------------
>
> Key: SPARK-40306
> URL: https://issues.apache.org/jira/browse/SPARK-40306
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Wan Kun
> Priority: Major
> Attachments: image-2022-09-01-23-02-15-955.png
>
>
> For SMJ, the number of the same join key records of the right table is greater than Integer.MAX_VALUE, the result will be incorrect.
> Before SMJ JOIN, we will put the records of the same join key into the ExternalAppendOnlyUnsafeRowArray. ExternalAppendOnlyUnsafeRowArray.numRows overflow may cause OOM. During SMJ JOIN, SpillableArrayIterator.startIndex overflow may cause incorrect result.
> For example, one of our production table:
> !image-2022-09-01-23-02-15-955.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org