You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zml1206 (via GitHub)" <gi...@apache.org> on 2025/09/04 03:21:15 UTC
[PR] [SPARK-53483] Support disable code-gen for sort merge join other than FULL OUTER and Existence [spark]
zml1206 opened a new pull request, #52227:
URL: https://github.com/apache/spark/pull/52227
### What changes were proposed in this pull request?
Support disable code-gen for sort merge join other than FULL OUTER and Existence. In addition, adjust the default value of spark.sql.sortMergeJoinExec.buffer.in.memory.threshold to be consistent with the BUFFER_IN_MEMORY_THRESHOLD of other operators, which is 4096.
### Why are the changes needed?
Avoid executor OOM when a single key matches too many rows in inner/left outer/ right outer sort merge join beccause of [currentRows](https://github.com/apache/spark/blob/e3133f4abf1cd5667abe5f0d05fa0af0df3033ae/sql/core/src/main/java/org/apache/spark/sql/execution/BufferedRowIterator.java#L34).
Before pr, we can only increase the executor memory. After pr, we can disable code-gen for sort merge join.
### Does this PR introduce _any_ user-facing change?
Yes, new configuration `spark.sql.codegen.join.SortMergeJoin.enabled`.
### How was this patch tested?
Local test, before pr will OOM, after pr run successfully.
```
withSQLConf(SQLConf.ENABLE_SORT_MERGE_JOIN_CODEGEN.key -> "false",
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
val df1 = spark.range(1).map(_ => ("testkey", "testvalue1")).toDF("key", "value")
val df2 = spark.range(50000000).map(_ => ("testkey", "testvalue2")).toDF("key", "value")
df1.join(df2, Seq("key"), "left").show()
}
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53483][SQL] Support disable code-gen for sort merge join other than FULL OUTER and Existence [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #52227:
URL: https://github.com/apache/spark/pull/52227#issuecomment-3649981083
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53483][SQL] Support disable code-gen for sort merge join other than FULL OUTER and Existence [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #52227: [SPARK-53483][SQL] Support disable code-gen for sort merge join other than FULL OUTER and Existence
URL: https://github.com/apache/spark/pull/52227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org