(spark) branch master updated: [SPARK-52724][SQL] Enhance broadcast join OOM error handling with SHUFFLE_MERGE hint support

[yu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yumwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new a5a3b7ce2209 [SPARK-52724][SQL] Enhance broadcast join OOM error handling with SHUFFLE_MERGE hint support
a5a3b7ce2209 is described below

commit a5a3b7ce2209d84c062a915e108f678243904437
Author: emmazhang <zr...@163.com>
AuthorDate: Sat Jul 12 01:08:57 2025 +0800

    [SPARK-52724][SQL] Enhance broadcast join OOM error handling with SHUFFLE_MERGE hint support
    
    ### What changes were proposed in this pull request?
    
    This PR enhances broadcast join OOM error handling by use shuffle sort merge join(`SHUFFLE_MERGE`) hint as a workaround for broadcast join OOM issues.
    
    ### Why are the changes needed?
    
    Reduce support workload by involving customers to diagnose and resolve broadcast join OOM issues independently.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Manual test:
    ```
    Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value or analyze these tables through: `ANALYZE TABLE `t1` COMPUTE STATISTICS; ANALYZE TABLE `t2` COMPUTE STATISTICS;` or apply the shuffle sort merge join hint as described in the Spark documentation: https://spark.apache.org/docs/la [...]
    
    Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value or apply the shuffle sort merge join hint as described in the Spark documentation: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html#join-hints.
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #51417 from Emma-82/SPARK-52724.
    
    Authored-by: emmazhang <zr...@163.com>
    Signed-off-by: Yuming Wang <yu...@ebay.com>
---
 common/utils/src/main/resources/error/error-conditions.json           | 2 +-
 .../main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index dc3b3c6aea55..f504a4ef6e4a 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -8688,7 +8688,7 @@
   },
   "_LEGACY_ERROR_TEMP_2250" : {
     "message" : [
-      "Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting <autoBroadcastJoinThreshold> to -1 or increase the spark driver memory by setting <driverMemory> to a higher value<analyzeTblMsg>"
+      "Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting <autoBroadcastJoinThreshold> to -1 or increase the spark driver memory by setting <driverMemory> to a higher value<analyzeTblMsg> or apply the shuffle sort merge join hint as described in the Spark documentation: https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-hints.html#join-hints."
     ]
   },
   "_LEGACY_ERROR_TEMP_2251" : {
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index f1f7e9681f6c..680d7b4b172d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -2106,9 +2106,9 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
       oe: OutOfMemoryError, tables: Seq[TableIdentifier]): Throwable = {
     val analyzeTblMsg = if (tables.nonEmpty) {
       " or analyze these tables through: " +
-        s"${tables.map(t => s"ANALYZE TABLE $t COMPUTE STATISTICS;").mkString(" ")}."
+        s"`${tables.map(t => s"ANALYZE TABLE $t COMPUTE STATISTICS;").mkString(" ")}`"
     } else {
-      "."
+      ""
     }
     new SparkException(
       errorClass = "_LEGACY_ERROR_TEMP_2250",


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org