(spark) branch branch-4.x updated: [SPARK-43253][SQL] Report unresolved MapObjects as an internal error

[ur...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 97cbf3d1ca74 [SPARK-43253][SQL] Report unresolved MapObjects as an internal error
97cbf3d1ca74 is described below

commit 97cbf3d1ca747d7d6f2c3e6f7c30d6e2c5a3ae83
Author: Deepak Jain (DJ) <de...@gmail.com>
AuthorDate: Mon Oct 12 08:50:04 2026 +0000

    [SPARK-43253][SQL] Report unresolved MapObjects as an internal error
    
    ### What changes were proposed in this pull request?
    
    Replace `_LEGACY_ERROR_TEMP_2017` with `SparkException.internalError` when `UnresolvedMapObjects.dataType` is requested without an explicit collection class. Remove the unused legacy error entry and exercise the owning expression in `ObjectExpressionsSuite`.
    
    ### Why are the changes needed?
    
    [SPARK-43253](https://issues.apache.org/jira/browse/SPARK-43253) asks for an internal error when the legacy condition cannot be reproduced from user SQL. `UnresolvedMapObjects` is an analyzer placeholder: its input must resolve before analysis replaces it with `MapObjects`. The new diagnostic describes this unresolved state instead of treating it as a user-facing unsupported operation.
    
    The regression calls the actual placeholder's `dataType`; a second assertion preserves the explicit collection-class path. This does not claim a user-SQL reproduction. Previous attempts are [#40984](https://github.com/apache/spark/pull/40984) and [#54900](https://github.com/apache/spark/pull/54900), both closed.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No expected change for valid analyzed expressions. An internal unresolved-expression access now reports `INTERNAL_ERROR` instead of `_LEGACY_ERROR_TEMP_2017`.
    
    ### How was this patch tested?
    
    Added a regression that fails against the original implementation with `SparkUnsupportedOperationException`, then passes with the internal-error factory. The same test verifies that an explicit collection class still returns its `ObjectType`.
    
    On Linux arm64 with Java 17, using the repository sbt launcher:
    
    - `catalyst/testOnly *ObjectExpressionsSuite`: 29 passed.
    - `core/testOnly *SparkThrowableSuite`: 38 passed, including error-catalog formatting and uniqueness.
    - `catalyst/scalastyle` and `catalyst/Test/scalastyle`: 0 errors and 0 warnings.
    - `git diff --check`: passed. Independent review found no remaining issue.
    
    The test JVM used `HEAP_SIZE=1g`; sbt used `-J-Xmx3072m -J-Xms512m -J-Xss4m`, two active processors, and serial task execution to fit the local container. Initial parallel compilation exceeded its memory limit. No build configuration is changed by this PR.
    
    All results above were repeated successfully after integrating current master `55bd2b643777918fe688e7bf498197e0c1ba17c6`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    **JIRA assignee for credit:** deepujain
    
    Closes #59028 from deepujain/fix/SPARK-43253-internal-error.
    
    Authored-by: Deepak Jain (DJ) <de...@gmail.com>
    Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
    (cherry picked from commit 13f7d1ecd08ca9750f4fd9e1307c616d48de1027)
    Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
 common/utils/src/main/resources/error/error-conditions.json |  5 -----
 .../org/apache/spark/sql/errors/QueryExecutionErrors.scala  |  5 +++--
 .../sql/catalyst/expressions/ObjectExpressionsSuite.scala   | 13 +++++++++++++
 3 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index c18e4a9988bb..04df7b966afb 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -11044,11 +11044,6 @@
       "Sinks cannot request distribution and ordering in continuous execution mode."
     ]
   },
-  "_LEGACY_ERROR_TEMP_2017" : {
-    "message" : [
-      "not resolved."
-    ]
-  },
   "_LEGACY_ERROR_TEMP_2026" : {
     "message" : [
       "Failed to convert value <value> (class of <cls>) with the type of <dataType> to JSON."
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index a230ca24cfc8..e066ed191c38 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -472,8 +472,9 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
     SparkException.internalError(s"Can not interpolate ${arg.getClass.getName} into code block.")
   }
 
-  def customCollectionClsNotResolvedError(): SparkUnsupportedOperationException = {
-    new SparkUnsupportedOperationException("_LEGACY_ERROR_TEMP_2017")
+  def customCollectionClsNotResolvedError(): SparkException = {
+    SparkException.internalError(
+      "Cannot determine the data type of an unresolved MapObjects without a collection class.")
   }
 
   def classUnsupportedByMapObjectsError(cls: Class[_]): SparkRuntimeException = {
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
index f41f3e7df1b5..697cd398db63 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ObjectExpressionsSuite.scala
@@ -108,6 +108,19 @@ class ObjectExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
     )
   }
 
+  test("SPARK-43253: unresolved map objects have no inferred data type") {
+    val input = Literal.create(Seq(1), ArrayType(IntegerType, containsNull = false))
+    val unresolved = UnresolvedMapObjects(identity, input)
+    checkError(
+      exception = intercept[SparkException] { unresolved.dataType },
+      condition = "INTERNAL_ERROR",
+      parameters = Map("message" ->
+        "Cannot determine the data type of an unresolved MapObjects without a collection class."))
+    val collectionClass = classOf[java.util.ArrayList[_]]
+    assert(unresolved.copy(customCollectionCls = Some(collectionClass)).dataType ==
+      ObjectType(collectionClass))
+  }
+
   test("MapObjects should make copies of unsafe-backed data") {
     // test UnsafeRow-backed data
     val structEncoder = ExpressionEncoder[Array[Tuple2[java.lang.Integer, java.lang.Integer]]]()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org