You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ch...@apache.org on 2026/08/14 19:52:00 UTC
(spark) branch branch-4.x updated: [SPARK-58619][CORE][CONNECT] Assign SQLSTATE 08003 to the INVALID_HANDLE session sub-conditions
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 114d9d5ac51c [SPARK-58619][CORE][CONNECT] Assign SQLSTATE 08003 to the INVALID_HANDLE session sub-conditions
114d9d5ac51c is described below
commit 114d9d5ac51cbd3c4b8399fbf4772d9204a015ef
Author: Jiwon Park <jp...@outlook.kr>
AuthorDate: Sat Aug 15 03:51:33 2026 +0800
[SPARK-58619][CORE][CONNECT] Assign SQLSTATE 08003 to the INVALID_HANDLE session sub-conditions
### What changes were proposed in this pull request?
Per the review discussion below, this PR returns to the sub-condition level SQLSTATE override so the wire-visible condition names stay unchanged, and hardens it as a documented exception:
- `[CORE]`: `ErrorSubInfo` gains an optional `sqlState`, and `ErrorClassesJsonReader.getSqlState` resolves the sub-condition's SQLSTATE first, falling back to the condition's. A new `SparkThrowableSuite` invariant fails any sub-condition `sqlState` outside an explicit allowlist (currently the three `INVALID_HANDLE.SESSION_*` sub-conditions), and the error README documents the exception, discourages new overrides even within the same error class, and notes that a future server-side com [...]
- `[CONNECT]`: `INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` declare SQLSTATE `08003` (connection does not exist). Names, messages, and every other sub-condition are unchanged.
### Why are the changes needed?
These sub-conditions mean the server-side session backing a Connect client is gone, which `08003` describes, but they inherit `INVALID_HANDLE`'s generic `HY000`, so tools detecting dead connections by SQLSTATE class `08` cannot recognize them and the Connect JDBC driver hard-codes a remapping (raised in the SPARK-57933 review). Moving them to a class-08 condition is not viable: released Connect clients match the condition names for session invalidation and reattach re-execution, and n [...]
### Does this PR introduce _any_ user-facing change?
Yes. The SQLSTATE of the three conditions changes from `HY000` to `08003`; the condition names are unchanged. Noted in `sql-migration-guide.md`.
### How was this patch tested?
- `SparkThrowableSuite`: sub-first resolution incl. degenerate inputs (confirmed failing before the fix), the allowlist invariant (any sub-condition declaring its own SQLSTATE must be on the documented allowlist; verified to fail on an injected non-allowlisted override), pins for `SESSION_*` = `08003` and every other `INVALID_HANDLE` sub-condition = `HY000`, and the error-states validation extended to sub-condition SQLSTATEs.
- `SparkConnectSessionManagerSuite` asserts `getSqlState == "08003"` at the server throw sites and `SparkSessionE2ESuite` asserts the client-received exception carries `08003` end-to-end (both previously untested).
- `JdbcErrorUtilsSuite` re-run: the JDBC driver's existing hard-coded mapping stays consistent.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-fable-5)
Closes #57831 from j1wonpark/SPARK-58619-subclass-sqlstate.
Authored-by: Jiwon Park <jp...@outlook.kr>
Signed-off-by: Cheng Pan <ch...@apache.org>
(cherry picked from commit bb8b4d9afd3396eeab509e377380667d3a9d00bd)
Signed-off-by: Cheng Pan <ch...@apache.org>
---
common/utils/src/main/resources/error/README.md | 8 ++
.../src/main/resources/error/error-conditions.json | 9 ++-
.../org/apache/spark/ErrorClassesJSONReader.scala | 16 +++-
.../org/apache/spark/SparkThrowableSuite.scala | 89 +++++++++++++++++++++-
docs/_plugins/build-error-docs.py | 5 +-
docs/sql-migration-guide.md | 1 +
.../spark/sql/connect/SparkSessionE2ESuite.scala | 1 +
.../service/SparkConnectSessionManagerSuite.scala | 4 +
8 files changed, 124 insertions(+), 9 deletions(-)
diff --git a/common/utils/src/main/resources/error/README.md b/common/utils/src/main/resources/error/README.md
index 575e2ebad35a..7a628a418c7c 100644
--- a/common/utils/src/main/resources/error/README.md
+++ b/common/utils/src/main/resources/error/README.md
@@ -157,6 +157,14 @@ Spark prefers to re-use existing SQLSTATEs, preferably used by multiple vendors.
For extension Spark claims the `K**` sub-class range.
If a new class is needed it will also claim the `K0` class.
+Every error condition and its sub-conditions normally belong to a single error state: the
+condition declares the SQLSTATE and the sub-conditions inherit it. As a documented
+exception, a sub-condition may declare its own `sqlState` when regrouping would break
+released clients that match the condition name on the wire. The only such exception is
+`INVALID_HANDLE.SESSION_*`, pinned by a test in `SparkThrowableSuite`. Do not add
+overrides, even within the same error class; a future compatibility layer rewriting
+condition names per client version may remove the existing exception.
+
Internal errors should use the `XX` class. You can subdivide internal errors by component.
For example: The existing `XXKD0` is used for an internal analyzer error.
diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index 5fb700af3504..c00dfc2ef3ed 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -4226,17 +4226,20 @@
"SESSION_CHANGED" : {
"message" : [
"The existing Spark server driver instance has restarted. Please reconnect."
- ]
+ ],
+ "sqlState" : "08003"
},
"SESSION_CLOSED" : {
"message" : [
"Session was closed."
- ]
+ ],
+ "sqlState" : "08003"
},
"SESSION_NOT_FOUND" : {
"message" : [
"Session not found."
- ]
+ ],
+ "sqlState" : "08003"
}
},
"sqlState" : "HY000"
diff --git a/common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala b/common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala
index 18a47e7ee37a..55153425ef13 100644
--- a/common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala
+++ b/common/utils/src/main/scala/org/apache/spark/ErrorClassesJSONReader.scala
@@ -124,10 +124,15 @@ class ErrorClassesJsonReader(jsonFileURLs: Seq[URL]) {
}
def getSqlState(errorClass: String): String = {
- Option(errorClass)
- .flatMap(_.split('.').headOption)
- .flatMap(errorInfoMap.get)
- .flatMap(_.sqlState)
+ val errorClasses = Option(errorClass).map(_.split('.')).getOrElse(Array.empty[String])
+ val errorInfo = errorClasses.headOption.flatMap(errorInfoMap.get)
+ val subClassSqlState = errorClasses match {
+ case Array(_, subClass) =>
+ errorInfo.flatMap(_.subClass).flatMap(_.get(subClass)).flatMap(_.sqlState)
+ case _ => None
+ }
+ subClassSqlState
+ .orElse(errorInfo.flatMap(_.sqlState))
.orNull
}
@@ -192,10 +197,13 @@ private case class ErrorInfo(
*
* @param message Message format with optional placeholders (e.g. <parm>).
* The error message is constructed by concatenating the lines with newlines.
+ * @param sqlState SQLSTATE associated with this subclass. If absent, the subclass inherits
+ * the SQLSTATE of its main error class.
* @param breakingChangeInfo Additional metadata if the error is due to a breaking change.
*/
private case class ErrorSubInfo(
message: Seq[String],
+ sqlState: Option[String] = None,
breakingChangeInfo: Option[BreakingChangeInfo] = None) {
// For compatibility with multi-line error messages
@JsonIgnore
diff --git a/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala b/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
index 5fb6924383f0..785325001be0 100644
--- a/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
@@ -125,7 +125,9 @@ class SparkThrowableSuite extends SparkFunSuite {
errorClassesJson.openStream(), new TypeReference[Map[String, String]]() {})
val errorStates = mapper.readValue(
errorStatesJson.openStream(), new TypeReference[Map[String, ErrorStateInfo]]() {})
- val errorConditionStates = errorReader.errorInfoMap.values.toSeq.flatMap(_.sqlState).toSet
+ val errorConditionStates = errorReader.errorInfoMap.values.toSeq.flatMap { i =>
+ i.sqlState ++ i.subClass.getOrElse(Map.empty).values.flatMap(_.sqlState)
+ }.toSet
assert(Set("22012", "22003", "42601").subsetOf(errorStates.keySet))
assert(errorClasses.keySet.filter(!_.matches("[A-Z0-9]{2}")).isEmpty)
assert(errorStates.keySet.filter(!_.matches("[A-Z0-9]{5}")).isEmpty)
@@ -133,6 +135,27 @@ class SparkThrowableSuite extends SparkFunSuite {
assert(errorConditionStates.diff(errorStates.keySet).isEmpty)
}
+ test("Sub-condition SQLSTATE overrides are limited to the documented exceptions") {
+ // Sub-conditions inherit their condition's SQLSTATE. The only permitted overrides are
+ // the wire-compatibility exceptions documented in the error README's SQLSTATE section.
+ val allowedOverrides = Set(
+ "INVALID_HANDLE.SESSION_CHANGED",
+ "INVALID_HANDLE.SESSION_CLOSED",
+ "INVALID_HANDLE.SESSION_NOT_FOUND")
+ errorReader.errorInfoMap.foreach { case (condition, info) =>
+ info.subClass.getOrElse(Map.empty).foreach { case (sub, subInfo) =>
+ subInfo.sqlState.foreach { subState =>
+ val name = s"$condition.$sub"
+ assert(
+ allowedOverrides(name),
+ s"$name declares its own SQLSTATE ($subState). Sub-conditions inherit their " +
+ "condition's SQLSTATE; do not add new overrides. See the SQLSTATE section " +
+ "of the error README.")
+ }
+ }
+ }
+ }
+
test("Message invariants") {
val messageSeq = errorReader.errorInfoMap.values.toSeq.flatMap { i =>
Seq(i.message) ++ i.subClass.getOrElse(Map.empty).values.toSeq.map(_.message)
@@ -631,6 +654,70 @@ class SparkThrowableSuite extends SparkFunSuite {
}
}
+ test("sub-condition SQLSTATE overrides the main condition's SQLSTATE") {
+ withTempDir { dir =>
+ val json = new File(dir, "errors.json")
+ Files.writeString(
+ json.toPath,
+ """
+ |{
+ | "TEST_MAIN_STATE": {
+ | "message": [
+ | "Main message."
+ | ],
+ | "sqlState": "42000",
+ | "subClass": {
+ | "SUB_WITHOUT_STATE": {
+ | "message": [
+ | "Sub-condition without its own SQLSTATE."
+ | ]
+ | },
+ | "SUB_WITH_STATE": {
+ | "message": [
+ | "Sub-condition with its own SQLSTATE."
+ | ],
+ | "sqlState": "08003"
+ | }
+ | }
+ | }
+ |}
+ |""".stripMargin,
+ StandardCharsets.UTF_8)
+
+ val reader =
+ new ErrorClassesJsonReader(Seq(errorJsonFilePath.toUri.toURL, json.toURI.toURL))
+ // A sub-condition with its own SQLSTATE overrides the main condition's.
+ assert(reader.getSqlState("TEST_MAIN_STATE.SUB_WITH_STATE") == "08003")
+ // A sub-condition without its own SQLSTATE inherits the main condition's.
+ assert(reader.getSqlState("TEST_MAIN_STATE.SUB_WITHOUT_STATE") == "42000")
+ assert(reader.getSqlState("TEST_MAIN_STATE") == "42000")
+ // Degenerate inputs keep the pre-existing non-throwing behavior: anything that is not
+ // a known "MAIN.SUB" pair resolves to the main condition's SQLSTATE, or null.
+ assert(reader.getSqlState("TEST_MAIN_STATE.NON_EXISTENT_SUB") == "42000")
+ assert(reader.getSqlState("TEST_MAIN_STATE.SUB_WITH_STATE.EXTRA") == "42000")
+ assert(reader.getSqlState("NON_EXISTENT") == null)
+ assert(reader.getSqlState(null) == null)
+ }
+ }
+
+ test("INVALID_HANDLE session sub-conditions carry SQLSTATE 08003") {
+ // The session sub-conditions mean the server-side session backing a Connect client is
+ // gone (08003, connection does not exist); the wire-visible names are unchanged.
+ val sessionSubConditions = Seq("SESSION_CHANGED", "SESSION_CLOSED", "SESSION_NOT_FOUND")
+ sessionSubConditions.foreach { sub =>
+ assert(errorReader.getSqlState(s"INVALID_HANDLE.$sub") == "08003", sub)
+ }
+ // Every other sub-condition concerns a single operation on a healthy session and keeps
+ // the condition's SQLSTATE.
+ val otherSubConditions = errorReader
+ .errorInfoMap("INVALID_HANDLE").subClass.get.keys.toSeq.diff(sessionSubConditions)
+ assert(otherSubConditions.nonEmpty)
+ otherSubConditions.foreach { sub =>
+ assert(errorReader.getSqlState(s"INVALID_HANDLE.$sub") == "HY000", sub)
+ }
+ assert(errorReader.getSqlState("INVALID_HANDLE") == "HY000")
+ }
+
test("detect unused message parameters") {
checkError(
exception = intercept[SparkException] {
diff --git a/docs/_plugins/build-error-docs.py b/docs/_plugins/build-error-docs.py
index df6b9e3c0527..068ec0da50c9 100644
--- a/docs/_plugins/build-error-docs.py
+++ b/docs/_plugins/build-error-docs.py
@@ -89,7 +89,7 @@ def generate_doc_rows(condition_name, condition_details):
sub_condition_rows.append(
"""
<tr id="{anchor}">
- <td></td>
+ <td>{sql_state}</td>
<td class="error-sub-condition">
<span class="error-condition-name">
<code>
@@ -103,6 +103,9 @@ def generate_doc_rows(condition_name, condition_details):
"""
.format(
anchor=anchor_name(condition_name, sub_condition_name),
+ sql_state=(
+ condition_details["subClass"][sub_condition_name].get("sqlState", "")
+ ),
# See comment above for explanation of `<wbr />`.
sub_condition_name=sub_condition_name.replace("_", "<wbr />_"),
message=condition_details["subClass"][sub_condition_name]["message"],
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index efe999bb036c..4e3df0fd56c0 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -45,6 +45,7 @@ license: |
- Since Spark 4.3, `hash()` and `xxhash64()` include the `days` field of `CalendarInterval` when computing the hash, so their output for interval values differs from earlier releases. Previously the codegen path dropped `days`, disagreeing with interpreted evaluation.
- Since Spark 4.3, when a `SELECT` or `INSERT` statement references the same table more than once with different `WITH (...)` options (for example a self-join, or `INSERT INTO t WITH (...) SELECT * FROM t WITH (...)`), each reference now uses its own options instead of the second reference silently inheriting the first reference's options via the analyzer's relation cache.
- Since Spark 4.3, `HAVING` is evaluated before window functions when the `SELECT` list also contains generator functions such as `explode`. Previously, window functions could include groups removed by `HAVING` and produce incorrect results.
+- Since Spark 4.3, the Spark Connect session errors `INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` carry SQLSTATE `08003` instead of `HY000`; the condition names are unchanged. Code matching these errors on SQLSTATE should match `08003` or class `08`.
## Upgrading from Spark SQL 4.1 to 4.2
diff --git a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/SparkSessionE2ESuite.scala b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/SparkSessionE2ESuite.scala
index 9c149a858018..d8061386527b 100644
--- a/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/SparkSessionE2ESuite.scala
+++ b/sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/SparkSessionE2ESuite.scala
@@ -433,6 +433,7 @@ class SparkSessionE2ESuite extends ConnectFunSuite with RemoteSparkSession {
}
assert(e.getMessage.contains("[INVALID_HANDLE.SESSION_CHANGED]"))
+ assert(e.getSqlState == "08003")
assert(!session1.client.isSessionValid)
assert(SparkSession.getActiveSession.isEmpty)
assert(SparkSession.getDefaultSession.isEmpty)
diff --git a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManagerSuite.scala b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManagerSuite.scala
index 4b846631d7b7..5fae135bcf18 100644
--- a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManagerSuite.scala
+++ b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManagerSuite.scala
@@ -76,6 +76,7 @@ class SparkConnectSessionManagerSuite extends SharedSparkSession {
Some(sessionHolder.session.sessionUUID + "invalid"))
}
assert(exGet.getCondition == "INVALID_HANDLE.SESSION_CHANGED")
+ assert(exGet.getSqlState == "08003")
}
test(
@@ -89,11 +90,13 @@ class SparkConnectSessionManagerSuite extends SharedSparkSession {
SparkConnectService.sessionManager.getOrCreateIsolatedSession(key, None)
}
assert(exGetOrCreate.getCondition == "INVALID_HANDLE.SESSION_CLOSED")
+ assert(exGetOrCreate.getSqlState == "08003")
val exGet = intercept[SparkSQLException] {
SparkConnectService.sessionManager.getIsolatedSession(key, None)
}
assert(exGet.getCondition == "INVALID_HANDLE.SESSION_CLOSED")
+ assert(exGet.getSqlState == "08003")
val sessionGetIfPresent = SparkConnectService.sessionManager.getIsolatedSessionIfPresent(key)
assert(sessionGetIfPresent.isEmpty)
@@ -106,6 +109,7 @@ class SparkConnectSessionManagerSuite extends SharedSparkSession {
SparkConnectService.sessionManager.getIsolatedSession(key, None)
}
assert(exGet.getCondition == "INVALID_HANDLE.SESSION_NOT_FOUND")
+ assert(exGet.getSqlState == "08003")
val sessionGetIfPresent = SparkConnectService.sessionManager.getIsolatedSessionIfPresent(key)
assert(sessionGetIfPresent.isEmpty)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org