You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "michaelmitchell-bit (via GitHub)" <gi...@apache.org> on 2026/06/28 19:49:59 UTC
[PR] [SPARK-57745][SQL] Assign named error class for unsupported datatype [spark]
michaelmitchell-bit opened a new pull request, #56857:
URL: https://github.com/apache/spark/pull/56857
### What changes were proposed in this pull request?
This PR replaces the legacy error class `_LEGACY_ERROR_TEMP_3192` with the existing named error condition `UNSUPPORTED_DATATYPE` in vectorized row/column conversion paths. It also removes the legacy error condition entry and adds regression coverage for the migrated error class.
### Why are the changes needed?
`_LEGACY_ERROR_TEMP_3192` is one of the temporary legacy error classes tracked by SPARK-57745. Reusing `UNSUPPORTED_DATATYPE` gives this unsupported datatype path a stable named error condition and SQLSTATE.
### Does this PR introduce any user-facing change?
No. This only changes the internal error class name used for an existing unsupported datatype error path.
### How was this patch tested?
- `jq empty common/utils/src/main/resources/error/error-conditions.json`
- `git diff --check`
- Verified no `_LEGACY_ERROR_TEMP_3192` references remain in the touched error/vectorized files
The targeted Spark SQL test was not run locally because this machine only has Java 8 installed, while current Spark requires Java 17/21/25 to build.
### License
This contribution is my original work and I license it to the project under the project's open source license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57745][SQL] Assign named error class for unsupported datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56857:
URL: https://github.com/apache/spark/pull/56857#discussion_r3489686453
##########
sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java:
##########
@@ -237,7 +237,7 @@ private static void appendValue(WritableColumnVector dst, DataType t, Object o)
dst.appendLong(DateTimeUtils.localDateTimeToMicros((LocalDateTime) o));
} else {
throw new SparkUnsupportedOperationException(
- "_LEGACY_ERROR_TEMP_3192", Map.of("dt", t.toString()));
+ "UNSUPPORTED_DATATYPE", Map.of("typeName", t.toString()));
Review Comment:
`typeName` here (and at the two `MutableColumnarRow` sites) is `t.toString()`, which renders the type in its unquoted internal form — e.g. `Unsupported data type DateType` / `ArrayType(...)` / `ObjectType(...)`.
Every other `UNSUPPORTED_DATATYPE` call site formats `typeName` as the double-quoted SQL form via `toSQLType` (`= quoteByDefault(dt.sql)`), e.g. `Unsupported data type "DATE"` (see `JsonFunctionsSuite`, `CsvFunctionsSuite`, `JDBCWriteSuite`, `ColumnTypeSuite`). Since the goal here is a *stable named* condition, please use `toSQLType` to match that contract rather than the raw `toString` (which just carried over from `_LEGACY_ERROR_TEMP_3192`).
Please apply `DataTypeErrors.toSQLType(...)` at all three sites, and update the new test to assert the `toSQLType` form instead of `objectType.toString`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57745][SQL] Assign named error class for unsupported datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56857:
URL: https://github.com/apache/spark/pull/56857#issuecomment-4835733066
+1, LGTM. Merging to master/4.x.
Thank you, @michaelmitchell-bit and @uros-b for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57745][SQL] Assign named error class for unsupported datatype [spark]
Posted by "michaelmitchell-bit (via GitHub)" <gi...@apache.org>.
michaelmitchell-bit commented on PR #56857:
URL: https://github.com/apache/spark/pull/56857#issuecomment-4835703427
Thanks @MaxGekk and @uros-b. I updated the three migrated sites to format `typeName` with `toSQLType`, updated the new assertions to expect the SQL-formatted type name, added direct coverage for `MutableColumnarRow.get` and `MutableColumnarRow.update`, and updated the PR description to call out the SparkThrowable-facing error-class / SQLSTATE / message change.
Retested with:
`./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryExecutionErrorsSuite -- -z SPARK-57745"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57745][SQL] Assign named error class for unsupported datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56857:
URL: https://github.com/apache/spark/pull/56857#discussion_r3489686453
##########
sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java:
##########
@@ -237,7 +237,7 @@ private static void appendValue(WritableColumnVector dst, DataType t, Object o)
dst.appendLong(DateTimeUtils.localDateTimeToMicros((LocalDateTime) o));
} else {
throw new SparkUnsupportedOperationException(
- "_LEGACY_ERROR_TEMP_3192", Map.of("dt", t.toString()));
+ "UNSUPPORTED_DATATYPE", Map.of("typeName", t.toString()));
Review Comment:
`typeName` here (and at the two `MutableColumnarRow` sites) is `t.toString()`, which renders the type in its unquoted internal form — e.g. `Unsupported data type DateType` / `ArrayType(...)` / `ObjectType(...)`.
Every other `UNSUPPORTED_DATATYPE` call site formats `typeName` as the double-quoted SQL form via `toSQLType` (`= quoteByDefault(dt.sql)`), e.g. `Unsupported data type "DATE"` (see `JsonFunctionsSuite`, `CsvFunctionsSuite`, `JDBCWriteSuite`, `ColumnTypeSuite`). Since the goal here is a *stable named* condition, it'd be good to match that contract rather than the raw `toString` (which just carried over from `_LEGACY_ERROR_TEMP_3192`).
Suggest formatting via the SQL form — e.g. `DataTypeErrors.toSQLType(t)` (or `"\"" + t.sql() + "\""`) — at all three sites, and updating the new test to assert the quoted form instead of `objectType.toString`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57745][SQL] Assign named error class for unsupported datatype [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #56857:
URL: https://github.com/apache/spark/pull/56857#issuecomment-4833918941
Another minor note about the PR description: the "No user-facing change" claim is inaccurate on the SparkThrowable surface: error-class name changes to UNSUPPORTED_DATATYPE, SQLSTATE goes from absent (legacy entry had no sqlState) to 0A000, and message text changes from Datatype not supported <dt> to Unsupported data type <typeName>.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57745][SQL] Assign named error class for unsupported datatype [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #56857: [SPARK-57745][SQL] Assign named error class for unsupported datatype
URL: https://github.com/apache/spark/pull/56857
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org