You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "ericyangliu (via GitHub)" <gi...@apache.org> on 2026/06/15 22:31:49 UTC
[I] [Bug] Spark VARIANT result metadata fails with "Unrecognized type name: variant" [kyuubi]
ericyangliu opened a new issue, #7515:
URL: https://github.com/apache/kyuubi/issues/7515
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct).
### Search before asking
- [x] I have searched the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Kyuubi fails when returning result set metadata for a Spark SQL result containing a `VARIANT` column.
The SQL operation itself completes successfully, and row fetching can succeed, but the REST metadata endpoint fails when Kyuubi converts the Spark result schema to HiveServer2/Thrift metadata.
The failure comes from:
```text
org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTypeId(SchemaHelper.scala:58)
java.lang.IllegalArgumentException: Unrecognized type name: variant
```
`SchemaHelper.toTTypeId` handles common Spark SQL types but does not appear to handle Spark 4 `VariantType`. Spark 4.0+ supports `VariantType`, and `parse_json(...)` returns a `VARIANT`.
### Steps to reproduce
Run Kyuubi with a Spark 4.x engine, then execute a query that returns a `VARIANT` column:
```sql
SELECT parse_json('{"a":1,"b":0.8}') AS v;
```
Then request result set metadata for the completed operation, for example through the Kyuubi Web UI or REST API:
```text
GET /api/v1/operations/<operationHandle>/resultsetmetadata
```
### Expected behavior
Kyuubi should return result set metadata successfully.
If HiveServer2/Thrift has no native `VARIANT` type, Kyuubi should expose Spark `VariantType` as a compatible metadata type, such as `STRING_TYPE` or another appropriate fallback, instead of failing the metadata request.
### Actual behavior
The query reaches `FINISHED_STATE`, but metadata retrieval fails:
```text
Error getting result set metadata for operation handle <operationHandle>
org.apache.kyuubi.KyuubiSQLException: Unrecognized type name: variant
Caused by: java.lang.IllegalArgumentException: Unrecognized type name: variant
at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTypeId(SchemaHelper.scala:58)
at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTypeDesc(SchemaHelper.scala:77)
at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTColumnDesc(SchemaHelper.scala:87)
at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTableSchema(SchemaHelper.scala:95)
```
### Affects Version(s)
Observed on Kyuubi `1.11.1` with Spark `4.1.1`.
The same `SchemaHelper.toTTypeId` mapping appears to be present on current `master`, so `master` may also be affected.
### Kyuubi Server Log Output
```logtalk
Processing query[<operationHandle>]: RUNNING_STATE -> FINISHED_STATE
GET /api/v1/operations/<operationHandle>/rowset ... status=200
ERROR org.apache.kyuubi.server.api.ApiUtils:
Error getting result set metadata for operation handle <operationHandle>
org.apache.kyuubi.KyuubiSQLException: Unrecognized type name: variant
Caused by: java.lang.IllegalArgumentException: Unrecognized type name: variant
at org.apache.kyuubi.engine.spark.schema.SchemaHelper$.toTTypeId(SchemaHelper.scala:58)
```
### Kyuubi Engine Log Output
No Spark query execution failure was observed. The query completed successfully; the failure occurs during Kyuubi result set metadata conversion.
### Kyuubi Server Configurations
```yaml
kyuubi.engine.type: SPARK_SQL
kyuubi.frontend.protocols: THRIFT_BINARY,REST
```
### Kyuubi Engine Configurations
The issue should reproduce without extra engine configuration using:
```sql
SELECT parse_json('{"a":1,"b":0.8}') AS v;
```
### Additional context
Spark 4.0 introduced `VariantType`. Kyuubi already stringifies non-primitive row values in rowset generation, but result metadata conversion currently throws before clients can render metadata for a result containing `VARIANT`.
A likely fix is to add Spark `VariantType` handling in Kyuubi's Spark schema conversion path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Spark VARIANT result metadata fails with "Unrecognized type name: variant" [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7515:
URL: https://github.com/apache/kyuubi/issues/7515#issuecomment-4713007632
Hello @ericyangliu,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Spark VARIANT result metadata fails with "Unrecognized type name: variant" [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #7515: [Bug] Spark VARIANT result metadata fails with "Unrecognized type name: variant"
URL: https://github.com/apache/kyuubi/issues/7515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org