You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2026/08/31 12:14:34 UTC
[PR] [BUILD] Support Spark 4.3 [kyuubi]
pan3793 opened a new pull request, #7704:
URL: https://github.com/apache/kyuubi/pull/7704
### Why are the changes needed?
Spark 4.3.0 is currently in the RC voting phase. This prepares Kyuubi for Spark 4.3 and validates it against the RC artifacts.
### What changes are proposed?
- Adds the `spark-4.3` profile and `kyuubi-extension-spark-4-3` module.
- Adapts the Spark extension, Hive connector tests, and Arrow runtime for Spark 4.3 API changes.
- Adds Spark 4.3 CI, release, distribution, codecov, formatting, and documentation coverage.
- Lakehouse engines still fall back to Spark 4.1 artifacts until native Spark 4.3 support is available.
- The temporary Spark staging repository and binary archive reference `v4.3.0-rc1`; they must be updated if the RC changes or after the final release is published.
### How was this patch tested?
- `build/mvn test -pl extensions/spark/kyuubi-extension-spark-4-3 -am -Pspark-4.3 -Pscala-2.13`
- `build/mvn test -pl extensions/spark/kyuubi-spark-connector-hive -am -Pspark-4.3 -Pscala-2.13`
- `build/mvn test -pl extensions/spark/kyuubi-spark-authz,extensions/spark/kyuubi-spark-lineage -am -Pspark-4.3 -Pscala-2.13`
- `build/mvn test -pl externals/kyuubi-spark-sql-engine -Pspark-4.3 -Pscala-2.13`
- `./dev/reformat`
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Codex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support Spark 4.3 [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7704:
URL: https://github.com/apache/kyuubi/pull/7704#issuecomment-5522553279
I found a Spark 4.3 compatibility gap while testing this PR through Kyuubi JDBC on Kubernetes.
Open a session with:
```text
jdbc:hive2://<host>:10009/default;#spark.sql.timestampNanosTypes.enabled=true
```
Then run:
```sql
SELECT localtimestamp(9);
```
Spark finishes the query, but Beeline fails while fetching the result-set metadata:
```text
IllegalArgumentException: Unrecognized type name: timestamp_ntz(9)
at SchemaHelper.toTTypeId(SchemaHelper.scala:59)
```
`SchemaHelper` currently handles `TimestampType` and `TimestampNTZType`. Spark 4.3's `TimestampNTZNanosType` and `TimestampLTZNanosType` extend `AnyTimestampNanoType`, so they reach the fallback case.
Could we add handling and tests for both nanos types at precisions 7–9, including metadata precision and Thrift/Arrow result conversion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support Spark 4.3 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7704:
URL: https://github.com/apache/kyuubi/pull/7704#issuecomment-6209137259
Spark 4.3.0 RC2 fails, but it's not likely to see surprising changes until GA, let me merge this as-is. @wangzhigang1999 and you can start the new data type support work when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support Spark 4.3 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7704:
URL: https://github.com/apache/kyuubi/pull/7704#issuecomment-5543764176
@wangzhigang1999 I rebased and resolved conflicts, added some `TODO ... once Spark 4.3.0 GA` marks, this should be ready to go. please review and merge to unblock your work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support Spark 4.3 [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7704:
URL: https://github.com/apache/kyuubi/pull/7704#issuecomment-5524485126
I tested this PR at `a0c81c9c49677ce005993847554529cb9395af15` on ACK using Spark 4.3.0 RC1 and Scala 2.13. I focused on Kyuubi's Spark 4.3 compatibility paths; CI covers the unit tests.
| Area | Coverage | Result |
|---|---|---|
| Build and packaging | Spark 4.3 distribution and Kyuubi image | Passed |
| Spark SQL Engine | JDBC sessions and Kubernetes Driver/Executor | Passed |
| Spark 4.3 extension | Module loading, parser, dynamic shuffle, Watchdog, and Z-order smoke tests | Passed |
| Arrow compatibility | Spark 4.3 `toArrowSchema` signature, `none`/`zstd`, and nested types | Passed |
| Spark 4.3 SQL compatibility | ASOF JOIN, map DISTINCT, zero-length Parquet, column comment removal, and new `unix_*` inputs | Passed |
| Nanosecond timestamps | NTZ/LTZ precisions 7, 8, and 9 through JDBC and Arrow | **Failed** |
With `spark.sql.timestampNanosTypes.enabled=true`, all six nanosecond timestamp variants fail during Kyuubi result metadata conversion:
```text
Unrecognized type name: timestamp_{ntz|ltz}(7|8|9)
```
This was the only Kyuubi compatibility issue found in this ACK validation. Since nanosecond timestamps are disabled by default, this could either be fixed in this PR or documented as a known Spark 4.3 limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support Spark 4.3 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #7704: Support Spark 4.3
URL: https://github.com/apache/kyuubi/pull/7704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support Spark 4.3 [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7704:
URL: https://github.com/apache/kyuubi/pull/7704#issuecomment-5542864725
Thanks, that makes sense. I can work on a dedicated follow-up PR covering both `TimestampNTZNanosType` and `TimestampLTZNanosType` at precisions 7–9, including JDBC/Thrift metadata and Arrow result conversion.
I’ll also take a quick look at other recently added Spark data types and report any additional gaps separately, so the scope of the follow-up PR stays focused.
No need to block this PR or wait for Spark 4.3.0 GA from my side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support Spark 4.3 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7704:
URL: https://github.com/apache/kyuubi/pull/7704#issuecomment-5525918155
@wangzhigang1999, thanks for the additional testing. for `TimestampLTZNanosType`, I suggest adding support for it in a dedicated PR, and I guess we are also missing some other new data types that spark added recently.
if you want to work on this eagerly, we can merge this PR without waiting for Spark 4.3.0 GA, thought?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Support Spark 4.3 [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7704:
URL: https://github.com/apache/kyuubi/pull/7704#issuecomment-6209148725
**Merge Summary:**
- merged into master https://github.com/apache/kyuubi/commit/564627ef621c3cf18ac5be4f856f20fbd90699cb
*Posted by `merge_kyuubi_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org