You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "byronwang123 (via GitHub)" <gi...@apache.org> on 2026/08/14 19:33:14 UTC
[I] [FEATURE] [kyuubi]
byronwang123 opened a new issue, #7635:
URL: https://github.com/apache/kyuubi/issues/7635
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Arrow-based query results are currently transferred as uncompressed Arrow IPC
bytes. The result fetch dominates query time on bandwidth-limited links
(60-97% of total on a cross-subnet cluster), so compressing the IPC body
would cut both bytes and wall-clock. This carries on #3877 (deferred in 2022
waiting for an upstream approach); Spark 4.1.0 has since added
spark.sql.execution.arrow.compression.codec / .zstd.level, so the
upstream-aligned way now exists.
### Motivation
On TPC-DS SF10 (nntest cluster): cs_numeric 1.17GB -> 247MB, fetch 19.3s -> 7.5s;
cs_full 4.67GB -> 850MB, fetch 205.8s -> 94.1s.
### Describe the solution
Reuse the upstream Spark config keys (getConfString on Spark < 4.1), compress
the Arrow IPC body with zstd (default none, zero behavior change), and let the
JDBC client decompress via CompressionCodec.Factory from the batch body
compression metadata -- no Kyuubi-specific wire protocol or config key.
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] Support Zstd compression for Arrow IPC query results [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #7635: [FEATURE] Support Zstd compression for Arrow IPC query results
URL: https://github.com/apache/kyuubi/issues/7635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [FEATURE] [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7635:
URL: https://github.com/apache/kyuubi/issues/7635#issuecomment-5297371725
Hello @byronwang123,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org