You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by "vishramachandran (via GitHub)" <gi...@apache.org> on 2026/01/04 16:10:01 UTC
[D] How to enable Gzip or Zstd compression on Java based Arrow Flight Service [arrow]
GitHub user vishramachandran created a discussion: How to enable Gzip or Zstd compression on Java based Arrow Flight Service
I have gotten my JVM based service to work with Arrow Flight, but need to enable compression.
The FlightServer.Builder and FlightClient.builder constructs don't expose any API for adding compression.
Is this possible? Can I get some pointers?
Generative AI tools are hallucinating on this question with spurious methods to add interceptors and compression on the builders. I don't see them on the Java Arrow Flight API.
GitHub link: https://github.com/apache/arrow/discussions/48720
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] How to enable Gzip or Zstd compression on Java based Arrow Flight Service [arrow]
Posted by "vishramachandran (via GitHub)" <gi...@apache.org>.
GitHub user vishramachandran closed the discussion with a comment: How to enable Gzip or Zstd compression on Java based Arrow Flight Service
Got an answer with cross-post in arrow-java https://github.com/apache/arrow-java/discussions/944
Thank you for the help!
GitHub link: https://github.com/apache/arrow/discussions/48720#discussioncomment-15419858
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] How to enable Gzip or Zstd compression on Java based Arrow Flight Service [arrow]
Posted by "xborder (via GitHub)" <gi...@apache.org>.
GitHub user xborder added a comment to the discussion: How to enable Gzip or Zstd compression on Java based Arrow Flight Service
Are you referring to gRPC-level compression or IPC compression?
- gRPC-level: It looks doable. FlightServer.Builder uses [NettyServerBuilder](https://grpc.github.io/grpc-java/javadoc/io/grpc/netty/NettyServerBuilder.html), which you can extend to configure transport compression via compressorRegistry() (outgoing) and decompressorRegistry() (incoming). Likewise, FlightClient.Builder creates a [NettyChannelBuilder](https://github.com/apache/arrow-java/blob/8d3fe93d493188f0d3df59028cfd852be2085820/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightClient.java#L827) that supports the same APIs. So there’s a plausible path here.
- IPC compression: I experimented with this a bit in the context of https://github.com/apache/arrow-java/pull/937. [VectorUnloader](https://github.com/apache/arrow-java/blob/b9e40fa0a90143eef36ca53d2f937f2fef494402/vector/src/main/java/org/apache/arrow/vector/VectorUnloader.java#L62) already appears to support data compression. What seems to be missing is initializing OutboundStreamListenerImpl with a codec. I did a quick [tested](https://github.com/xborder/arrow-java/commit/5489d106d9a10289b94edeefa3066acb3edab6f8#diff-a715f331d1a7c6178b02e73b0f8c3232e8dd5707c927df5de94ebc9479b5e5d0) that seemed to work, but it needs a more thorough review.
GitHub link: https://github.com/apache/arrow/discussions/48720#discussioncomment-15414913
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] How to enable Gzip or Zstd compression on Java based Arrow Flight Service [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
GitHub user kou closed the discussion with a comment: How to enable Gzip or Zstd compression on Java based Arrow Flight Service
Let's use https://github.com/apache/arrow-java/discussions/944 instead of here.
GitHub link: https://github.com/apache/arrow/discussions/48720#discussioncomment-15419346
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] How to enable Gzip or Zstd compression on Java based Arrow Flight Service [arrow]
Posted by "xborder (via GitHub)" <gi...@apache.org>.
GitHub user xborder edited a comment on the discussion: How to enable Gzip or Zstd compression on Java based Arrow Flight Service
Are you referring to gRPC-level compression or IPC compression?
- gRPC-level: It looks doable. FlightServer.Builder uses [NettyServerBuilder](https://grpc.github.io/grpc-java/javadoc/io/grpc/netty/NettyServerBuilder.html), which you can extend to configure transport compression via compressorRegistry() (outgoing) and decompressorRegistry() (incoming). Likewise, FlightClient.Builder creates a [NettyChannelBuilder](https://github.com/apache/arrow-java/blob/8d3fe93d493188f0d3df59028cfd852be2085820/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightClient.java#L827) that supports the same APIs. So there’s a plausible path here.
- IPC compression: I experimented with this a bit in the context of https://github.com/apache/arrow-java/pull/937. [VectorUnloader](https://github.com/apache/arrow-java/blob/b9e40fa0a90143eef36ca53d2f937f2fef494402/vector/src/main/java/org/apache/arrow/vector/VectorUnloader.java#L62) already appears to support data compression. What seems to be missing is initializing OutboundStreamListenerImpl with a codec. I did a quick [tested](https://github.com/xborder/arrow-java/commit/5489d106d9a10289b94edeefa3066acb3edab6f8#diff-a715f331d1a7c6178b02e73b0f8c3232e8dd5707c927df5de94ebc9479b5e5d0) that seemed to work, but it needs a more thorough review.
BTW, java implementation of arrow was moved to [arrow-java](https://github.com/apache/arrow-java)
GitHub link: https://github.com/apache/arrow/discussions/48720#discussioncomment-15414913
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] How to enable Gzip or Zstd compression on Java based Arrow Flight Service [arrow]
Posted by "vishramachandran (via GitHub)" <gi...@apache.org>.
GitHub user vishramachandran closed a discussion: How to enable Gzip or Zstd compression on Java based Arrow Flight Service
I have gotten my JVM based service to work with Arrow Flight, but need to enable compression.
The FlightServer.Builder and FlightClient.builder constructs don't expose any API for adding compression.
Is this possible? Can I get some pointers?
Generative AI tools are hallucinating on this question with spurious methods to add interceptors and compression on the builders. I don't see them on the Java Arrow Flight API.
GitHub link: https://github.com/apache/arrow/discussions/48720
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org