You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "suryaprasanna (via GitHub)" <gi...@apache.org> on 2026/03/11 23:57:56 UTC
[I] Support for Hudi on Velox Engine [hudi]
suryaprasanna opened a new issue, #18308:
URL: https://github.com/apache/hudi/issues/18308
### Feature Description
**What the feature achieves:**
This feature would enable Apache Hudi table format support in Meta's Velox execution engine, allowing Velox-based query engines (Presto, Gluten, etc.) to read and potentially write Hudi tables natively.
Key capabilities:
- Native C++ reader for Hudi Copy-on-Write (COW) tables
- Native C++ reader for Hudi Merge-on-Read (MOR) tables with real-time view support
- Integration with Velox's vectorized execution model for high-performance data processing
- Support for Hudi's time travel and incremental queries
- Compatibility with Hudi's metadata table for efficient file pruning
- Support for Hudi's column statistics and data skipping
**Why this feature is needed:**
Velox is a high-performance C++ vectorized execution engine that powers multiple query engines including Meta's Presto, Apache Spark (via Gluten), and other data processing systems. Currently:
- **No native Hudi support**: Velox lacks native support for reading Hudi tables, limiting its ability to process Hudi datasets efficiently
- **Performance opportunity**: Velox's C++ vectorized engine can provide significant performance improvements (minimum 10x in some cases) compared to JVM-based readers
- **Growing adoption**: Velox is being adopted by major projects (Presto, Gluten for Spark, Meta's infrastructure) and Hudi support would enable these ecosystems
- **Modern lake house architecture**: As Hudi becomes a standard lakehouse format, Velox integration is essential for high-performance query engines
- **Reduced dependency on JVM**: Native C++ implementation would enable non-JVM systems to consume Hudi tables efficiently
### User Experience
**How users will use this feature:**
- Configuration changes needed
Velox connector configuration to recognize Hudi table format
- API changes
Velox Connector API to register Hudi file format
- Usage examples
All the read queries remain the same no change.
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (Yes/No)
- Does this change storage format? (Yes/No)
- Justification:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4043643762
Gluten already has some Hudi support (CoW). We plan to build on `hudi-rs` a native scanner for MoR.
@suryaprasanna are you talking about velox (single node) or gluten/prestissimo (distributed spark/presto)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "jja725 (via GitHub)" <gi...@apache.org>.
jja725 commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4683307269
Either 1 or 3. I feel 3 would be more intuitive like iceberg C++ client @vinothchandar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4043512303
+1, do we want to maintain this feature as a separate project or integrate with the Hudi repo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4043694158
It'd be great if we can trim the issue desc to essentials.. and capture these requirements tightly there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "jaystarshot (via GitHub)" <gi...@apache.org>.
jaystarshot commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4112749139
@vinothchandar At Uber, our primary focus is on Presto/Velox (Prestissimo). Since we operate in a multi-query environment, we have concerns about adopting hudi-rs, particularly around memory management.
1) Is it used in production in any other companies?
2) In Prestissimo, memory allocation is tightly controlled at the per-query level in C++, with enforced limits. If new memory is allocated in Rust (hudi-rs), it may not integrate with our existing memory management model, potentially leading to issues such as uncontrolled memory usage or increased risk of worker OOMs.
3) How is the data transfer handled between hudi-rs and velox, is it 0 copy? If it uses velox ABI is velox ABI stable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4042981577
CC @prashantwason @shangxinli.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4677642530
> The fix there was ultimately to bring the reader inside Velox's allocation umbrella
this is same as 1 ? This was something we were also discussing internally cc @rmahindra
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4454975485
hey @jaystarshot for 3), yes it's zero-copy, hudi-rs rust core exposes Arrow's RecordBatch via [Arrow C stream interface](https://arrow.apache.org/docs/format/CStreamInterface.html) using a raw pointer. See the api in this block https://github.com/apache/hudi-rs/blob/0910ca7dbadd763a873d97ba9c0d1a7842dedc1b/cpp/src/lib.rs#L29
cc @codope @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "suryaprasanna (via GitHub)" <gi...@apache.org>.
suryaprasanna commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4100551049
CC @shangxinli
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "jja725 (via GitHub)" <gi...@apache.org>.
jja725 commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4664593343
Hi @xushiyan , nice to see you again in the community.
Memory accounting across the FFI boundary. Even if RecordBatch exchange is zero-copy, the buffer allocations happen outside Velox's MemoryPool. Velox's memory arbitrator can't see them, so it won't throttle, spill, or reclaim against them. From Velox's accounting perspective the operator looks empty while real RSS keeps climbing — and the first thing to notice is the OS via OOM kill, not Velox.
We hit exactly this with the early arrow-parquet reader integration in Velox a few years back. The fix there was ultimately to bring the reader inside Velox's allocation umbrella (which is how the native Parquet reader exists today). Anything routed through arrow-rs / object_store / Rust's global allocator will reproduce the same shape unless the design accounts for it up front.
A few directions worth considering before locking in the design:
1. Custom allocator on the Rust side. arrow-rs Buffer supports custom allocations — hudi-rs could optionally allocate through a Velox-supplied allocator (Velox's MemoryPool* exposed as a C ABI: alloc(size) -> void*, free(ptr, size)). Adds complexity but makes accounting exact.
2. Memory ticket / bounded reservation. Velox reserves N bytes in its pool before each next() call; the FFI returns batches sized to fit; the reservation is released when the RowVector is destroyed. Approximate but enforceable, and survives backpressure scenarios.
3. C++ native hudi client so the memory allocation is controlled by Velox natively
Whichever direction the community goes, getting this into the design doc before code lands will save a lot of pain — retrofitting allocator hooks into a working Rust pipeline is much harder than designing for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support for Hudi on Velox Engine [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on issue #18308:
URL: https://github.com/apache/hudi/issues/18308#issuecomment-4084638988
@rmahindra for your 👀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org