You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xushiyan (via GitHub)" <gi...@apache.org> on 2026/05/01 23:54:48 UTC
[I] Stream-style get_file_slices for large-table scalability [hudi-rs]
xushiyan opened a new issue, #590:
URL: https://github.com/apache/hudi-rs/issues/590
## Summary
`Table::get_file_slices` is fully eager today: it materializes the complete `Vec<FileSlice>` before returning. For tables with thousands of partitions or many file groups per partition, this produces noticeable startup latency and a large in-memory snapshot before any reads can begin. A streaming form would let engines start dispatching read tasks as slices are discovered.
## Current state
`Table::get_file_slices(options)` → `get_file_slices_internal(...)` → `FileSystemView::get_file_slices(...)` returns `Vec<FileSlice>` after:
1. Listing all partitions (filesystem walk OR metadata table query).
2. For each partition, listing all files.
3. Grouping files into file groups + slices.
4. Applying pruning (partition + file stats).
Every layer is eager. Engines that want bucketing call `hudi_core::util::collection::split_into_chunks` on the materialized vec — which still requires the full vec to be in memory first.
## Proposed shape
Add a streaming companion:
```rust
pub async fn get_file_slices_stream(&self, options: &ReadOptions)
-> Result<BoxStream<'static, Result<FileSlice>>>
```
Per-partition granularity is the natural cut: stream partitions, materialize slices for each partition (still cheap — file groups per partition are bounded), apply pruning inline, emit. Partitions are the wide axis; per-partition slice materialization is fine.
## Layers that need streaming variants
| Layer | Today | Streamable form |
| ------------------------------ | ------------------------- | ---------------------------------------------------------------- |
| Object store listing | `list()` returns `Vec<Path>` | `list_stream()` returns `Stream<Item = Path>` (object_store crate already supports this) |
| Partition discovery | Build all paths upfront | Yield partitions as listed |
| File listing per partition | List all files | Yield per partition (cheap to buffer per-partition) |
| Slice grouping | Group all files of a partition | Group within a partition; still buffer per-partition |
| Pruning | Apply to materialized vec | Apply per slice as yielded |
| Metadata table backend | Returns full result set | Stream rows from metadata table queries |
## Out of scope
- Per-slice streaming within a partition. The wide axis is partitions; slices per partition are bounded.
- Predicate pushdown changes — current pruning logic just runs inline in the stream.
## Use cases
- Large tables with thousands of partitions where startup latency on `get_file_slices` is observable.
- Engines that want pipelined planning (Ray Data, Spark with pipelined plans) that can dispatch task work before full planning completes.
- Memory-bounded environments where holding a 100k-slice vec is expensive.
## Notes
- The metadata-table-backed path needs the most refactor: `MetadataTable::get_file_slices` currently reads from another Hudi table; making it stream means streaming that read.
- Dispatch on `ReadOptions::query_type` as `get_file_slices` already does.
- Tests need to cover: streaming order, error propagation per-partition, empty-table handling, MDT-on/MDT-off parity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Stream-style get_file_slices for large-table scalability [hudi-rs]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #590:
URL: https://github.com/apache/hudi-rs/issues/590#issuecomment-5460194441
@xushiyan Implemented this feature in #701.Also, metadata-table records and the incremental touched-group lookup remain eager and are called out in the changes. Please let me know if you have any feedback. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org