You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/07/09 06:06:09 UTC
[I] [SUBTASK] Port trinodb/trino#29842 split-inflation fix (blockSize=0 in HudiTrinoStorage) into hudi-trino [hudi]
voonhous opened a new issue, #19231:
URL: https://github.com/apache/hudi/issues/19231
Subtask of #18780 (RFC-105 hudi-trino migration).
### What
Port the fix for trinodb/trino#29842 into the Hudi-side connector module, so the migrated `hudi-trino` artifact does not re-ship the bug.
`HudiTrinoStorage.convertToPathInfo` (Hudi side: `hudi-trino/src/main/java/io/trino/plugin/hudi/storage/HudiTrinoStorage.java`, same code as Trino-side `TrinoHudiStorage`) hardcodes `blockSize = 0` when converting a `FileEntry` to `StoragePathInfo`:
```java
return new StoragePathInfo(
convertToPath(fileEntry.location()),
fileEntry.length(),
false,
(short) 0,
0, // <- block size hardcoded to 0
fileEntry.lastModified().toEpochMilli());
```
Downstream, `HudiReadOptimizedDirectoryLister.listStatus` computes `max(blockSize, min(fileLength, MIN_BLOCK_SIZE))`, which collapses to the 32MB `MIN_BLOCK_SIZE` fallback, and `HudiSplitFactory` then slices every large file at 32MB boundaries. On S3 (no real block boundaries) a ~120MB parquet file yields 4 splits instead of 1; trinodb/trino#29842 measured ~4x split inflation (5,834 -> 23,411 splits on a ~6000-file table) with compounding physical-IO overhead from extra GetObject calls and redundant footer reads.
### Fix (per the upstream issue)
- Pass `fileEntry.length()` as the block size in `convertToPathInfo` (1 split per file on object stores).
- The `max/min` guard and `MIN_BLOCK_SIZE` in the directory lister then simplify to `fileEntry.getLength()` and the constant can be removed.
- Add/port a regression test asserting one split per large file.
### Why here
Under the RFC-105 ownership flip, connector internals (including this class) live in `org.apache.hudi:hudi-trino`; fixes reported against the Trino-side plugin must land in this repo to reach users of the shim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Port trinodb/trino#29842 split-inflation fix (blockSize=0 in HudiTrinoStorage) into hudi-trino [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on issue #19231:
URL: https://github.com/apache/hudi/issues/19231#issuecomment-5164533965
A note on scope after digging into the migrated module: `HudiReadOptimizedDirectoryLister` and `MIN_BLOCK_SIZE` never made it into `hudi-trino`. They were dropped in the RFC-105 migration (#18837, c3c936790727) when the connector was rewritten around `FileSlice` with `HudiSnapshotDirectoryLister` and the filegroup reader (MOR support), so the fix site this issue describes does not exist in this repo. The class only survives as prose in rfc-105.md.
The blockSize=0 root cause did get migrated though. `HudiTrinoStorage.convertToPathInfo` (and `getPathInfo`) still hardcode 0, and the sole consumer today is `HudiSplitFactory`, which sizes base file splits with `max(target_split_size, blockSize)`. With blockSize always 0 (direct listing) or 32MB (metadata table listing via `getDefaultBlockSize`), that floor is inert and slicing is governed by `hudi.target-split-size` (default 128MB). So the 4x inflation from trinodb/trino#29842 does not reproduce here: a ~120MB file already yields one split, and only files above ~141MB (128MB x 1.1 split slop) get sliced, which is essentially upstream's post-fix behavior (they cap at a configurable 120MB).
Re-scoping the port accordingly:
- `HudiTrinoStorage` reports the file length as the block size (upstream parity at the storage layer).
- `HudiSplitFactory` slices solely by `target_split_size`. Porting blockSize=length while keeping the old `max()` would have silently disabled the knob and made every file a single split; dropping the `max()` keeps the config honest and preserves scan parallelism on large base files. Anyone wanting upstream's strict one-split-per-file behavior can raise `target_split_size`.
- Regression tests: one split for a ~120MB file at the default target, target-boundary slicing above it, and blockSize == length assertions on the storage layer.
PR coming up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Port trinodb/trino#29842 split-inflation fix (blockSize=0 in HudiTrinoStorage) into hudi-trino [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous closed issue #19231: Port trinodb/trino#29842 split-inflation fix (blockSize=0 in HudiTrinoStorage) into hudi-trino
URL: https://github.com/apache/hudi/issues/19231
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org