You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/08/25 22:36:27 UTC

[PR] fix(storage): resolve HoodieStorage from a path instead of a default local URI [hudi]

yihua opened a new pull request, #19737:
URL: https://github.com/apache/hudi/pull/19737

### Describe the issue this Pull Request addresses

closes #19736

Stacked on #19735, review that one first. Until it merges this PR's diff shows its commit too.

### Summary and Changelog

`HoodieStorageUtils.getStorage(StorageConfiguration)` forwarded to a `"file:///"` default URI, so it always returned storage bound to the local filesystem regardless of the configuration passed in, while its signature read as "build storage from this configuration".

`BatchedBlobReader` used it to build executor-side storage and then read `blob.reference.external_path` through the result, so any out-of-line blob reference on `s3a` or `gs` failed with `Wrong FS ... expected: file:///`. It now holds the configuration and resolves storage per referenced filesystem, cached by scheme and authority and closed at task completion, since a partition's blob references only become known as its rows arrive and one partition may reference more than one filesystem.

With that call site gone the overload has no production callers, so it and `DEFAULT_URI` move to `HoodieTestUtils`. Every remaining `HoodieStorageUtils` factory takes a path. The non-local-scheme filesystem harness added by #19735 moves from `TestRollbackHelperV1` to `hudi-hadoop-common` test utils so the blob reader tests can use it too.

### Impact

Out-of-line blob reads on object storage work; they previously failed unconditionally. `BatchedBlobReader`'s constructor now takes `StorageConfiguration` instead of `HoodieStorage`; the `BatchedBlobReader` object's public `readBatched` and `processRDD` entry points are unchanged. `HoodieStorageUtils.getStorage(StorageConfiguration)` and `HoodieStorageUtils.DEFAULT_URI` are removed from `hudi-common` main; both were only reachable from test code after this change.

### Risk Level

low

Verified red-then-green: reverting `HoodieStorageUtils` and `BatchedBlobReader` while keeping the new tests fails all three non-local-scheme tests with `IllegalArgumentException: Wrong FS: s3a://..., expected: file:///`, the production symptom.

215 tests green on JDK 11 across the affected suites, including `TestBatchedBlobReader` (19, four new), `TestBatchedBlobReaderMerge` (10), `TestRollbackHelperV1` (5), `TestRollbackHelper` (13), `TestFSUtils` (96), `TestEightToSevenDowngradeHandler` (10), `TestHoodieIncrSource` (21) and the remaining repointed suites. Checkstyle and scalastyle clean.

`TestCommitsCommand` fails on this base with `Failed to construct HoodieInstant: The provided file name 102_<ts>.commit does not conform to the required format`. That reproduces identically at the base commit with these changes stashed, so it is pre-existing and unrelated.

The new tests expose the local filesystem under `s3a` and `gs` rather than needing a remote object store, so they run anywhere. `RawLocalFileSystem.pathToFile` keeps only the path component of a URI, so the only thing the subclass changes is the identity the filesystem reports, which is what `FileSystem.checkPath` validates against. One test covers two filesystems in a single partition, which a resolve-once fix would fail, and one asserts the borrowed schemes really reach local files so the others cannot pass for the wrong reason.

### Documentation Update

none

### Contributor's checklist

- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org