You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2026/02/16 01:18:49 UTC
[PR] HG-3394: Cache FileStatus in Footer to reduce redundant NameNode RPC calls [parquet-java]
wangyum opened a new pull request, #3395:
URL: https://github.com/apache/parquet-java/pull/3395
### Rationale for this change
When reading Parquet files from HDFS, `getFileStatus()` is called twice for each file:
1. During footer reading in `ParquetFileReader.readAllFootersInParallel()`
2. During split generation in `ParquetInputFormat.getSplits()`
This creates redundant NameNode RPC calls. For workloads processing thousands of files, this redundancy significantly increases NameNode load and job startup time.
This PR caches `FileStatus` in the `Footer` object to eliminate redundant RPC calls, reducing NameNode RPC calls during Parquet file processing.
### What changes are included in this PR?
1. **`Footer.java`**: Added `FileStatus` field with backward-compatible constructors
2. **`ParquetFileReader.java`**: Pass `FileStatus` when creating `Footer` objects
3. **`ParquetInputFormat.java`**: Reuse cached `FileStatus` instead of calling `fs.getFileStatus()` again
4. **`TestFooterFileStatusCaching.java`**: New test suite with 5 tests proving RPC reduction
### Are these changes tested?
**Yes.** Added comprehensive test suite `TestFooterFileStatusCaching` with 5 test cases:
- ✅ Footer stores and returns FileStatus correctly
- ✅ ParquetFileReader passes FileStatus to Footer
- ✅ Cached FileStatus is reused (saves 3 RPCs in test)
- ✅ Complete workflow verification (saves 5 RPCs in test)
- ✅ Backward compatibility verified
### Are there any user-facing changes?
No.
Closes #3394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3394: Cache FileStatus in Footer to reduce redundant NameNode RPC calls [parquet-java]
Posted by "dossett (via GitHub)" <gi...@apache.org>.
dossett commented on PR #3395:
URL: https://github.com/apache/parquet-java/pull/3395#issuecomment-4568282139
@steveloughran Your idea about passing length down is also very interesting (after codex helped me understand it). I'd be happy to pick that up as a follow-on to this when it's merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3394: Cache FileStatus in Footer to reduce redundant NameNode RPC calls [parquet-java]
Posted by "dossett (via GitHub)" <gi...@apache.org>.
dossett commented on code in PR #3395:
URL: https://github.com/apache/parquet-java/pull/3395#discussion_r3320570191
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java:
##########
Review Comment:
@wangyum Should this be updated to use the new constructor and pass in `footer.getFileStatus()` ? I think that's needed to actually optimize the `ParquetInputFormat.getSplits()` call since `getFooters` uses the cache wrapper construct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] HG-3394: Cache FileStatus in Footer to reduce redundant NameNode RPC calls [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3395:
URL: https://github.com/apache/parquet-java/pull/3395#issuecomment-3956608850
@steveloughran What do you think of this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] HG-3394: Cache FileStatus in Footer to reduce redundant NameNode RPC calls [parquet-java]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #3395:
URL: https://github.com/apache/parquet-java/pull/3395#issuecomment-3905946915
cc @wgtmac
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3394: Cache FileStatus in Footer to reduce redundant NameNode RPC calls [parquet-java]
Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #3395:
URL: https://github.com/apache/parquet-java/pull/3395#issuecomment-4268040711
this saves a lot of IO against cloud storage, and 100ms per footer.
it could be even better if we just relied on file length being passed down, as iceberg stores that in its metadata -if it can make its way all the way down then a lot of time and money is ultimately saved.
that'd need extensions to the public API, but passing down just the length here would prepare for those future changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3394: Cache FileStatus in Footer to reduce redundant NameNode RPC calls [parquet-java]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum closed pull request #3395: GH-3394: Cache FileStatus in Footer to reduce redundant NameNode RPC calls
URL: https://github.com/apache/parquet-java/pull/3395
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org