You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "majian1998 (via GitHub)" <gi...@apache.org> on 2026/03/20 04:07:14 UTC
[PR] [core] Extract Parquet stats from in-memory footer instead of re-reading file [paimon]
majian1998 opened a new pull request, #7489:
URL: https://github.com/apache/paimon/pull/7489
### Purpose
Closes #7467.
Currently, Paimon extracts Parquet column statistics by re-reading the file footer after the writer is closed. On object stores like OSS, a newly closed file may not be immediately visible for reading due to eventual consistency, causing `FileNotFoundException`.
This PR caches the `ParquetMetadata` in memory during writer close (via `ParquetWriter.getFooter()`), and passes it through the stats extraction pipeline to avoid re-reading the file. When the in-memory metadata is unavailable (e.g. non-Parquet formats), it falls back to the original file-based extraction path.
### Tests
- `ParquetInMemoryStatsTest#testInMemoryStatsMatchFileStats` — verifies in-memory stats match file-based stats across multiple data types
- `ParquetInMemoryStatsTest#testInMemoryStatsFallbackWhenMetadataIsNull` — verifies null metadata falls back to file reading
- `ParquetInMemoryStatsTest#testWriterMetadataDefaultIsNull` — verifies default `FormatWriter` returns null metadata
### API and Format
No. All new interface methods use default implementations with null/fallback behavior. No breaking changes to public APIs or storage format.
### Documentation
No.
### Generative AI tooling
Generated-by: Claude Opus 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Extract Parquet stats from in-memory footer instead of re-reading file [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #7489: [core] Extract Parquet stats from in-memory footer instead of re-reading file
URL: https://github.com/apache/paimon/pull/7489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Extract Parquet stats from in-memory footer instead of re-reading file [paimon]
Posted by "majian1998 (via GitHub)" <gi...@apache.org>.
majian1998 commented on PR #7489:
URL: https://github.com/apache/paimon/pull/7489#issuecomment-4166940760
@JingsongLi Hi, could you help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Extract Parquet stats from in-memory footer instead of re-reading file [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7489:
URL: https://github.com/apache/paimon/pull/7489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org