You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "QuakeWang (via GitHub)" <gi...@apache.org> on 2026/07/07 13:28:49 UTC
[PR] feat: write value stats for append data files [paimon-rust]
QuakeWang opened a new pull request, #477:
URL: https://github.com/apache/paimon-rust/pull/477
<!--
Thank you very much for contributing to Paimon Rust - we are happy that you want to help us improve it. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/apache/paimon-rust/issues). Exceptions are made for typos in documentation or comments, which need no issue.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `cargo test` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- Linking this pull request to the issue -->
Rust append writes currently emit empty `_VALUE_STATS`, so manifest-level stats pruning cannot skip data files written by Rust even when min/max/null-count information is available.
<!-- What is the purpose of the change -->
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
- Accumulate min/max/null-count value stats across batches in `DataFileWriter`.
- Persist stats in existing `DataFileMeta.value_stats` and dense `value_stats_cols`.
- Skip unsupported variable-length value stats and add nanosecond timestamp extraction support.
- Add tests for value stats writing, timestamp(9), skipped variable-length columns, and scan pruning.
### Tests
<!-- List unit tests or integration cases to verify this change -->
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets --workspace --features fulltext,vortex,mosaic -- -D warnings`
- `cargo test -p paimon --all-targets --features fulltext,vortex,mosaic`
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature or require documentation updates -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#issuecomment-4954639816
The dedicated bloom/vector table will incorrectly crop data. A normal Parquet file only writes some fields, but 'writable' equals' None '; When the statistics cover all 'normal fields',' value_dats_coms' is also 'None'. The reading end therefore performs position mapping according to the complete table schema. Tested the schema '[payload (blob), a, b]' and data 'a=100, b=0', filtered the statistics that 'a>50' would cause misreading of 'b', and cut the file to 'finalization files=0'. The Java implementation will write 'normalColumnNames' to' writable Cols'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#issuecomment-4954642534
`normalize_insert_batch_to_table_schema` prioritizes name and position for each column, and will reuse the source column. The target '[a, b]' and source '[b, x]' are converted to '[b, b]' instead of obtaining '[b, x]' based on the positional semantics of 'Insert ROW/*'. Temporary regression testing failed to stabilize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#issuecomment-4915073424
> In Java Side, the performance of the accumulator is poor, and the only truly useful link is the one that extracts stats from Parquet. Perhaps we should optimize its implementation instead of the accumulator.
@JingsongLi Thanks, addressed.
I changed the implementation to avoid per-row stats accumulation. Append value stats are now extracted from Parquet writer metadata on close, aligned with the Java ParquetSimpleStatsExtractor path.
Added/updated tests for append value stats, stats pruning, and timestamp stats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#discussion_r3548519605
##########
crates/paimon/src/table/data_file_writer.rs:
##########
@@ -247,23 +252,19 @@ impl DataFileWriter {
file_source: Option<i32>,
first_row_id: Option<i64>,
write_cols: Option<Vec<String>>,
+ write_fields: &[DataField],
+ parquet_metadata: Option<&parquet::file::metadata::ParquetMetaData>,
) -> DataFileMeta {
+ let (value_stats, value_stats_cols) =
+ crate::arrow::format::parquet::extract_value_stats(parquet_metadata, write_fields);
Review Comment:
Just return stats from parquet? It is not good to expose `ParquetMetaData` in core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#discussion_r3548523750
##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -625,6 +631,168 @@ fn build_row_group_column_indices(
.collect()
}
+pub(crate) fn extract_value_stats(
Review Comment:
We need to support default stats mode: truncate(16), otherwise the stats are too big.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#issuecomment-4923680619
Thanks for the update. I think there are still a couple of Java compatibility gaps here.
1. The manifest stats mode is currently hard-coded to the default truncate(16) behavior. In Java, stats collectors are created from `metadata.stats-mode`, `fields.<name>.stats-mode`, and `metadata.stats-keep-first-n-columns`. With this implementation, a table configured with `metadata.stats-mode = none/counts/full` or a field-level override would still get truncate(16)-style stats in Rust-written manifests. Could we thread the table stats options into this extraction path and apply the same collector semantics as Java?
2. `supports_manifest_value_stats` currently skips unsupported min/max types completely. Java does not do that: for unsupported Parquet min/max types such as BINARY/VARBINARY, `ParquetSimpleStatsExtractor` still returns `SimpleColStats(null, null, nullCount)`, and dense stats storage keeps the column when the null count is known. The current Rust behavior drops those columns from `value_stats_cols`, so their null counts are lost. Also, Java supports TIME_WITHOUT_TIME_ZONE through IntStatistics, while Rust's conversion code has Time support but the support list excludes `DataType::Time`. Could we keep null counts for unsupported min/max columns and include Time to match Java?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #477:
URL: https://github.com/apache/paimon-rust/pull/477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#discussion_r3556692453
##########
crates/paimon/src/arrow/format/parquet.rs:
##########
@@ -87,11 +108,23 @@ impl ParquetFormatWriter {
schema: arrow_schema::SchemaRef,
compression: &str,
zstd_level: i32,
+ write_fields: Option<&[DataField]>,
+ format_options: &HashMap<String, String>,
) -> crate::Result<Self> {
let async_write = output.async_writer().await?;
let codec = parse_compression(compression, zstd_level);
let inner = create_parquet_arrow_writer(async_write, schema, codec)?;
- Ok(Self { inner })
+ let stats_modes = write_fields
+ .map(|fields| {
+ CoreOptions::new(format_options)
Review Comment:
Parsing the per-column stats modes is not sufficient here because Java also passes `metadata.stats-dense-store` (default `true`) into `SimpleStatsConverter`. Rust currently ignores that option and `extract_value_stats` always returns dense, filtered stats with `Some(column_names)`. With `metadata.stats-dense-store=false`, Java requires full-width stats and `valueStatsCols = null`; with dense storage enabled, Java also uses `null` when all write columns are present. Please thread this flag through and add compatibility tests. A local one-column `counts` regression with dense store disabled currently returns `Some(["id"])` instead of `None`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: write value stats for append data files [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on PR #477:
URL: https://github.com/apache/paimon-rust/pull/477#issuecomment-4921492114
@JingsongLi Addressed, thanks. Conflicts are resolved, ParquetMetaData is no longer exposed to core, and string value stats now use default truncate(16) with test coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org