You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/05/28 04:59:46 UTC
[PR] Truncate Parquet page statistics [arrow-rs]
etseidl opened a new pull request, #7555:
URL: https://github.com/apache/arrow-rs/pull/7555
# Which issue does this PR close?
Enables workaround for #7489
# Rationale for this change
When `WriterProperties::statistics_truncate_length` is set, the column chunk statistics are truncated, but the page statistics are not. This can lead to very large page headers that blow up some readers.
# What changes are included in this PR?
Page statistics are now truncated as well.
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page statistics [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2918695904
Aah yes sorry, read things too quickly, carry on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2914928310
Leaving as draft until tests can be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page data page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2992674939
`label_issue.py` automatically added labels {'next-major-release'} from #7594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#discussion_r2118165341
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -3766,4 +3769,67 @@ mod tests {
.unwrap();
assert_eq!(batches.len(), 0);
}
+
+ #[test]
+ fn test_page_stats_truncation() {
+ let string_field = Field::new("a", DataType::Utf8, false);
+ let binary_field = Field::new("b", DataType::Binary, false);
+ let schema = Schema::new(vec![string_field, binary_field]);
+
+ let raw_string_values = vec!["Blart Versenwald III"];
+ let raw_binary_values = [b"Blart Versenwald III".to_vec()];
+ let raw_binary_value_refs = raw_binary_values
+ .iter()
+ .map(|x| x.as_slice())
+ .collect::<Vec<_>>();
+
+ let string_values = StringArray::from(raw_string_values.clone());
+ let binary_values = BinaryArray::from(raw_binary_value_refs);
+ let batch = RecordBatch::try_new(
+ Arc::new(schema),
+ vec![Arc::new(string_values), Arc::new(binary_values)],
+ )
+ .unwrap();
+
+ let props = WriterProperties::builder()
+ .set_statistics_truncate_length(Some(2))
+ .set_dictionary_enabled(false)
+ .set_encoding(Encoding::PLAIN)
+ .set_compression(crate::basic::Compression::UNCOMPRESSED)
+ .build();
+
+ let mut file = roundtrip_opts(&batch, props);
+
+ // read file and decode page headers
Review Comment:
It took me a while to realize there are statistics embedded in the page headers that there is no corresponding rust API to read. I think a comment like this would help
```suggestion
// read file and decode page headers
// Note: use the thrift API as there is no Rust API to access the statistics in the page headers
```
##########
parquet/src/column/writer/mod.rs:
##########
@@ -949,6 +949,58 @@ impl<'a, E: ColumnValueEncoder> GenericColumnWriter<'a, E> {
.unwrap_or_else(|| (data.to_vec(), false))
}
+ // Truncate the min and max values in a page or column chunk Statistics
Review Comment:
Pedantic suggestion:
```suggestion
/// Truncate the min and max values that will be written to a data page
/// header or column chunk Statistics
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page data page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page statistics [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2917894757
> This functionality I think already exists under a different option? - https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_column_index_truncate_length
AFAICT that option is solely for the min and max in the column index. But I will verify that. This PR is for the statistics embedded in the page header. I believe it was these stats blowing up the arrow-cpp reader in the linked issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2925590025
BTW I am pretty sure @adriangb was showing me some data the other day with relatively large metadata. I think this PR will substantially reduce the size of the metadata for his files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page data page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2925688300
BTW I also filed a ticket to discuss a way to disable writing the redundant copy of statistics to the data page header statistics entirely:
- https://github.com/apache/arrow-rs/issues/7580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page data page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2992674855
`label_issue.py` automatically added labels {'enhancement'} from #7594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2923463270
I am doing some tests of this PR but I am likely to run out of time today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page statistics [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2917799704
This functionality I think already exists under a different option? - https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterPropertiesBuilder.html#method.set_column_index_truncate_length
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page data page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2929912531
I took the liberty of merging up from main to resolve some conflicts and tweaked the docs a bit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Truncate Parquet page data page statistics [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7555:
URL: https://github.com/apache/arrow-rs/pull/7555#issuecomment-2934808278
🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org