You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "DeviousCardi (via GitHub)" <gi...@apache.org> on 2026/09/15 08:46:52 UTC
[PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
DeviousCardi opened a new pull request, #11092:
URL: https://github.com/apache/arrow-rs/pull/11092
## Summary
Fixes #11073.
Writing a Decimal-logical-type column backed by an Arrow `BinaryArray`/`ByteArray` through `ArrowColumnWriter` produced wrong min/max column statistics, because the statistics computation used plain unsigned lexicographic `Ord` comparison on the raw two's-complement-encoded bytes. The direct `parquet::file::writer` column-writer path (and the `FixedSizeBinaryArray` path) already handled this correctly via a signed, decimal-aware comparator.
## Changes
- `parquet/src/column/writer/mod.rs`: widened `compare_greater_byte_array_decimals` to `pub(crate)`; added a shared `is_decimal_descr()` helper (used by both this file's `compare_greater` and the Arrow writer) to avoid duplicating the decimal-type-detection check.
- `parquet/src/arrow/arrow_writer/byte_array.rs`: `ByteArrayEncoder` now detects decimal columns at construction and uses the signed comparator for both the running min/max computation and the cross-batch min/max merge.
- `truncate_statistics`/`can_truncate_value` (`column/writer/mod.rs`): `BYTE_ARRAY` decimal columns are now excluded from statistics truncation, matching the existing guard already in place for `FIXED_LEN_BYTE_ARRAY` decimals — truncating a decimal's two's-complement bytes with unsigned `increment()` could otherwise reintroduce a corrupted bound for long decimals.
- Two regression tests added, based on the issue's repro and a truncation-specific case.
## Test plan
- [x] `cargo test -p parquet --lib arrow_writer` — 166 passed
- [x] `cargo test -p parquet --lib column::writer` — 112 passed
- [x] `cargo clippy -p parquet --lib --all-targets` — clean
- [x] `cargo build -p parquet` — clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#discussion_r4022321992
##########
parquet/src/arrow/arrow_writer/byte_array.rs:
##########
@@ -800,12 +823,30 @@ fn count_within_budget_offsets<T: ByteArrayType>(
n
}
+/// Returns `true` if `a > b`.
+///
+/// `BYTE_ARRAY` columns logically typed as `DECIMAL` store values as
+/// two's-complement, big-endian bytes, so they must be compared with
+/// [`compare_greater_byte_array_decimals`] rather than plain unsigned
+/// byte-wise `Ord`, or negative values would sort as the largest values.
+/// This mirrors the comparator `compare_greater` uses in the non-Arrow
+/// column writer path (`crate::column::writer`).
Review Comment:
```suggestion
```
Again, I think this level of explanation is unnecessary
##########
parquet/src/arrow/arrow_writer/byte_array.rs:
##########
@@ -432,6 +433,15 @@ pub struct ByteArrayEncoder {
bloom_filter: Option<Sbbf>,
bloom_filter_target_fpp: f64,
geo_stats_accumulator: Option<Box<dyn GeoStatsAccumulator>>,
+ /// Whether this column is a `BYTE_ARRAY` logically typed as `DECIMAL`.
+ ///
+ /// Decimal values stored as `BYTE_ARRAY` use two's-complement, big-endian
+ /// encoding, so plain unsigned byte-wise comparison (used for min/max
+ /// statistics on every other `BYTE_ARRAY` column) gives the wrong
+ /// ordering for negative values. When this is set, statistics use
+ /// [`compare_greater_byte_array_decimals`] instead, matching the
+ /// comparator used by the non-Arrow column writer path.
Review Comment:
```suggestion
```
I think the field name is pretty self explanatory.
##########
parquet/src/column/writer/mod.rs:
##########
@@ -2001,8 +2000,25 @@ fn compare_greater_f16(a: &[u8], b: &[u8]) -> bool {
a.total_cmp(&b) == Ordering::Greater
}
+/// Returns `true` if the column described by `basic_type_info` is a Decimal column
+/// (either via `ConvertedType::DECIMAL` or `LogicalType::Decimal`), regardless of
+/// whether its physical type is `BYTE_ARRAY` or `FIXED_LEN_BYTE_ARRAY`.
+///
+/// Decimal values stored as (FIXED_LEN_)BYTE_ARRAY use two's-complement, big-endian
+/// signed-integer encoding, which sorts differently from the unsigned, byte-wise
+/// lexicographic order used for other BYTE_ARRAY/FIXED_LEN_BYTE_ARRAY data. Callers
+/// that need unsigned byte-wise comparisons or increments (e.g. statistics
+/// truncation) must special-case or skip Decimal columns.
Review Comment:
```suggestion
/// Returns `true` if the column described by `basic_type_info` is a Decimal column
```
##########
parquet/src/column/writer/mod.rs:
##########
@@ -1352,14 +1352,17 @@ impl<'a, E: ColumnValueEncoder> GenericColumnWriter<'a, E> {
// from that of FIXED_LEN_BYTE_ARRAY sort order.
// So truncation of those types could lead to inaccurate min/max statistics
Type::FIXED_LEN_BYTE_ARRAY
- if !matches!(
- self.descr.logical_type_ref(),
- Some(&LogicalType::Decimal { .. } | &LogicalType::Float16)
- ) =>
+ if !is_decimal_descr(self.descr.get_basic_info())
+ && !matches!(self.descr.logical_type_ref(), Some(&LogicalType::Float16)) =>
{
true
}
- Type::BYTE_ARRAY => true,
+ // Decimal values encoded as BYTE_ARRAY use two's-complement, signed
+ // big-endian comparison, which differs from the unsigned, byte-wise
+ // comparison used to truncate/increment other BYTE_ARRAY values.
+ // Truncating such a value could produce an incorrect min/max, so skip
+ // truncation for Decimal BYTE_ARRAY columns as well.
Review Comment:
```suggestion
// As with FIXED_LEN_BYTE_ARRAY, do not truncate Decimal values
```
##########
parquet/src/column/writer/mod.rs:
##########
@@ -4568,6 +4584,58 @@ mod tests {
}
}
+ #[test]
+ fn test_decimal_byte_array_min_max_no_statistics_truncation() {
+ // Regression test for the truncation path re-introducing the unsigned vs.
+ // signed two's-complement comparison bug fixed for apache/arrow-rs#11073:
+ // `truncate_statistics` must not byte-wise truncate/increment
+ // `Statistics::ByteArray` min/max for a Decimal-typed BYTE_ARRAY column,
+ // even when `statistics_truncate_length` is configured and the encoded
+ // value is longer than the truncate length.
Review Comment:
```suggestion
// See https://github.com/apache/arrow-rs/issues/11073
```
##########
parquet/src/column/writer/mod.rs:
##########
@@ -2001,8 +2000,25 @@ fn compare_greater_f16(a: &[u8], b: &[u8]) -> bool {
a.total_cmp(&b) == Ordering::Greater
}
+/// Returns `true` if the column described by `basic_type_info` is a Decimal column
+/// (either via `ConvertedType::DECIMAL` or `LogicalType::Decimal`), regardless of
+/// whether its physical type is `BYTE_ARRAY` or `FIXED_LEN_BYTE_ARRAY`.
+///
+/// Decimal values stored as (FIXED_LEN_)BYTE_ARRAY use two's-complement, big-endian
+/// signed-integer encoding, which sorts differently from the unsigned, byte-wise
+/// lexicographic order used for other BYTE_ARRAY/FIXED_LEN_BYTE_ARRAY data. Callers
+/// that need unsigned byte-wise comparisons or increments (e.g. statistics
+/// truncation) must special-case or skip Decimal columns.
+pub(crate) fn is_decimal_descr(basic_type_info: &BasicTypeInfo) -> bool {
Review Comment:
perhaps inline this
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -6565,4 +6565,110 @@ mod tests {
);
assert_eq!(parquet_schema.column(1).path().string(), "row.b");
}
+
+ /// Regression test for https://github.com/apache/arrow-rs/issues/11073
+ ///
+ /// Decimal values backed by `BYTE_ARRAY`/`FIXED_LEN_BYTE_ARRAY` are
+ /// stored as two's-complement, big-endian bytes. Statistics for such a
+ /// column must therefore be compared with sign-awareness rather than
+ /// plain unsigned byte-wise `Ord`, or negative values (whose leading
+ /// byte has the sign bit set) sort as the largest values.
+ ///
+ /// This checks that the `ArrowColumnWriter` path (going through
+ /// `byte_array.rs`'s `ByteArrayEncoder`) produces the same min/max
+ /// statistics as writing the column directly with the low-level
+ /// `SerializedFileWriter` API, for both `BinaryArray` (the buggy case)
+ /// and `FixedSizeBinaryArray` (which already worked correctly), using
+ /// the exact 1-byte-decimal repro from the issue: values -1, 0, 1.
Review Comment:
```suggestion
/// This checks that the `ArrowColumnWriter` path produces the same min/max
/// statistics as writing the column directly with the low-level
/// `SerializedFileWriter` API, for both `BinaryArray` (the buggy case)
/// and `FixedSizeBinaryArray` (which already worked correctly).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "yuefdev (via GitHub)" <gi...@apache.org>.
yuefdev commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5916140485
Thanks for the heads-up! #11087 covers the comparison path, but misses the truncation guard added here. Happy to consolidate around this PR so the decimal fix lives in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5691288544
we have an existing PR for this
- https://github.com/apache/arrow-rs/pull/11087
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "DeviousCardi (via GitHub)" <gi...@apache.org>.
DeviousCardi commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5767543715
Benchmark came back neutral: base 1415.3s wall / 3.0 GiB peak vs branch 1420.3s / 2.9 GiB — ~0.35%, within noise, with slightly lower peak memory.
`is_decimal_descr` is now `#[inline]`, matching the comparators above it — let me know if you meant inlining it at the call sites instead. Comments are trimmed, and I ran rustfmt over this PR's own files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "DeviousCardi (via GitHub)" <gi...@apache.org>.
DeviousCardi commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5692371871
Trimmed the verbose comments per your suggestions, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "DeviousCardi (via GitHub)" <gi...@apache.org>.
DeviousCardi commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-6086121903
Thanks @yuefdev for closing #11087 in favour of this one, so the decimal fix lives in one place.
@etseidl the benchmark came back neutral and the comments are trimmed. This is ready for another look whenever you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5691597144
fyi @yuefdev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5691957713
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11092#issuecomment-5691929793)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5691929793-2387-vr48x 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing fix/11073-arrow-decimal-byte-array-stats (6626f93859f11de09774b09ddf7676a0c8b06a87) to 911721c (merge-base) [diff](https://github.com/apache/arrow-rs/compare/911721c69e21a2c6921b0bdb6097ba11cc101c22..6626f93859f11de09774b09ddf7676a0c8b06a87)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_writer
env:
BENCH_FILTER: "string"
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_writer
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5691929793
run benchmark arrow_writer
```yaml
env:
BENCH_FILTER: string
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl merged PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5692329630
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11092#issuecomment-5691929793)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing fix/11073-arrow-decimal-byte-array-stats (6626f93859f11de09774b09ddf7676a0c8b06a87) to 911721c (merge-base) [diff](https://github.com/apache/arrow-rs/compare/911721c69e21a2c6921b0bdb6097ba11cc101c22..6626f93859f11de09774b09ddf7676a0c8b06a87)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_writer
env:
BENCH_FILTER: "string"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group fix_11073-arrow-decimal-byte-array-stats main
----- ---------------------------------------- ----
large_string_distinct/delta_byte_array 1.00 59.4±0.37ms 4.2 GB/sec 1.01 60.0±0.48ms 4.2 GB/sec
large_string_distinct/plain 1.00 59.7±1.17ms 4.2 GB/sec 1.01 60.6±1.07ms 4.1 GB/sec
large_string_distinct_nullable/delta_byte_array 1.00 55.7±0.32ms 4.5 GB/sec 1.01 56.5±0.51ms 4.4 GB/sec
large_string_non_null/bloom_filter 1.00 51.8±0.08ms 4.8 GB/sec 1.00 51.6±0.07ms 4.8 GB/sec
large_string_non_null/cdc 1.00 239.8±0.84ms 1067.7 MB/sec 1.00 239.8±0.83ms 1067.5 MB/sec
large_string_non_null/default 1.00 51.7±0.07ms 4.8 GB/sec 1.00 51.5±0.06ms 4.9 GB/sec
large_string_non_null/number_distinct_values 1.00 68.2±0.14ms 3.7 GB/sec 1.01 68.6±0.14ms 3.6 GB/sec
large_string_non_null/parquet_2 1.00 51.7±0.07ms 4.8 GB/sec 1.00 51.6±0.08ms 4.8 GB/sec
large_string_non_null/zstd 1.00 51.7±0.06ms 4.8 GB/sec 1.00 51.7±0.17ms 4.8 GB/sec
large_string_non_null/zstd_parquet_2 1.00 51.7±0.06ms 4.8 GB/sec 1.00 51.8±0.13ms 4.8 GB/sec
large_string_shared_prefix/delta_byte_array 1.00 42.8±0.08ms 5.8 GB/sec 1.00 42.9±0.11ms 5.8 GB/sec
large_string_shared_prefix/plain 1.00 90.7±0.42ms 2.8 GB/sec 1.02 92.7±0.57ms 2.7 GB/sec
large_string_shared_prefix_list/delta_byte_array 1.00 53.3±0.22ms 4.7 GB/sec 1.01 54.1±0.19ms 4.6 GB/sec
large_string_shared_prefix_nullable/delta_byte_array 1.00 40.1±0.07ms 6.2 GB/sec 1.00 40.3±0.08ms 6.2 GB/sec
large_string_shared_prefix_nullable/plain 1.00 63.4±0.28ms 3.9 GB/sec 1.02 64.9±0.50ms 3.9 GB/sec
large_string_shared_prefix_nullable_dense/delta_byte_array 1.00 21.7±0.07ms 5.8 GB/sec 1.01 21.8±0.07ms 5.7 GB/sec
large_string_shared_prefix_nullable_trailing/delta_byte_array 1.00 40.2±0.22ms 6.2 GB/sec 1.00 40.3±0.08ms 6.2 GB/sec
medium_string_shared_prefix_nullable/delta_byte_array 1.00 34.7±0.07ms 7.2 GB/sec 1.04 35.9±0.08ms 7.0 GB/sec
short_string_non_null/bloom_filter 1.00 27.7±0.11ms 433.8 MB/sec 1.11 30.8±0.14ms 390.1 MB/sec
short_string_non_null/cdc 1.00 20.1±0.07ms 597.3 MB/sec 1.25 25.0±0.19ms 479.5 MB/sec
short_string_non_null/default 1.00 16.1±0.10ms 747.0 MB/sec 1.28 20.5±0.13ms 585.3 MB/sec
short_string_non_null/number_distinct_values 1.00 67.7±0.18ms 177.3 MB/sec 1.08 72.8±0.26ms 164.8 MB/sec
short_string_non_null/parquet_2 1.00 26.2±0.09ms 458.0 MB/sec 1.23 32.2±0.13ms 372.4 MB/sec
short_string_non_null/zstd 1.00 36.4±0.13ms 329.6 MB/sec 1.12 40.9±0.18ms 293.1 MB/sec
short_string_non_null/zstd_parquet_2 1.00 29.2±0.10ms 411.5 MB/sec 1.21 35.2±0.16ms 340.7 MB/sec
small_string_distinct/delta_byte_array 1.00 953.9±6.10µs 8.2 GB/sec 1.06 1008.3±8.46µs 7.8 GB/sec
small_string_distinct/plain 1.00 727.8±5.44µs 10.8 GB/sec 1.06 774.3±2.83µs 10.1 GB/sec
small_string_partial_prefix/delta_byte_array 1.00 1002.7±8.51µs 7.8 GB/sec 1.02 1024.4±4.90µs 7.7 GB/sec
small_string_partial_prefix/plain 1.00 987.6±5.36µs 7.9 GB/sec 1.01 993.0±4.89µs 7.9 GB/sec
small_string_shared_prefix/delta_byte_array 1.01 1002.7±2.81µs 7.8 GB/sec 1.00 987.9±1.53µs 7.9 GB/sec
small_string_shared_prefix/plain 1.02 1110.2±6.42µs 7.1 GB/sec 1.00 1087.1±5.67µs 7.2 GB/sec
string/bloom_filter 1.02 194.6±1.81ms 2.6 GB/sec 1.00 190.7±2.06ms 2.7 GB/sec
string/cdc 1.00 200.6±0.89ms 2.6 GB/sec 1.04 208.4±6.78ms 2.5 GB/sec
string/default 1.00 103.6±1.77ms 4.9 GB/sec 1.00 103.5±1.76ms 4.9 GB/sec
string/number_distinct_values 1.00 206.4±1.85ms 2.5 GB/sec 1.04 215.1±3.82ms 2.4 GB/sec
string/parquet_2 1.00 99.5±6.27ms 5.1 GB/sec 1.02 101.7±6.03ms 5.0 GB/sec
string/zstd 1.00 419.1±1.48ms 1250.8 MB/sec 1.01 424.3±1.08ms 1235.7 MB/sec
string/zstd_parquet_2 1.00 389.3±4.95ms 1346.7 MB/sec 1.00 387.5±3.82ms 1353.0 MB/sec
string_and_binary_view/bloom_filter 1.02 63.7±0.22ms 506.6 MB/sec 1.00 62.7±0.21ms 514.3 MB/sec
string_and_binary_view/cdc 1.00 60.3±0.16ms 535.3 MB/sec 1.01 60.8±0.18ms 530.4 MB/sec
string_and_binary_view/default 1.00 49.4±0.14ms 653.4 MB/sec 1.01 50.1±0.13ms 643.7 MB/sec
string_and_binary_view/number_distinct_values 1.00 49.3±0.12ms 654.4 MB/sec 1.02 50.1±0.13ms 643.7 MB/sec
string_and_binary_view/parquet_2 1.00 60.2±0.16ms 536.1 MB/sec 1.05 63.3±0.14ms 509.3 MB/sec
string_and_binary_view/zstd 1.00 86.4±0.19ms 373.2 MB/sec 1.01 87.3±0.13ms 369.3 MB/sec
string_and_binary_view/zstd_parquet_2 1.00 74.1±0.15ms 435.0 MB/sec 1.04 77.3±0.15ms 417.0 MB/sec
string_dictionary/bloom_filter 1.03 85.1±0.71ms 3.0 GB/sec 1.00 82.9±0.28ms 3.1 GB/sec
string_dictionary/cdc 1.00 46.3±0.84ms 5.6 GB/sec 1.00 46.4±0.40ms 5.6 GB/sec
string_dictionary/default 1.00 41.8±0.89ms 6.2 GB/sec 1.01 42.4±0.17ms 6.1 GB/sec
string_dictionary/number_distinct_values 1.01 73.1±0.26ms 3.5 GB/sec 1.00 72.3±0.29ms 3.6 GB/sec
string_dictionary/parquet_2 1.01 48.8±0.12ms 5.3 GB/sec 1.00 48.1±0.09ms 5.4 GB/sec
string_dictionary/zstd 1.00 204.3±0.64ms 1292.7 MB/sec 1.00 204.4±0.21ms 1292.3 MB/sec
string_dictionary/zstd_parquet_2 1.00 192.8±0.19ms 1370.0 MB/sec 1.00 192.4±0.19ms 1372.5 MB/sec
string_dictionary_low_cardinality_100/bloom_filter 1.00 24.9±0.13ms 161.2 MB/sec 1.01 25.2±0.11ms 158.9 MB/sec
string_dictionary_low_cardinality_100/cdc 1.00 26.9±0.13ms 148.8 MB/sec 1.01 27.3±0.09ms 146.7 MB/sec
string_dictionary_low_cardinality_100/default 1.00 24.8±0.12ms 161.7 MB/sec 1.01 25.1±0.09ms 159.4 MB/sec
string_dictionary_low_cardinality_100/number_distinct_values 1.00 40.5±0.18ms 99.0 MB/sec 1.01 40.8±0.16ms 98.1 MB/sec
string_dictionary_low_cardinality_100/parquet_2 1.00 24.8±0.12ms 161.6 MB/sec 1.01 25.1±0.11ms 159.5 MB/sec
string_dictionary_low_cardinality_100/zstd 1.00 25.1±0.13ms 159.5 MB/sec 1.01 25.5±0.11ms 157.2 MB/sec
string_dictionary_low_cardinality_100/zstd_parquet_2 1.00 25.1±0.11ms 159.4 MB/sec 1.01 25.5±0.10ms 157.2 MB/sec
string_dictionary_low_cardinality_20/bloom_filter 1.00 24.5±0.11ms 163.4 MB/sec 1.01 24.7±0.09ms 162.1 MB/sec
string_dictionary_low_cardinality_20/cdc 1.00 26.7±0.13ms 150.3 MB/sec 1.01 26.8±0.07ms 149.4 MB/sec
string_dictionary_low_cardinality_20/default 1.00 24.4±0.10ms 164.1 MB/sec 1.01 24.6±0.09ms 163.0 MB/sec
string_dictionary_low_cardinality_20/number_distinct_values 1.00 39.8±0.15ms 100.6 MB/sec 1.01 40.1±0.14ms 99.9 MB/sec
string_dictionary_low_cardinality_20/parquet_2 1.00 24.4±0.11ms 164.0 MB/sec 1.01 24.6±0.09ms 163.0 MB/sec
string_dictionary_low_cardinality_20/zstd 1.00 24.6±0.10ms 163.1 MB/sec 1.01 24.7±0.09ms 162.0 MB/sec
string_dictionary_low_cardinality_20/zstd_parquet_2 1.00 24.6±0.10ms 162.9 MB/sec 1.01 24.7±0.07ms 162.0 MB/sec
string_dictionary_low_cardinality_400/bloom_filter 1.00 25.5±0.12ms 157.4 MB/sec 1.00 25.4±0.16ms 157.7 MB/sec
string_dictionary_low_cardinality_400/cdc 1.00 27.5±0.15ms 146.0 MB/sec 1.00 27.6±0.15ms 145.4 MB/sec
string_dictionary_low_cardinality_400/default 1.00 25.3±0.14ms 158.6 MB/sec 1.00 25.3±0.15ms 158.4 MB/sec
string_dictionary_low_cardinality_400/number_distinct_values 1.00 41.2±0.19ms 97.5 MB/sec 1.00 41.3±0.21ms 97.1 MB/sec
string_dictionary_low_cardinality_400/parquet_2 1.00 25.3±0.14ms 158.4 MB/sec 1.00 25.4±0.14ms 158.3 MB/sec
string_dictionary_low_cardinality_400/zstd 1.00 25.8±0.14ms 155.4 MB/sec 1.00 25.9±0.15ms 155.2 MB/sec
string_dictionary_low_cardinality_400/zstd_parquet_2 1.00 25.8±0.14ms 155.4 MB/sec 1.00 25.9±0.13ms 155.0 MB/sec
string_non_null/bloom_filter 1.05 233.6±12.46ms 2.2 GB/sec 1.00 222.1±11.64ms 2.3 GB/sec
string_non_null/cdc 1.01 252.5±6.36ms 2.0 GB/sec 1.00 250.0±3.16ms 2.0 GB/sec
string_non_null/default 1.04 123.4±14.26ms 4.1 GB/sec 1.00 118.6±12.82ms 4.3 GB/sec
string_non_null/number_distinct_values 1.02 282.5±8.34ms 1855.2 MB/sec 1.00 276.6±8.09ms 1894.4 MB/sec
string_non_null/parquet_2 1.03 125.1±10.63ms 4.1 GB/sec 1.00 121.1±7.74ms 4.2 GB/sec
string_non_null/zstd 1.00 534.2±2.02ms 980.9 MB/sec 1.03 548.2±4.57ms 955.9 MB/sec
string_non_null/zstd_parquet_2 1.00 499.9±4.45ms 1048.1 MB/sec 1.01 503.5±8.08ms 1040.8 MB/sec
string_ree/bloom_filter 1.00 154.2±0.56ms 106.4 MB/sec 1.00 154.9±0.77ms 105.9 MB/sec
string_ree/cdc 1.00 224.0±0.53ms 73.2 MB/sec 1.01 225.8±1.73ms 72.7 MB/sec
string_ree/default 1.00 129.3±0.61ms 126.9 MB/sec 1.00 129.6±1.46ms 126.6 MB/sec
string_ree/number_distinct_values 1.00 164.7±0.51ms 99.6 MB/sec 1.02 168.6±0.68ms 97.3 MB/sec
string_ree/parquet_2 1.00 130.1±0.36ms 126.1 MB/sec 1.03 134.3±0.41ms 122.2 MB/sec
string_ree/zstd 1.00 186.4±0.77ms 88.0 MB/sec 1.00 186.4±0.68ms 88.0 MB/sec
string_ree/zstd_parquet_2 1.00 144.0±1.08ms 113.9 MB/sec 1.03 148.8±0.36ms 110.3 MB/sec
string_ree_95pct_null/bloom_filter 1.00 24.6±0.08ms 57.3 MB/sec 1.01 24.7±0.10ms 56.9 MB/sec
string_ree_95pct_null/cdc 1.00 32.2±0.15ms 43.7 MB/sec 1.00 32.2±0.08ms 43.7 MB/sec
string_ree_95pct_null/default 1.00 24.3±0.06ms 57.9 MB/sec 1.01 24.4±0.05ms 57.6 MB/sec
string_ree_95pct_null/number_distinct_values 1.00 26.5±0.08ms 53.2 MB/sec 1.00 26.6±0.08ms 52.9 MB/sec
string_ree_95pct_null/parquet_2 1.00 24.3±0.05ms 58.0 MB/sec 1.01 24.5±0.10ms 57.5 MB/sec
string_ree_95pct_null/zstd 1.00 25.1±0.06ms 56.1 MB/sec 1.01 25.2±0.07ms 55.7 MB/sec
string_ree_95pct_null/zstd_parquet_2 1.00 25.0±0.06ms 56.2 MB/sec 1.01 25.2±0.09ms 55.9 MB/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 1415.3s |
| Peak memory | 3.0 GiB |
| Avg memory | 2.6 GiB |
| CPU user | 1332.1s |
| CPU sys | 76.3s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 1420.3s |
| Peak memory | 2.9 GiB |
| Avg memory | 2.6 GiB |
| CPU user | 1338.0s |
| CPU sys | 75.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix incorrect Decimal statistics for BYTE_ARRAY columns written via ArrowColumnWriter [arrow-rs]
Posted by "DeviousCardi (via GitHub)" <gi...@apache.org>.
DeviousCardi commented on PR #11092:
URL: https://github.com/apache/arrow-rs/pull/11092#issuecomment-5691576386
Thanks for the pointer — #11087 fixes the same min/max comparison bug, but doesn't touch `can_truncate_value`/`truncate_statistics`, so a Decimal BYTE_ARRAY column with `statistics_truncate_length` set can still get its stats byte-truncated with unsigned semantics, reintroducing the same corruption via a different path. This PR closes that gap too, with a regression test (`test_decimal_byte_array_min_max_no_statistics_truncation`) covering it. Happy to defer to whichever lands first, but wanted to flag the difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org