You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mkleen (via GitHub)" <gi...@apache.org> on 2026/09/29 09:06:08 UTC
[PR] Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
mkleen opened a new pull request, #11285:
URL: https://github.com/apache/arrow-rs/pull/11285
# Which issue does this PR close?
- Closes #9296.
# Rationale for this change
Today, getting page-level statistics as Arrow arrays copies the data twice:
1. `decode_column_index` reads the Thrift bytes into `ThriftColumnIndex`. `PrimitiveColumnIndex::try_new` then copies the mins and maxes into typed `Vec<T>`s, putting default values in place for null pages.
2. `data_page_mins`, `data_page_maxes`, `data_page_null_counts` and `data_page_nan_counts` go through that `ColumnIndexMetaData` again and push each value into an Arrow builder one at a time. Each call walks the index separately.
# What changes are included in this PR?
This PR adds a way to skip loading the column index, then decode only the columns you need straight into Arrow arrays. The decoder lives in the arrow reader, so `ParquetMetaData`, `ColumnIndexMetaData` and `PageIndexProvider` are unchanged and contain no Arrow types.
### New API
```rust
pub struct DataPageStatistics {
pub mins: ArrayRef,
pub maxes: ArrayRef,
pub null_counts: UInt64Array,
pub nan_counts: UInt64Array,
}
impl StatisticsConverter<'_> {
pub fn data_page_statistics_from_bytes<'b, I>(&self, column_indexes: I) -> Result<DataPageStatistics>
where I: IntoIterator<Item = (usize, Option<&'b [u8]>)>;
}
```
- **Input:** one `(num_pages, bytes)` pair per row group. The bytes are the column chunk's serialized `ColumnIndex`, found with `ColumnChunkMetaData::column_index_range()`. To avoid parsing the index when the metadata is loaded, load with `PageIndexPolicy::Skip`.
- **Output:** one call returns all four arrays, matching `data_page_mins`, `data_page_maxes`, `data_page_null_counts` and `data_page_nan_counts`.
- **Missing data:** a row group with `None` bytes, or a column that isn't in the Parquet file, gives `num_pages` nulls in all four arrays.
The decoder is `parquet/src/arrow/arrow_reader/statistics/page_index.rs`. It reads the Thrift compact encoding of `ColumnIndex` in one pass, on top of `ThriftSliceInputProtocol`.
| Field | Thrift type | Handling |
|---|---|---|
| 1 `null_pages` | `list<bool>` | Stored one byte per element (`0x01` = true; `0x00` and `0x02` = false; any other byte is an error). Inverted to "page has a min/max" and packed into the `NullBuffer` shared by mins and maxes. |
| 2 `min_values`, 3 `max_values` | `list<binary>` | Each element is decoded straight into a buffer of the column's physical type (see below). The bytes of null pages are skipped without being checked. |
| 4 `boundary_order` | `i32` | Read and checked, then discarded. |
| 5 `null_counts`, 8 `nan_counts` | `list<i64>` | Zigzag varints decoded into `Vec<u64>`. A negative count is an error. A row group without the list gives null entries. |
| 6, 7 level histograms, and unknown fields | any | Skipped. For integer lists, skipping just counts the bytes whose top bit is clear, which is much faster than decoding each varint. |
**Physical type to buffer:**
| Physical type | Buffer | How each value is decoded |
|---|---|---|
| `BOOLEAN` | `Vec<bool>` | First byte ≠ 0 |
| `INT32` | `Vec<i32>` | First 4 bytes, little-endian |
| `INT64` | `Vec<i64>` | First 8 bytes, little-endian |
| `FLOAT` | `Vec<f32>` | First 4 bytes, little-endian |
| `DOUBLE` | `Vec<f64>` | First 8 bytes, little-endian |
| `BYTE_ARRAY` / `FIXED_LEN_BYTE_ARRAY` read as `Decimal32`/`64`/`128`/`256` | `Vec<i32>` … `Vec<i256>` | Big-endian two's complement, sign-extended while reading (`from_bytes_to_i*`). There is no intermediate `BinaryArray`. A value must be 1 to 4/8/16/32 bytes long. |
| Other `BYTE_ARRAY` / `FIXED_LEN_BYTE_ARRAY` | `i32` offsets + `Vec<u8>` → `BinaryArray` | Bytes copied as they are. Fixed-length values are kept as variable-length because a stored min or max may have been truncated. |
| `INT96` | Count only | Each value is checked to be at least 12 bytes. The result is always nulls, as before. |
As in the old decoder, a fixed-width value with extra trailing bytes is accepted, and one with too few bytes is an error with the same message: `error converting value, expected N bytes got M`.
### Benchmark
`parquet/benches/arrow_statistics.rs` compares `data_page_statistics_from_bytes` with `decode_column_index` plus all four `data_page_*` calls, starting from the same raw bytes in both cases:
| Type | Pages | Existing route | `data_page_statistics_from_bytes` | Speedup |
|---|---:|---:|---:|---:|
| Int64 | 2,000 | 29.7 µs | 7.0 µs | 4.22× |
| Int64 | 10,000 | 121.3 µs | 29.4 µs | 4.12× |
| Utf8 | 2,000 | 109.7 µs | 32.6 µs | 3.36× |
| Utf8 | 10,000 | 500.5 µs | 159.1 µs | 3.15× |
| Utf8View | 2,000 | 109.9 µs | 41.7 µs | 2.64× |
| Utf8View | 10,000 | 515.9 µs | 203.7 µs | 2.53× |
| Decimal128(20, 2) | 2,000 | 49.2 µs | 15.9 µs | 3.10× |
| Decimal128(20, 2) | 10,000 | 220.2 µs | 71.8 µs | 3.07× |
Each file has 20 row groups with 10 rows per page, and every 7th value is null. The Utf8 values are 14 bytes, so as views they point into the data buffer rather than being stored inline.
# Are these changes tested?
- **Unit tests** in `page_index.rs`:
- Every physical type is compared against every Arrow type with the existing route
- Specific tests cover INT96, all-null pages, zero pages, no row groups, a column missing from the file, unknown fields and histograms, and mins stored before `null_pages`.
- Every error case is checked, including empty and over-wide decimal values for both byte column types, and decoding is tried on every truncation of a valid index.
- **Integration tests:** `parquet/tests/arrow_reader/statistics.rs` now also checks every existing page-level case through `data_page_statistics_from_bytes`, using the column index bytes read from the file.
# Are there any user-facing changes?
Yes. This PR adds a new public API but nothing existing changes.
- New method StatisticsConverter::data_page_statistics_from_bytes.
- New struct parquet::arrow::arrow_reader::statistics::DataPageStatistics
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#discussion_r4149181045
##########
parquet/benches/arrow_statistics.rs:
##########
@@ -347,33 +342,18 @@ fn page_index_benchmark(c: &mut Criterion) {
(num_pages, &data[range.start as usize..range.end as usize])
})
.collect();
- let row_group_indices: Vec<usize> = (0..row_groups).collect();
- let num_columns = reader.parquet_schema().num_columns();
let mut group = c.benchmark_group(format!(
"Decode page index statistics for {data_type} ({} pages)",
column_indexes.iter().map(|(n, _)| n).sum::<usize>()
));
group.bench_function("full page index", |b| {
b.iter(|| {
- let mut builder = PageIndexBuilder::new(row_groups, num_columns);
- for (rg, (_, bytes)) in column_indexes.iter().enumerate() {
- let index = decode_column_index(bytes, physical_type).unwrap();
- builder.put_column_index(index, rg, column);
- }
- let page_index = builder.build();
- let _ = converter
- .data_page_mins(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_maxes(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_null_counts(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_nan_counts(&page_index, &row_group_indices)
- .unwrap();
+ converter
Review Comment:
ok, sounds good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5920979423
Numbers look great. It's a large PR, so I'll likely review in pieces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#discussion_r4146270465
##########
parquet/src/arrow/arrow_reader/statistics/page_index.rs:
##########
@@ -0,0 +1,1594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Reads the page statistics stored in a Parquet `ColumnIndex` and puts them
+//! straight into Arrow arrays.
+//!
+//! The usual route first builds a [`ColumnIndexMetaData`] and then copies its
+//! values into Arrow arrays one by one. This module skips the middle step: it
+//! reads the stored bytes once and writes each value directly into the memory
+//! that will become the final Arrow array.
+//!
+//! [`ColumnIndexMetaData`]: crate::file::page_index::column_index::ColumnIndexMetaData
+
+use super::{DataPageStatistics, from_bytes_to_i256};
+use super::{from_bytes_to_f16, from_bytes_to_i32, from_bytes_to_i64, from_bytes_to_i128};
+use crate::basic::{BoundaryOrder, Type as PhysicalType};
+use crate::errors::{ParquetError, Result};
+use crate::parquet_thrift::{
+ ElementType, FieldType, ReadThrift, ThriftCompactInputProtocol, ThriftSliceInputProtocol,
+ validate_list_type,
+};
+use arrow_array::types::{
+ Date32Type, Date64Type, Decimal32Type, Decimal64Type, Decimal128Type, Decimal256Type, Int8Type,
+ Int16Type, Time32MillisecondType, Time32SecondType, Time64MicrosecondType,
+ Time64NanosecondType, TimestampMicrosecondType, TimestampMillisecondType,
+ TimestampNanosecondType, TimestampSecondType, UInt8Type, UInt16Type, UInt32Type, UInt64Type,
+};
+use arrow_array::{
+ Array, ArrayRef, BinaryArray, BinaryViewArray, BooleanArray, Decimal32Array, Decimal64Array,
+ Decimal128Array, Decimal256Array, FixedSizeBinaryArray, Float16Array, Float32Array,
+ Float64Array, Int32Array, Int64Array, LargeBinaryArray, LargeStringArray, StringArray,
+ StringViewArray, UInt64Array, new_null_array,
+};
+use arrow_buffer::{
+ BooleanBuffer, BooleanBufferBuilder, NullBuffer, OffsetBuffer, ScalarBuffer, i256,
+};
+use arrow_schema::{DataType, TimeUnit};
+use std::sync::Arc;
+
+/// The min or max values of every page, kept in the form Parquet stores them in
+/// (its "physical type"), before they are turned into the Arrow type the
+/// caller asked for.
+#[derive(Debug)]
+enum PhysicalValues {
+ Boolean(Vec<bool>),
+ Int32(Vec<i32>),
+ Int64(Vec<i64>),
+ Float(Vec<f32>),
+ Double(Vec<f64>),
+ /// Used for both `BYTE_ARRAY` and `FIXED_LEN_BYTE_ARRAY`. Fixed length
+ /// values are kept here too, because a stored min or max may have been
+ /// shortened and so may not have the expected length.
+ Bytes {
+ offsets: Vec<i32>,
+ values: Vec<u8>,
+ /// `true` for `FIXED_LEN_BYTE_ARRAY`
+ fixed_len: bool,
+ },
+ /// `INT96` values are only counted, not kept. No Arrow type is read
+ /// from them, so their mins and maxes always come out as nulls. They are
+ /// still checked for length, as the older decoder does.
+ Int96(Vec<()>),
+ /// Decimals stored as bytes (in either kind of byte column), turned
+ /// into numbers while reading, with their precision and scale. This skips
+ /// keeping a copy of the bytes and converting them afterwards, which is
+ /// much faster.
+ Decimal32(Vec<i32>, u8, i8),
+ Decimal64(Vec<i64>, u8, i8),
+ Decimal128(Vec<i128>, u8, i8),
+ Decimal256(Vec<i256>, u8, i8),
+}
+
+impl PhysicalValues {
+ /// `data_type` is the Arrow type the caller wants. It only matters for
+ /// decimals stored as bytes, which are turned into numbers right away.
+ fn new(physical_type: PhysicalType, data_type: &DataType, capacity: usize) -> Self {
Review Comment:
I guess we could simplify this with some macros but this would it only make it harder to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6197368764
@mkleen I haven't forgotten this. Some early feedback is I don't think encryption has been addressed here, either through implementation nor documentation. If you could give some thought to that while I shake loose time to do a proper review we can at least work in parallel 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6223948703
Am I missing something -- the most recent benchmark run https://github.com/apache/arrow-rs/pull/11285#issuecomment-5919627152 doesn't look much faster 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6223951157
🤖 Benchmark starting (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11285#issuecomment-6223949657)
**Target:** arrow_statistics
**Sharding factor: 1** (1 workers).
Comparing parquet-page-stats-from-bytes (f70c9679a75ff056afbad9d5334409e41b1d53c8) to 6b34163 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6b34163d51aa59f7cac2edcb74e8bd1ddc6b9120..f70c9679a75ff056afbad9d5334409e41b1d53c8)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_statistics
shards: 1
```
</details>
Results will be posted when all workers finish.
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6223949657:start:7a669538111b92174af083cf6bc57df7 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6223949657
run benchmark arrow_statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5906478399
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11285#issuecomment-5906265145)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5906265145-2953-b6xfk 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing parquet-page-stats-from-bytes (690abec061e5131692bd7d1be6d36f4d048b3d53) to 176171f (merge-base) [diff](https://github.com/apache/arrow-rs/compare/176171f708f40e73c2bf1d45ba74c83e6bdaa9cd..690abec061e5131692bd7d1be6d36f4d048b3d53)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_statistics
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_statistics
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#discussion_r4146270465
##########
parquet/src/arrow/arrow_reader/statistics/page_index.rs:
##########
@@ -0,0 +1,1594 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Reads the page statistics stored in a Parquet `ColumnIndex` and puts them
+//! straight into Arrow arrays.
+//!
+//! The usual route first builds a [`ColumnIndexMetaData`] and then copies its
+//! values into Arrow arrays one by one. This module skips the middle step: it
+//! reads the stored bytes once and writes each value directly into the memory
+//! that will become the final Arrow array.
+//!
+//! [`ColumnIndexMetaData`]: crate::file::page_index::column_index::ColumnIndexMetaData
+
+use super::{DataPageStatistics, from_bytes_to_i256};
+use super::{from_bytes_to_f16, from_bytes_to_i32, from_bytes_to_i64, from_bytes_to_i128};
+use crate::basic::{BoundaryOrder, Type as PhysicalType};
+use crate::errors::{ParquetError, Result};
+use crate::parquet_thrift::{
+ ElementType, FieldType, ReadThrift, ThriftCompactInputProtocol, ThriftSliceInputProtocol,
+ validate_list_type,
+};
+use arrow_array::types::{
+ Date32Type, Date64Type, Decimal32Type, Decimal64Type, Decimal128Type, Decimal256Type, Int8Type,
+ Int16Type, Time32MillisecondType, Time32SecondType, Time64MicrosecondType,
+ Time64NanosecondType, TimestampMicrosecondType, TimestampMillisecondType,
+ TimestampNanosecondType, TimestampSecondType, UInt8Type, UInt16Type, UInt32Type, UInt64Type,
+};
+use arrow_array::{
+ Array, ArrayRef, BinaryArray, BinaryViewArray, BooleanArray, Decimal32Array, Decimal64Array,
+ Decimal128Array, Decimal256Array, FixedSizeBinaryArray, Float16Array, Float32Array,
+ Float64Array, Int32Array, Int64Array, LargeBinaryArray, LargeStringArray, StringArray,
+ StringViewArray, UInt64Array, new_null_array,
+};
+use arrow_buffer::{
+ BooleanBuffer, BooleanBufferBuilder, NullBuffer, OffsetBuffer, ScalarBuffer, i256,
+};
+use arrow_schema::{DataType, TimeUnit};
+use std::sync::Arc;
+
+/// The min or max values of every page, kept in the form Parquet stores them in
+/// (its "physical type"), before they are turned into the Arrow type the
+/// caller asked for.
+#[derive(Debug)]
+enum PhysicalValues {
+ Boolean(Vec<bool>),
+ Int32(Vec<i32>),
+ Int64(Vec<i64>),
+ Float(Vec<f32>),
+ Double(Vec<f64>),
+ /// Used for both `BYTE_ARRAY` and `FIXED_LEN_BYTE_ARRAY`. Fixed length
+ /// values are kept here too, because a stored min or max may have been
+ /// shortened and so may not have the expected length.
+ Bytes {
+ offsets: Vec<i32>,
+ values: Vec<u8>,
+ /// `true` for `FIXED_LEN_BYTE_ARRAY`
+ fixed_len: bool,
+ },
+ /// `INT96` values are only counted, not kept. No Arrow type is read
+ /// from them, so their mins and maxes always come out as nulls. They are
+ /// still checked for length, as the older decoder does.
+ Int96(Vec<()>),
+ /// Decimals stored as bytes (in either kind of byte column), turned
+ /// into numbers while reading, with their precision and scale. This skips
+ /// keeping a copy of the bytes and converting them afterwards, which is
+ /// much faster.
+ Decimal32(Vec<i32>, u8, i8),
+ Decimal64(Vec<i64>, u8, i8),
+ Decimal128(Vec<i128>, u8, i8),
+ Decimal256(Vec<i256>, u8, i8),
+}
+
+impl PhysicalValues {
+ /// `data_type` is the Arrow type the caller wants. It only matters for
+ /// decimals stored as bytes, which are turned into numbers right away.
+ fn new(physical_type: PhysicalType, data_type: &DataType, capacity: usize) -> Self {
Review Comment:
I guess we could simplify this with some macros but this would it only make it harder to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#discussion_r4146263175
##########
parquet/benches/arrow_statistics.rs:
##########
@@ -347,33 +342,18 @@ fn page_index_benchmark(c: &mut Criterion) {
(num_pages, &data[range.start as usize..range.end as usize])
})
.collect();
- let row_group_indices: Vec<usize> = (0..row_groups).collect();
- let num_columns = reader.parquet_schema().num_columns();
let mut group = c.benchmark_group(format!(
"Decode page index statistics for {data_type} ({} pages)",
column_indexes.iter().map(|(n, _)| n).sum::<usize>()
));
group.bench_function("full page index", |b| {
b.iter(|| {
- let mut builder = PageIndexBuilder::new(row_groups, num_columns);
- for (rg, (_, bytes)) in column_indexes.iter().enumerate() {
- let index = decode_column_index(bytes, physical_type).unwrap();
- builder.put_column_index(index, rg, column);
- }
- let page_index = builder.build();
- let _ = converter
- .data_page_mins(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_maxes(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_null_counts(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_nan_counts(&page_index, &row_group_indices)
- .unwrap();
+ converter
Review Comment:
This measures `data_page_statistics_from_bytes` against `data_page_mins`, `data_page_maxes`, `data_page_null_counts` and `data_page_nan_counts`. So i think this is a fair comparision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5919455120
run benchmark arrow_statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6198303924
> @mkleen I haven't forgotten this. Some early feedback is I don't think encryption has been addressed here, either through implementation nor documentation. If you could give some thought to that while I shake loose time to do a proper review we can at least work in parallel 😅
No worries, take your time and thanks for effort. I will look into encryption in parallel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5914192558
@etseidl This is now ready for review! I have more ideas, then the pr would too big. So I would say let's start from here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6224152589
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11285#issuecomment-6223949657)
**Target:** arrow_statistics
Comparing parquet-page-stats-from-bytes (f70c9679a75ff056afbad9d5334409e41b1d53c8) to 6b34163 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/6b34163d51aa59f7cac2edcb74e8bd1ddc6b9120..f70c9679a75ff056afbad9d5334409e41b1d53c8)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_statistics
shards: 1
```
</details>
<details><summary>Details</summary>
```
group base changed
----- ---- -------
Decode page index statistics for Decimal128(20, 2) (10000 pages)/from bytes 1.00 148.1±0.70µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (10000 pages)/full page index 1.00 285.5±0.31µs ? ?/sec 1.00 284.7±0.44µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (2000 pages)/from bytes 1.00 31.8±0.19µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (2000 pages)/full page index 1.00 65.4±0.13µs ? ?/sec 1.00 65.5±0.19µs ? ?/sec
Decode page index statistics for Int64 (10000 pages)/from bytes 1.00 30.6±0.04µs ? ?/sec
Decode page index statistics for Int64 (10000 pages)/full page index 1.00 157.3±0.19µs ? ?/sec 1.00 156.7±0.19µs ? ?/sec
Decode page index statistics for Int64 (2000 pages)/from bytes 1.00 8.0±0.02µs ? ?/sec
Decode page index statistics for Int64 (2000 pages)/full page index 1.00 38.0±0.04µs ? ?/sec 1.00 38.0±0.06µs ? ?/sec
Decode page index statistics for Utf8 (10000 pages)/from bytes 1.00 109.0±0.54µs ? ?/sec
Decode page index statistics for Utf8 (10000 pages)/full page index 1.00 381.5±7.95µs ? ?/sec 1.00 381.3±0.35µs ? ?/sec
Decode page index statistics for Utf8 (2000 pages)/from bytes 1.00 24.4±0.17µs ? ?/sec
Decode page index statistics for Utf8 (2000 pages)/full page index 1.00 85.6±1.59µs ? ?/sec 1.00 85.4±0.10µs ? ?/sec
Decode page index statistics for Utf8View (10000 pages)/from bytes 1.00 148.9±0.22µs ? ?/sec
Decode page index statistics for Utf8View (10000 pages)/full page index 1.03 409.6±2.76µs ? ?/sec 1.00 399.6±0.51µs ? ?/sec
Decode page index statistics for Utf8View (2000 pages)/from bytes 1.00 33.0±0.18µs ? ?/sec
Decode page index statistics for Utf8View (2000 pages)/full page index 1.02 89.6±0.20µs ? ?/sec 1.00 88.0±0.22µs ? ?/sec
Extract data page statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 8.4±0.07µs ? ?/sec 1.04 8.8±0.06µs ? ?/sec
Extract data page statistics for F64/extract_statistics/F64 1.00 3.6±0.01µs ? ?/sec 1.02 3.6±0.01µs ? ?/sec
Extract data page statistics for Int64/extract_statistics/Int64 1.01 3.5±0.00µs ? ?/sec 1.00 3.5±0.01µs ? ?/sec
Extract data page statistics for String/extract_statistics/String 1.02 8.5±0.06µs ? ?/sec 1.00 8.3±0.07µs ? ?/sec
Extract data page statistics for UInt64/extract_statistics/UInt64 1.00 3.6±0.01µs ? ?/sec 1.00 3.6±0.01µs ? ?/sec
Extract row group statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 813.0±5.81ns ? ?/sec 1.00 810.9±1.79ns ? ?/sec
Extract row group statistics for F64/extract_statistics/F64 1.00 573.5±7.35ns ? ?/sec 1.01 576.6±4.68ns ? ?/sec
Extract row group statistics for Int64/extract_statistics/Int64 1.01 569.1±6.98ns ? ?/sec 1.00 565.1±1.56ns ? ?/sec
Extract row group statistics for String/extract_statistics/String 1.02 823.0±6.28ns ? ?/sec 1.00 810.8±3.91ns ? ?/sec
Extract row group statistics for UInt64/extract_statistics/UInt64 1.01 568.9±6.11ns ? ?/sec 1.00 564.7±1.83ns ? ?/sec
```
</details>
<details><summary>Per-runner information</summary>
<details><summary>arrow_statistics — shard 1/1</summary>
**Node:** gk3-benchmark-cluster-nap-3hyljax2-f91c57f4-sfdt
**Instance:** c4a-highmem-16 (12 vCPU / 65 GiB)
**uname:**
```text
Linux bench-c6223949657-3521-2mdgx 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux
```
**BENCH_COMMAND:**
```sh
cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_statistics
```
<details><summary>CPU Details (lscpu)</summary>
```text
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 180.0s |
| Peak memory | 19.1 MiB |
| Avg memory | 15.3 MiB |
| CPU user | 175.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 255.1s |
| Peak memory | 19.1 MiB |
| Avg memory | 16.5 MiB |
| CPU user | 251.3s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
</details>
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
<!-- benchmark-request:6223949657:finish:1a92b870bc328ab9cbb9e85d24a05624 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6230987588
> Thanks @mkleen -- I was thinking maybe we could break this PR up into smaller pieces for easier review. For example:
>
> For example:
>
> 1. the basic API and the decoder for int32 statistics
>
> Then other follow on PRs for the other types?
This is a really good idea. I will do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5906265145
run benchmark arrow_statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5894336697
This is really nice @mkleen...it's going to obviate most of the changes I've been making recently 😄 😭
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5906782135
Benchmarks are looking promising.
```
Decode page index statistics for Decimal128(20, 2) (10000 pages)/full page index 1.89 284.3±0.21µs ? ?/sec 1.00 150.6±0.11µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (2000 pages)/full page index 2.03 65.6±0.09µs ? ?/sec 1.00 32.4±0.06µs ? ?/sec
Decode page index statistics for Int64 (10000 pages)/full page index 5.12 156.7±0.13µs ? ?/sec 1.00 30.6±0.03µs ? ?/sec
Decode page index statistics for Int64 (2000 pages)/full page index 4.68 37.9±0.03µs ? ?/sec 1.00 8.1±0.04µs ? ?/sec
Decode page index statistics for Utf8 (10000 pages)/full page index 3.60 379.5±0.35µs ? ?/sec 1.00 105.6±0.56µs ? ?/sec
Decode page index statistics for Utf8 (2000 pages)/full page index 3.51 85.5±0.09µs ? ?/sec 1.00 24.4±0.08µs ? ?/sec
Decode page index statistics for Utf8View (10000 pages)/full page index 2.56 397.2±0.31µs ? ?/sec 1.00 154.9±0.58µs ? ?/sec
Decode page index statistics for Utf8View (2000 pages)/full page index 2.57 88.2±0.14µs ? ?/sec 1.00 34.3±0.14µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5906582657
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11285#issuecomment-5906265145)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing parquet-page-stats-from-bytes (690abec061e5131692bd7d1be6d36f4d048b3d53) to 176171f (merge-base) [diff](https://github.com/apache/arrow-rs/compare/176171f708f40e73c2bf1d45ba74c83e6bdaa9cd..690abec061e5131692bd7d1be6d36f4d048b3d53)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_statistics
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main parquet-page-stats-from-bytes
----- ---- -----------------------------
Decode page index statistics for Decimal128(20, 2) (10000 pages)/full page index 1.89 284.3±0.21µs ? ?/sec 1.00 150.6±0.11µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (2000 pages)/full page index 2.03 65.6±0.09µs ? ?/sec 1.00 32.4±0.06µs ? ?/sec
Decode page index statistics for Int64 (10000 pages)/full page index 5.12 156.7±0.13µs ? ?/sec 1.00 30.6±0.03µs ? ?/sec
Decode page index statistics for Int64 (2000 pages)/full page index 4.68 37.9±0.03µs ? ?/sec 1.00 8.1±0.04µs ? ?/sec
Decode page index statistics for Utf8 (10000 pages)/full page index 3.60 379.5±0.35µs ? ?/sec 1.00 105.6±0.56µs ? ?/sec
Decode page index statistics for Utf8 (2000 pages)/full page index 3.51 85.5±0.09µs ? ?/sec 1.00 24.4±0.08µs ? ?/sec
Decode page index statistics for Utf8View (10000 pages)/full page index 2.56 397.2±0.31µs ? ?/sec 1.00 154.9±0.58µs ? ?/sec
Decode page index statistics for Utf8View (2000 pages)/full page index 2.57 88.2±0.14µs ? ?/sec 1.00 34.3±0.14µs ? ?/sec
Extract data page statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 8.4±0.07µs ? ?/sec 1.00 8.3±0.06µs ? ?/sec
Extract data page statistics for F64/extract_statistics/F64 1.01 3.6±0.01µs ? ?/sec 1.00 3.5±0.01µs ? ?/sec
Extract data page statistics for Int64/extract_statistics/Int64 1.00 3.6±0.01µs ? ?/sec 1.00 3.6±0.01µs ? ?/sec
Extract data page statistics for String/extract_statistics/String 1.00 8.5±0.06µs ? ?/sec 1.01 8.5±0.07µs ? ?/sec
Extract data page statistics for UInt64/extract_statistics/UInt64 1.00 3.6±0.01µs ? ?/sec 1.07 3.8±0.01µs ? ?/sec
Extract row group statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 809.7±3.27ns ? ?/sec 1.03 830.7±3.60ns ? ?/sec
Extract row group statistics for F64/extract_statistics/F64 1.01 574.9±4.33ns ? ?/sec 1.00 569.5±3.13ns ? ?/sec
Extract row group statistics for Int64/extract_statistics/Int64 1.00 569.1±2.45ns ? ?/sec 1.00 570.9±5.87ns ? ?/sec
Extract row group statistics for String/extract_statistics/String 1.00 841.5±5.10ns ? ?/sec 1.01 848.4±5.65ns ? ?/sec
Extract row group statistics for UInt64/extract_statistics/UInt64 1.01 568.6±3.20ns ? ?/sec 1.00 564.3±1.58ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 180.0s |
| Peak memory | 19.3 MiB |
| Avg memory | 15.4 MiB |
| CPU user | 175.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 175.0s |
| Peak memory | 18.8 MiB |
| Avg memory | 15.0 MiB |
| CPU user | 170.2s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5919627152
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11285#issuecomment-5919455120)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing parquet-page-stats-from-bytes (e8d3b6cebfcea26304c453c130cb8cd7a1cd2ba6) to 176171f (merge-base) [diff](https://github.com/apache/arrow-rs/compare/176171f708f40e73c2bf1d45ba74c83e6bdaa9cd..e8d3b6cebfcea26304c453c130cb8cd7a1cd2ba6)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_statistics
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group main parquet-page-stats-from-bytes
----- ---- -----------------------------
Decode page index statistics for Decimal128(20, 2) (10000 pages)/from bytes 1.00 147.9±0.30µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (10000 pages)/full page index 1.00 284.9±0.30µs ? ?/sec 1.00 284.5±0.56µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (2000 pages)/from bytes 1.00 31.8±0.08µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (2000 pages)/full page index 1.01 65.4±0.12µs ? ?/sec 1.00 65.0±0.18µs ? ?/sec
Decode page index statistics for Int64 (10000 pages)/from bytes 1.00 30.6±0.03µs ? ?/sec
Decode page index statistics for Int64 (10000 pages)/full page index 1.00 156.4±0.15µs ? ?/sec 1.00 157.0±0.17µs ? ?/sec
Decode page index statistics for Int64 (2000 pages)/from bytes 1.00 8.0±0.05µs ? ?/sec
Decode page index statistics for Int64 (2000 pages)/full page index 1.00 37.9±0.04µs ? ?/sec 1.01 38.2±0.12µs ? ?/sec
Decode page index statistics for Utf8 (10000 pages)/from bytes 1.00 108.2±0.56µs ? ?/sec
Decode page index statistics for Utf8 (10000 pages)/full page index 1.00 379.1±0.28µs ? ?/sec 1.00 380.2±0.37µs ? ?/sec
Decode page index statistics for Utf8 (2000 pages)/from bytes 1.00 24.6±0.18µs ? ?/sec
Decode page index statistics for Utf8 (2000 pages)/full page index 1.00 85.4±0.10µs ? ?/sec 1.00 85.7±0.18µs ? ?/sec
Decode page index statistics for Utf8View (10000 pages)/from bytes 1.00 154.7±0.66µs ? ?/sec
Decode page index statistics for Utf8View (10000 pages)/full page index 1.00 404.9±1.27µs ? ?/sec 1.00 406.4±1.13µs ? ?/sec
Decode page index statistics for Utf8View (2000 pages)/from bytes 1.00 34.3±0.18µs ? ?/sec
Decode page index statistics for Utf8View (2000 pages)/full page index 1.00 89.8±0.12µs ? ?/sec 1.00 89.6±0.14µs ? ?/sec
Extract data page statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 8.4±0.06µs ? ?/sec 1.00 8.4±0.07µs ? ?/sec
Extract data page statistics for F64/extract_statistics/F64 1.01 3.6±0.01µs ? ?/sec 1.00 3.6±0.00µs ? ?/sec
Extract data page statistics for Int64/extract_statistics/Int64 1.00 3.5±0.01µs ? ?/sec 1.01 3.6±0.01µs ? ?/sec
Extract data page statistics for String/extract_statistics/String 1.00 8.5±0.07µs ? ?/sec 1.00 8.5±0.06µs ? ?/sec
Extract data page statistics for UInt64/extract_statistics/UInt64 1.00 3.6±0.01µs ? ?/sec 1.07 3.8±0.01µs ? ?/sec
Extract row group statistics for Dictionary(Int32, String)/extract_statistics/Dictionary(Int32, String) 1.00 816.4±9.46ns ? ?/sec 1.03 838.3±2.94ns ? ?/sec
Extract row group statistics for F64/extract_statistics/F64 1.02 584.4±4.98ns ? ?/sec 1.00 571.4±4.04ns ? ?/sec
Extract row group statistics for Int64/extract_statistics/Int64 1.01 576.1±5.11ns ? ?/sec 1.00 569.5±3.32ns ? ?/sec
Extract row group statistics for String/extract_statistics/String 1.00 835.9±9.50ns ? ?/sec 1.02 852.9±3.17ns ? ?/sec
Extract row group statistics for UInt64/extract_statistics/UInt64 1.02 584.8±6.23ns ? ?/sec 1.00 571.7±4.65ns ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 180.0s |
| Peak memory | 19.2 MiB |
| Avg memory | 15.4 MiB |
| CPU user | 175.9s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 255.1s |
| Peak memory | 19.8 MiB |
| Avg memory | 16.0 MiB |
| CPU user | 253.3s |
| CPU sys | 0.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#discussion_r4148813298
##########
parquet/benches/arrow_statistics.rs:
##########
@@ -347,33 +342,18 @@ fn page_index_benchmark(c: &mut Criterion) {
(num_pages, &data[range.start as usize..range.end as usize])
})
.collect();
- let row_group_indices: Vec<usize> = (0..row_groups).collect();
- let num_columns = reader.parquet_schema().num_columns();
let mut group = c.benchmark_group(format!(
"Decode page index statistics for {data_type} ({} pages)",
column_indexes.iter().map(|(n, _)| n).sum::<usize>()
));
group.bench_function("full page index", |b| {
b.iter(|| {
- let mut builder = PageIndexBuilder::new(row_groups, num_columns);
- for (rg, (_, bytes)) in column_indexes.iter().enumerate() {
- let index = decode_column_index(bytes, physical_type).unwrap();
- builder.put_column_index(index, rg, column);
- }
- let page_index = builder.build();
- let _ = converter
- .data_page_mins(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_maxes(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_null_counts(&page_index, &row_group_indices)
- .unwrap();
- let _ = converter
- .data_page_nan_counts(&page_index, &row_group_indices)
- .unwrap();
+ converter
Review Comment:
I agree this is the comparison we want to make for this PR, but it does change the meaning of the bench, making it hard to measure any gains we might make in the original path. How about adding a third set of benches that measure the performance of this new work. I know you can't run critcmp then, but I think seeing the old and new in a single run suffices to prove impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-5919504216
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/11285#issuecomment-5919455120)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5919455120-2978-zzkwm 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing parquet-page-stats-from-bytes (e8d3b6cebfcea26304c453c130cb8cd7a1cd2ba6) to 176171f (merge-base) [diff](https://github.com/apache/arrow-rs/compare/176171f708f40e73c2bf1d45ba74c83e6bdaa9cd..e8d3b6cebfcea26304c453c130cb8cd7a1cd2ba6)
<details><summary>Run configuration</summary>
```yaml
run benchmark arrow_statistics
```
</details>
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_statistics
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6226190324
> Am I missing something -- the most recent benchmark run [#11285 (comment)](https://github.com/apache/arrow-rs/pull/11285#issuecomment-5919627152) doesn't look much faster 🤔
@alamb I believe you want to compare the "from bytes" added here to the "full page index" which uses the normal statistics converter. In this case we go from 284us to 148us. Int64 saw a 4-5X speedup.
```
Decode page index statistics for Decimal128(20, 2) (10000 pages)/from bytes 1.00 147.9±0.30µs ? ?/sec
Decode page index statistics for Decimal128(20, 2) (10000 pages)/full page index 1.00 284.9±0.30µs ? ?/sec 1.00 284.5±0.56µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(parquet): Decode Parquet `ColumnIndex` bytes straight into Arrow arrays [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11285:
URL: https://github.com/apache/arrow-rs/pull/11285#issuecomment-6231084912
@alamb Here is an intepretation of the last benchmark run. It's not straight forward to read because we need to compare `full page index` with `from bytes`:
| Type | Pages | Old (full page index) | New (from bytes) | Speedup |
|---|---:|---:|---:|---:|
| Int64 | 10,000 | 156.7 µs | 30.6 µs | **5.1×** |
| Int64 | 2,000 | 38.0 µs | 8.0 µs | **4.8×** |
| Utf8 | 10,000 | 381.3 µs | 109.0 µs | **3.5×** |
| Utf8 | 2,000 | 85.4 µs | 24.4 µs | **3.5×** |
| Utf8View | 10,000 | 399.6 µs | 148.9 µs | **2.7×** |
| Utf8View | 2,000 | 88.0 µs | 33.0 µs | **2.7×** |
| Decimal128(20, 2) | 10,000 | 284.7 µs | 148.1 µs | **1.9×** |
| Decimal128(20, 2) | 2,000 | 65.5 µs | 31.8 µs | **2.1×** |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org