You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/03/18 20:14:25 UTC
[PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Dandandan opened a new pull request, #9574:
URL: https://github.com/apache/arrow-rs/pull/9574
## Summary
- Adds dictionary page pruning for row filter predicates in the parquet reader
- When evaluating predicates on dictionary-encoded columns, the predicate is first evaluated against dictionary values before decoding data pages
- If **no** dictionary values match (`AllFalse`): skip the entire column chunk
- If **all** dictionary values match (`AllTrue`): skip per-row predicate evaluation
- Adds `evaluate_dictionary` method to `ArrowPredicate` trait with a default implementation that delegates to `evaluate`
## Details
- Supports BYTE_ARRAY (strings), INT32, and INT64 physical types
- Only applies when all data pages use dictionary encoding (no plain fallback)
- Uses column encoding metadata and page encoding stats to verify safety
- Currently implemented for the async push decoder path
## Benchmark Results (ClickBench async_object_store)
| Query | Before | After | Change | Notes |
|-------|--------|-------|--------|-------|
| Q19 | 2.57ms | 1.66ms | **-35%** | `CounterID=62` — prunes 1 of 3 row groups |
| Q42 | 3.63ms | 3.35ms | **-8%** | Same CounterID filter |
| Q36 | 17.3ms | 16.7ms | **-3%** | CounterID + string predicates |
| Others | — | — | ~0% | No regressions |
The optimization is most effective for selective equality filters on dictionary-encoded columns where the target value doesn't appear in some row groups' dictionaries.
## Test plan
- [ ] Existing parquet tests pass
- [ ] ClickBench benchmark verifies correctness (row counts match expected)
- [ ] Tested with columns that have plain encoding fallback (correctly skips pruning)
- [ ] Structure size test unchanged (no new state machine variants)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4092443748
Is this similar to
- https://github.com/apache/arrow-rs/pull/9464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4172905627
BTW Codex says it found issues with this PR; I haven't had time to review them carefully but I would like to before we merge this. I hope to find time for this PR over the next few days
<details><summary>Details</summary>
<p>
• Findings
- High: LargeUtf8 filtering is broken, and LargeBinary is very likely broken for the same reason. In parquet/src/arrow/arrow_reader/dictionary_pruning.rs:177 the code
handles both Utf8 and LargeUtf8 by building a StringArray, and in parquet/src/arrow/arrow_reader/dictionary_pruning.rs:192 the fallback always builds a BinaryArray.
But the synthetic schema uses the real Arrow type at parquet/src/arrow/arrow_reader/dictionary_pruning.rs:123, so RecordBatch::try_new fails for LargeUtf8/
LargeBinary. I reproduced this in a disposable worktree with an async row-filter test; it panicked with: expected LargeUtf8 but found Utf8 at column index 0.
- High: the fallback “all pages are dictionary encoded” check is not safe and can return wrong query results on files without page encoding stats. The code explicitly
falls back to a heuristic in parquet/src/arrow/arrow_reader/dictionary_pruning.rs:234 and treats Encoding::PLAIN as acceptable at parquet/src/arrow/arrow_reader/
dictionary_pruning.rs:253. But PLAIN is exactly the ambiguous case for dictionary fallback pages. The metadata docs in parquet/src/file/metadata/mod.rs:1086 say
page encoding stats were added specifically to answer this question safely. If a column chunk contains both dictionary pages and plain fallback data pages and lacks
stats, this code can incorrectly prune rows.
- Medium: the new logic is not protected by meaningful tests. I reverted the PR’s implementation files in a temporary worktree and reran the existing decoder-focused
tests; they still passed. That means the current tests do not fail when dictionary pruning is removed. Coverage confirms this: target/llvm-cov/html/coverage/Users/
andrewlamb/Software/arrow-rs/parquet/src/arrow/arrow_reader/dictionary_pruning.rs.html is only 34.06% line-covered, with the BYTE_ARRAY path (target/llvm-cov/html/
coverage/Users/andrewlamb/Software/arrow-rs/parquet/src/arrow/arrow_reader/dictionary_pruning.rs.html:142), the full page-stats path (target/llvm-cov/html/coverage/
Users/andrewlamb/Software/arrow-rs/parquet/src/arrow/arrow_reader/dictionary_pruning.rs.html:217), the heuristic fallback (target/llvm-cov/html/coverage/Users/
andrewlamb/Software/arrow-rs/parquet/src/arrow/arrow_reader/dictionary_pruning.rs.html:240), and all INT32 conversion logic (target/llvm-cov/html/coverage/Users/
andrewlamb/Software/arrow-rs/parquet/src/arrow/arrow_reader/dictionary_pruning.rs.html:269) uncovered.
Public API: the PR adds ArrowPredicate::evaluate_dictionary and exports DictionaryPredicateResult from parquet/src/arrow/arrow_reader/filter.rs:25. That is a public
API expansion, but it does not look breaking because the new trait method has a default implementation.
Verification
I ran cargo llvm-cov --html test -p parquet; the full parquet suite passed and the report is under target/llvm-cov/html/index.html. I also did an ablative run in a
disposable worktree by reverting the PR’s implementation files and running cargo test -p parquet test_decoder_ -- --nocapture; all 13 decoder tests still passed.
Finally, I added a temporary repro test in another disposable worktree and confirmed the LargeUtf8 failure above.
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085358733
run benchmark arrow_reader_clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#discussion_r2967012443
##########
parquet/src/arrow/arrow_reader/dictionary_pruning.rs:
##########
@@ -0,0 +1,374 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Dictionary page pruning for row filters
+//!
+//! When a predicate targets a dictionary-encoded column, we can evaluate the
+//! predicate against the dictionary values before decoding any data pages.
+//! If no dictionary values match, the entire column chunk can be skipped.
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::array_reader::RowGroups;
+use crate::arrow::arrow_reader::ArrowPredicate;
+use crate::arrow::arrow_reader::filter::DictionaryPredicateResult;
+use crate::arrow::in_memory_row_group::InMemoryRowGroup;
+use crate::basic::{Encoding, Type as PhysicalType};
+use crate::errors::Result;
+use crate::file::metadata::ParquetMetaData;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use std::sync::Arc;
+
+/// Try to prune a predicate using dictionary pages from the in-memory row group.
+///
+/// Returns `Some(DictionaryPredicateResult)` if dictionary pruning was performed,
+/// `None` if the column is not dictionary-encoded or pruning is not applicable.
+pub(crate) fn try_dictionary_prune_in_memory(
+ predicate: &mut dyn ArrowPredicate,
+ row_group: &InMemoryRowGroup<'_>,
+ metadata: &ParquetMetaData,
+ fields: Option<&crate::arrow::schema::ParquetField>,
+) -> Result<Option<DictionaryPredicateResult>> {
+ let projection = predicate.projection();
+ let schema_descr = metadata.file_metadata().schema_descr();
+
+ // Only support single-column predicates
+ let col_idx = single_leaf_column(projection, schema_descr.num_columns());
+ let Some(col_idx) = col_idx else {
+ return Ok(None);
+ };
+
+ let row_group_meta = metadata.row_group(row_group.row_group_idx);
+ let col_meta = row_group_meta.column(col_idx);
+
+ // Only proceed if the column has a dictionary page
+ if col_meta.dictionary_page_offset().is_none() {
+ return Ok(None);
+ }
+
+ // Only safe to prune if ALL data pages are dictionary-encoded.
+ // If some pages fell back to plain encoding, the dictionary doesn't
+ // cover all values and we can't safely skip based on dictionary alone.
+ if !is_all_dictionary_encoded(col_meta) {
+ return Ok(None);
+ }
+
+ let physical_type = schema_descr.column(col_idx).physical_type();
+
+ // Only support BYTE_ARRAY and INT32/INT64 columns
+ if !matches!(
+ physical_type,
+ PhysicalType::BYTE_ARRAY | PhysicalType::INT32 | PhysicalType::INT64
+ ) {
+ return Ok(None);
+ }
+
+ // Get the arrow type for this column from the ParquetField tree.
+ // Only supports top-level primitive columns (not nested in structs/lists).
+ let Some(arrow_type) = fields.and_then(|f| find_top_level_leaf_arrow_type(f, col_idx)) else {
+ return Ok(None);
+ };
+
+ // Create a page reader for this column
+ let mut page_iter = row_group.column_chunks(col_idx)?;
+ let Some(page_reader) = page_iter.next() else {
+ return Ok(None);
+ };
+ let mut page_reader = page_reader?;
+
+ // Read the first page - should be the dictionary page
+ let first_page = page_reader.get_next_page()?;
+ let Some(page) = first_page else {
+ return Ok(None);
+ };
+
+ if !page.is_dictionary_page() {
+ return Ok(None);
+ }
+
+ let crate::column::page::Page::DictionaryPage {
+ buf, num_values, ..
+ } = page
+ else {
+ return Ok(None);
+ };
+
+ // Decode PLAIN-encoded dictionary values based on physical type,
+ // then cast to the target arrow type if needed
+ let array: ArrayRef = match physical_type {
+ PhysicalType::BYTE_ARRAY => {
+ decode_plain_byte_array(&buf, num_values as usize, &arrow_type)?
+ }
+ PhysicalType::INT32 => decode_plain_int32_as(&buf, num_values as usize, &arrow_type)?,
+ PhysicalType::INT64 => decode_plain_int64_as(&buf, num_values as usize, &arrow_type)?,
+ _ => return Ok(None),
+ };
+
+ // Build a RecordBatch with the dictionary values using a synthetic field
+ let col_name = schema_descr.column(col_idx).name().to_string();
+ let field = Field::new(&col_name, arrow_type, true);
+ let schema = Arc::new(Schema::new(vec![field]));
+ let batch = RecordBatch::try_new(schema, vec![array]).map_err(|e| {
+ crate::errors::ParquetError::General(format!("Failed to create dictionary batch: {}", e))
+ })?;
+
+ // Evaluate the predicate against dictionary values
+ let result = predicate.evaluate_dictionary(batch).map_err(|e| {
+ crate::errors::ParquetError::General(format!(
+ "Failed to evaluate dictionary predicate: {}",
+ e
+ ))
+ })?;
+
+ Ok(Some(result))
+}
+
+/// Decode PLAIN-encoded BYTE_ARRAY values into a string/binary array
+/// matching the target arrow type.
+fn decode_plain_byte_array(
+ buf: &[u8],
+ num_values: usize,
+ arrow_type: &DataType,
+) -> Result<ArrayRef> {
+ // Parse all byte array values
+ let mut values: Vec<&[u8]> = Vec::with_capacity(num_values);
+ let mut offset = 0;
+ for _ in 0..num_values {
+ if offset + 4 > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ let len = u32::from_le_bytes(buf[offset..offset + 4].try_into().unwrap()) as usize;
+ offset += 4;
+ if offset + len > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ values.push(&buf[offset..offset + len]);
+ offset += len;
+ }
+
+ match arrow_type {
+ DataType::Utf8View => {
+ let mut builder = arrow_array::builder::StringViewBuilder::with_capacity(num_values);
+ for v in &values {
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ let s = unsafe { std::str::from_utf8_unchecked(v) };
+ builder.append_value(s);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ let strs: Vec<&str> = values
+ .iter()
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ .map(|v| unsafe { std::str::from_utf8_unchecked(v) })
+ .collect();
+ Ok(Arc::new(arrow_array::StringArray::from(strs)))
+ }
+ DataType::BinaryView => {
+ let mut builder = arrow_array::builder::BinaryViewBuilder::with_capacity(num_values);
+ for v in &values {
+ builder.append_value(v);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ _ => {
+ // Default to BinaryArray for unknown types
+ let binary_values: Vec<&[u8]> = values;
+ Ok(Arc::new(arrow_array::BinaryArray::from(binary_values)))
+ }
+ }
+}
+
+/// Check if all data pages in a column chunk are dictionary-encoded.
+///
+/// Uses page encoding stats if available, otherwise falls back to checking
+/// column-level encodings. Returns false if we can't determine conclusively.
+fn is_all_dictionary_encoded(col_meta: &crate::file::metadata::ColumnChunkMetaData) -> bool {
+ // No dictionary page -> definitely not all dictionary encoded
+ if col_meta.dictionary_page_offset().is_none() {
+ return false;
+ }
+
+ // Method 1: Use page encoding stats mask if available (most reliable)
+ if let Some(mask) = col_meta.page_encoding_stats_mask() {
+ return mask.is_only(Encoding::PLAIN_DICTIONARY) || mask.is_only(Encoding::RLE_DICTIONARY);
+ }
+
+ // Method 1a: Use full page encoding stats if the mask form wasn't used
+ // (i.e. ParquetMetaDataOptions::with_encoding_stats_as_mask was set to false)
+ if let Some(stats) = col_meta.page_encoding_stats() {
+ return stats
+ .iter()
+ .filter(|s| {
+ matches!(
+ s.page_type,
+ crate::basic::PageType::DATA_PAGE | crate::basic::PageType::DATA_PAGE_V2
+ )
+ })
+ .all(|s| {
+ matches!(
+ s.encoding,
+ Encoding::PLAIN_DICTIONARY | Encoding::RLE_DICTIONARY
+ )
+ });
+ }
+
+ // Method 2: Check column-level encodings.
+ // Dictionary-encoded columns have PLAIN_DICTIONARY or RLE_DICTIONARY for data,
+ // plus PLAIN and/or RLE for definition/repetition levels.
+ // If PLAIN appears as a data encoding (fallback), the column also has
+ // PLAIN_DICTIONARY/RLE_DICTIONARY, so we'd see both.
+ // Safe heuristic: only dictionary + level encodings = all dict encoded.
+ let mut has_dict_data_encoding = false;
+ let mut has_non_dict_data_encoding = false;
+ for enc in col_meta.encodings() {
+ match enc {
+ Encoding::PLAIN_DICTIONARY | Encoding::RLE_DICTIONARY => {
+ has_dict_data_encoding = true;
+ }
+ // These are used for definition/repetition levels, not data
+ #[allow(deprecated)]
+ Encoding::RLE | Encoding::BIT_PACKED => {}
+ // PLAIN is ambiguous - used for def/rep levels in V1 pages AND
Review Comment:
or more simply:
```rust
for enc in col_meta.encodings() {
match enc {
#[allow(deprecated)]
// Either V1 dict encoded or level data
Encoding::PLAIN_DICTIONARY | Encoding::RLE | Encoding::BIT_PACKED => {}
// Any other encoding means non-dictionary data
_ => return false,
}
}
true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#discussion_r2966837415
##########
parquet/src/arrow/arrow_reader/dictionary_pruning.rs:
##########
@@ -0,0 +1,374 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Dictionary page pruning for row filters
+//!
+//! When a predicate targets a dictionary-encoded column, we can evaluate the
+//! predicate against the dictionary values before decoding any data pages.
+//! If no dictionary values match, the entire column chunk can be skipped.
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::array_reader::RowGroups;
+use crate::arrow::arrow_reader::ArrowPredicate;
+use crate::arrow::arrow_reader::filter::DictionaryPredicateResult;
+use crate::arrow::in_memory_row_group::InMemoryRowGroup;
+use crate::basic::{Encoding, Type as PhysicalType};
+use crate::errors::Result;
+use crate::file::metadata::ParquetMetaData;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use std::sync::Arc;
+
+/// Try to prune a predicate using dictionary pages from the in-memory row group.
+///
+/// Returns `Some(DictionaryPredicateResult)` if dictionary pruning was performed,
+/// `None` if the column is not dictionary-encoded or pruning is not applicable.
+pub(crate) fn try_dictionary_prune_in_memory(
+ predicate: &mut dyn ArrowPredicate,
+ row_group: &InMemoryRowGroup<'_>,
+ metadata: &ParquetMetaData,
+ fields: Option<&crate::arrow::schema::ParquetField>,
+) -> Result<Option<DictionaryPredicateResult>> {
+ let projection = predicate.projection();
+ let schema_descr = metadata.file_metadata().schema_descr();
+
+ // Only support single-column predicates
+ let col_idx = single_leaf_column(projection, schema_descr.num_columns());
+ let Some(col_idx) = col_idx else {
+ return Ok(None);
+ };
+
+ let row_group_meta = metadata.row_group(row_group.row_group_idx);
+ let col_meta = row_group_meta.column(col_idx);
+
+ // Only proceed if the column has a dictionary page
+ if col_meta.dictionary_page_offset().is_none() {
+ return Ok(None);
+ }
+
+ // Only safe to prune if ALL data pages are dictionary-encoded.
+ // If some pages fell back to plain encoding, the dictionary doesn't
+ // cover all values and we can't safely skip based on dictionary alone.
+ if !is_all_dictionary_encoded(col_meta) {
+ return Ok(None);
+ }
+
+ let physical_type = schema_descr.column(col_idx).physical_type();
+
+ // Only support BYTE_ARRAY and INT32/INT64 columns
+ if !matches!(
+ physical_type,
+ PhysicalType::BYTE_ARRAY | PhysicalType::INT32 | PhysicalType::INT64
+ ) {
+ return Ok(None);
+ }
+
+ // Get the arrow type for this column from the ParquetField tree.
+ // Only supports top-level primitive columns (not nested in structs/lists).
+ let Some(arrow_type) = fields.and_then(|f| find_top_level_leaf_arrow_type(f, col_idx)) else {
+ return Ok(None);
+ };
+
+ // Create a page reader for this column
+ let mut page_iter = row_group.column_chunks(col_idx)?;
+ let Some(page_reader) = page_iter.next() else {
+ return Ok(None);
+ };
+ let mut page_reader = page_reader?;
+
+ // Read the first page - should be the dictionary page
+ let first_page = page_reader.get_next_page()?;
+ let Some(page) = first_page else {
+ return Ok(None);
+ };
+
+ if !page.is_dictionary_page() {
+ return Ok(None);
+ }
+
+ let crate::column::page::Page::DictionaryPage {
+ buf, num_values, ..
+ } = page
+ else {
+ return Ok(None);
+ };
+
+ // Decode PLAIN-encoded dictionary values based on physical type,
+ // then cast to the target arrow type if needed
+ let array: ArrayRef = match physical_type {
+ PhysicalType::BYTE_ARRAY => {
+ decode_plain_byte_array(&buf, num_values as usize, &arrow_type)?
+ }
+ PhysicalType::INT32 => decode_plain_int32_as(&buf, num_values as usize, &arrow_type)?,
+ PhysicalType::INT64 => decode_plain_int64_as(&buf, num_values as usize, &arrow_type)?,
+ _ => return Ok(None),
+ };
+
+ // Build a RecordBatch with the dictionary values using a synthetic field
+ let col_name = schema_descr.column(col_idx).name().to_string();
+ let field = Field::new(&col_name, arrow_type, true);
+ let schema = Arc::new(Schema::new(vec![field]));
+ let batch = RecordBatch::try_new(schema, vec![array]).map_err(|e| {
+ crate::errors::ParquetError::General(format!("Failed to create dictionary batch: {}", e))
+ })?;
+
+ // Evaluate the predicate against dictionary values
+ let result = predicate.evaluate_dictionary(batch).map_err(|e| {
+ crate::errors::ParquetError::General(format!(
+ "Failed to evaluate dictionary predicate: {}",
+ e
+ ))
+ })?;
+
+ Ok(Some(result))
+}
+
+/// Decode PLAIN-encoded BYTE_ARRAY values into a string/binary array
+/// matching the target arrow type.
+fn decode_plain_byte_array(
+ buf: &[u8],
+ num_values: usize,
+ arrow_type: &DataType,
+) -> Result<ArrayRef> {
+ // Parse all byte array values
+ let mut values: Vec<&[u8]> = Vec::with_capacity(num_values);
+ let mut offset = 0;
+ for _ in 0..num_values {
+ if offset + 4 > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ let len = u32::from_le_bytes(buf[offset..offset + 4].try_into().unwrap()) as usize;
+ offset += 4;
+ if offset + len > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ values.push(&buf[offset..offset + len]);
+ offset += len;
+ }
+
+ match arrow_type {
+ DataType::Utf8View => {
+ let mut builder = arrow_array::builder::StringViewBuilder::with_capacity(num_values);
+ for v in &values {
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ let s = unsafe { std::str::from_utf8_unchecked(v) };
+ builder.append_value(s);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ let strs: Vec<&str> = values
+ .iter()
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ .map(|v| unsafe { std::str::from_utf8_unchecked(v) })
+ .collect();
+ Ok(Arc::new(arrow_array::StringArray::from(strs)))
+ }
+ DataType::BinaryView => {
+ let mut builder = arrow_array::builder::BinaryViewBuilder::with_capacity(num_values);
+ for v in &values {
+ builder.append_value(v);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ _ => {
+ // Default to BinaryArray for unknown types
+ let binary_values: Vec<&[u8]> = values;
+ Ok(Arc::new(arrow_array::BinaryArray::from(binary_values)))
+ }
+ }
+}
+
+/// Check if all data pages in a column chunk are dictionary-encoded.
+///
+/// Uses page encoding stats if available, otherwise falls back to checking
+/// column-level encodings. Returns false if we can't determine conclusively.
+fn is_all_dictionary_encoded(col_meta: &crate::file::metadata::ColumnChunkMetaData) -> bool {
+ // No dictionary page -> definitely not all dictionary encoded
+ if col_meta.dictionary_page_offset().is_none() {
+ return false;
+ }
+
+ // Method 1: Use page encoding stats mask if available (most reliable)
+ if let Some(mask) = col_meta.page_encoding_stats_mask() {
+ return mask.is_only(Encoding::PLAIN_DICTIONARY) || mask.is_only(Encoding::RLE_DICTIONARY);
+ }
+
+ // Method 1a: Use full page encoding stats if the mask form wasn't used
+ // (i.e. ParquetMetaDataOptions::with_encoding_stats_as_mask was set to false)
+ if let Some(stats) = col_meta.page_encoding_stats() {
+ return stats
+ .iter()
+ .filter(|s| {
+ matches!(
+ s.page_type,
+ crate::basic::PageType::DATA_PAGE | crate::basic::PageType::DATA_PAGE_V2
+ )
+ })
+ .all(|s| {
+ matches!(
+ s.encoding,
+ Encoding::PLAIN_DICTIONARY | Encoding::RLE_DICTIONARY
+ )
+ });
+ }
+
+ // Method 2: Check column-level encodings.
+ // Dictionary-encoded columns have PLAIN_DICTIONARY or RLE_DICTIONARY for data,
+ // plus PLAIN and/or RLE for definition/repetition levels.
+ // If PLAIN appears as a data encoding (fallback), the column also has
+ // PLAIN_DICTIONARY/RLE_DICTIONARY, so we'd see both.
+ // Safe heuristic: only dictionary + level encodings = all dict encoded.
+ let mut has_dict_data_encoding = false;
+ let mut has_non_dict_data_encoding = false;
+ for enc in col_meta.encodings() {
+ match enc {
+ Encoding::PLAIN_DICTIONARY | Encoding::RLE_DICTIONARY => {
+ has_dict_data_encoding = true;
+ }
+ // These are used for definition/repetition levels, not data
+ #[allow(deprecated)]
+ Encoding::RLE | Encoding::BIT_PACKED => {}
+ // PLAIN is ambiguous - used for def/rep levels in V1 pages AND
Review Comment:
I'm not so sure about this one. So the way the encodings work is:
- levels are RLE or BIT_PACKED (as noted above)
- V1 dictionary uses the PLAIN_DICTIONARY variant for both the dictionary and data
- V2 dictionary uses PLAIN for dictionary and RLE_DICTIONARY for data
- all other encodings are for data
So if we see PLAIN_DICTIONARY and PLAIN, we can know that fallback has definitely occurred. In the case of RLE_DICTIONARY, it's a coin flip, but I'd err on the side of caution and return `false` in that case. In my experience fallback is quite common.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085372199
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085358733)
`Linux bench-c4085358733-437-vzs9l 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing dictionary-page-pruning (fe5dc07eb90425c3d55e0dce15dee50750570d9f) to 88422cb (merge-base) [diff](https://github.com/apache/arrow-rs/compare/88422cbdcbfa8f4e2411d66578dd3582fafbf2a1..fe5dc07eb90425c3d55e0dce15dee50750570d9f)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#discussion_r2956560233
##########
parquet/src/arrow/arrow_reader/dictionary_pruning.rs:
##########
@@ -0,0 +1,356 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Dictionary page pruning for row filters
+//!
+//! When a predicate targets a dictionary-encoded column, we can evaluate the
+//! predicate against the dictionary values before decoding any data pages.
+//! If no dictionary values match, the entire column chunk can be skipped.
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::array_reader::RowGroups;
+use crate::arrow::arrow_reader::ArrowPredicate;
+use crate::arrow::arrow_reader::filter::DictionaryPredicateResult;
+use crate::arrow::in_memory_row_group::InMemoryRowGroup;
+use crate::basic::{Encoding, Type as PhysicalType};
+use crate::errors::Result;
+use crate::file::metadata::ParquetMetaData;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use std::sync::Arc;
+
+/// Try to prune a predicate using dictionary pages from the in-memory row group.
+///
+/// Returns `Some(DictionaryPredicateResult)` if dictionary pruning was performed,
+/// `None` if the column is not dictionary-encoded or pruning is not applicable.
+pub(crate) fn try_dictionary_prune_in_memory(
+ predicate: &mut dyn ArrowPredicate,
+ row_group: &InMemoryRowGroup<'_>,
+ metadata: &ParquetMetaData,
+ fields: Option<&crate::arrow::schema::ParquetField>,
+) -> Result<Option<DictionaryPredicateResult>> {
+ let projection = predicate.projection();
+ let schema_descr = metadata.file_metadata().schema_descr();
+
+ // Only support single-column predicates
+ let col_idx = single_leaf_column(projection, schema_descr.num_columns());
+ let Some(col_idx) = col_idx else {
+ return Ok(None);
+ };
+
+ let row_group_meta = metadata.row_group(row_group.row_group_idx);
+ let col_meta = row_group_meta.column(col_idx);
+
+ // Only proceed if the column has a dictionary page
+ if col_meta.dictionary_page_offset().is_none() {
+ return Ok(None);
+ }
+
+ // Only safe to prune if ALL data pages are dictionary-encoded.
+ // If some pages fell back to plain encoding, the dictionary doesn't
+ // cover all values and we can't safely skip based on dictionary alone.
+ if !is_all_dictionary_encoded(col_meta) {
+ return Ok(None);
+ }
+
+ let physical_type = schema_descr.column(col_idx).physical_type();
+
+ // Only support BYTE_ARRAY and INT32/INT64 columns
+ if !matches!(
+ physical_type,
+ PhysicalType::BYTE_ARRAY | PhysicalType::INT32 | PhysicalType::INT64
+ ) {
+ return Ok(None);
+ }
+
+ // Get the arrow type for this column from the ParquetField tree.
+ // Only supports top-level primitive columns (not nested in structs/lists).
+ let Some(arrow_type) = fields.and_then(|f| find_top_level_leaf_arrow_type(f, col_idx))
+ else {
+ return Ok(None);
+ };
+
+ // Create a page reader for this column
+ let mut page_iter = row_group.column_chunks(col_idx)?;
+ let Some(page_reader) = page_iter.next() else {
+ return Ok(None);
+ };
+ let mut page_reader = page_reader?;
+
+ // Read the first page - should be the dictionary page
+ let first_page = page_reader.get_next_page()?;
+ let Some(page) = first_page else {
+ return Ok(None);
+ };
+
+ if !page.is_dictionary_page() {
+ return Ok(None);
+ }
+
+ let crate::column::page::Page::DictionaryPage {
+ buf, num_values, ..
+ } = page
+ else {
+ return Ok(None);
+ };
+
+ // Decode PLAIN-encoded dictionary values based on physical type,
+ // then cast to the target arrow type if needed
+ let array: ArrayRef = match physical_type {
+ PhysicalType::BYTE_ARRAY => {
+ decode_plain_byte_array(&buf, num_values as usize, &arrow_type)?
+ }
+ PhysicalType::INT32 => decode_plain_int32_as(&buf, num_values as usize, &arrow_type)?,
+ PhysicalType::INT64 => decode_plain_int64_as(&buf, num_values as usize, &arrow_type)?,
+ _ => return Ok(None),
+ };
+
+ // Build a RecordBatch with the dictionary values using a synthetic field
+ let col_name = schema_descr.column(col_idx).name().to_string();
+ let field = Field::new(&col_name, arrow_type, true);
+ let schema = Arc::new(Schema::new(vec![field]));
+ let batch = RecordBatch::try_new(schema, vec![array]).map_err(|e| {
+ crate::errors::ParquetError::General(format!("Failed to create dictionary batch: {}", e))
+ })?;
+
+ // Evaluate the predicate against dictionary values
+ let result = predicate.evaluate_dictionary(batch).map_err(|e| {
+ crate::errors::ParquetError::General(format!(
+ "Failed to evaluate dictionary predicate: {}",
+ e
+ ))
+ })?;
+
+ Ok(Some(result))
+}
+
+/// Decode PLAIN-encoded BYTE_ARRAY values into a string/binary array
+/// matching the target arrow type.
+fn decode_plain_byte_array(
+ buf: &[u8],
+ num_values: usize,
+ arrow_type: &DataType,
+) -> Result<ArrayRef> {
+ // Parse all byte array values
+ let mut values: Vec<&[u8]> = Vec::with_capacity(num_values);
+ let mut offset = 0;
+ for _ in 0..num_values {
+ if offset + 4 > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ let len = u32::from_le_bytes(buf[offset..offset + 4].try_into().unwrap()) as usize;
+ offset += 4;
+ if offset + len > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ values.push(&buf[offset..offset + len]);
+ offset += len;
+ }
+
+ match arrow_type {
+ DataType::Utf8View => {
+ let mut builder = arrow_array::builder::StringViewBuilder::with_capacity(num_values);
+ for v in &values {
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ let s = unsafe { std::str::from_utf8_unchecked(v) };
+ builder.append_value(s);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ let strs: Vec<&str> = values
+ .iter()
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ .map(|v| unsafe { std::str::from_utf8_unchecked(v) })
+ .collect();
+ Ok(Arc::new(arrow_array::StringArray::from(strs)))
+ }
+ DataType::BinaryView => {
+ let mut builder = arrow_array::builder::BinaryViewBuilder::with_capacity(num_values);
+ for v in &values {
+ builder.append_value(v);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ _ => {
+ // Default to BinaryArray for unknown types
+ let binary_values: Vec<&[u8]> = values;
+ Ok(Arc::new(arrow_array::BinaryArray::from(binary_values)))
+ }
+ }
+}
+
+/// Check if all data pages in a column chunk are dictionary-encoded.
+///
+/// Uses page encoding stats if available, otherwise falls back to checking
+/// column-level encodings. Returns false if we can't determine conclusively.
+fn is_all_dictionary_encoded(col_meta: &crate::file::metadata::ColumnChunkMetaData) -> bool {
+ // No dictionary page -> definitely not all dictionary encoded
+ if col_meta.dictionary_page_offset().is_none() {
+ return false;
+ }
+
+ // Method 1: Use page encoding stats mask if available (most reliable)
+ if let Some(mask) = col_meta.page_encoding_stats_mask() {
+ return mask.is_only(Encoding::PLAIN_DICTIONARY) || mask.is_only(Encoding::RLE_DICTIONARY);
+ }
Review Comment:
Method 1a: check `col_meta.page_encoding_stats()` if the mask form wasn't used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#discussion_r2962058562
##########
parquet/tests/arrow_reader/io/async_reader.rs:
##########
@@ -275,9 +275,7 @@ async fn test_read_multiple_row_filter() {
"Read Multi:",
" Row Group 1, column 'a': MultiPage(dictionary_page: true, data_pages: [0, 1]) (1856 bytes, 1 requests) [data]",
"Read Multi:",
- " Row Group 1, column 'b': DictionaryPage (1617 bytes, 1 requests) [data]",
- " Row Group 1, column 'b': DataPage(0) (113 bytes , 1 requests) [data]",
- " Row Group 1, column 'b': DataPage(1) (126 bytes , 1 requests) [data]",
+ " Row Group 1, column 'b': MultiPage(dictionary_page: true, data_pages: [0, 1]) (1856 bytes, 1 requests) [data]",
Review Comment:
Yes :)
I think for object storage it will still coalesce the ranges afterwards - but on localfilesystem it will do a number of syscalls (which as it does it one after another should be less efficient then one go anyway).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085313791
run benchmark arrow_reader_clickbench
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#discussion_r2958349157
##########
parquet/tests/arrow_reader/io/async_reader.rs:
##########
@@ -275,9 +275,7 @@ async fn test_read_multiple_row_filter() {
"Read Multi:",
" Row Group 1, column 'a': MultiPage(dictionary_page: true, data_pages: [0, 1]) (1856 bytes, 1 requests) [data]",
"Read Multi:",
- " Row Group 1, column 'b': DictionaryPage (1617 bytes, 1 requests) [data]",
- " Row Group 1, column 'b': DataPage(0) (113 bytes , 1 requests) [data]",
- " Row Group 1, column 'b': DataPage(1) (126 bytes , 1 requests) [data]",
+ " Row Group 1, column 'b': MultiPage(dictionary_page: true, data_pages: [0, 1]) (1856 bytes, 1 requests) [data]",
Review Comment:
Was a bit surprised to see this changed (it's more optimal).
Seems when all values pass it creates 3 requests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4192496585
I'll try to give this some attention soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085331368
Benchmark for [this request](https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085313791) failed.
Last 20 lines of output:
<details><summary>Click to expand</summary>
```
Updating crates.io index
Locking 417 packages to latest compatible versions
Adding generic-array v0.14.7 (available: v0.14.9)
Adding lz4_flex v0.12.1 (available: v0.13.0)
Adding matchit v0.8.4 (available: v0.8.6)
Adding rand v0.9.2 (available: v0.10.0)
rustc 1.94.0 (4a4ef493e 2026-03-02)
d2d1d28adf23dcb75dbb2ae0c01b02b9410d05ac
393117979882e97a15125edd142c70a5e2c16386
Looking for ClickBench files starting in current_dir and all parent directories: "/workspace/arrow-rs-base/parquet"
Finished `bench` profile [optimized] target(s) in 0.15s
Running benches/arrow_reader_clickbench.rs (target/release/deps/arrow_reader_clickbench-783851a0a179a92e)
Could not find hits_1.parquet in directory or parents: "/workspace/arrow-rs-base/parquet". Download it via
wget --continue https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet
thread 'main' (15622) panicked at parquet/benches/arrow_reader_clickbench.rs:618:9:
Stopping
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: bench failed, to rerun pass `-p parquet --bench arrow_reader_clickbench`
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085512968
🤖 Arrow criterion benchmark completed (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085358733)
<details><summary>Details</summary>
<p>
```
group dictionary-page-pruning main
----- ----------------------- ----
arrow_reader_clickbench/async/Q1 1.00 1087.0±6.86µs ? ?/sec 1.01 1093.3±3.29µs ? ?/sec
arrow_reader_clickbench/async/Q10 1.00 6.7±0.03ms ? ?/sec 1.01 6.8±0.05ms ? ?/sec
arrow_reader_clickbench/async/Q11 1.00 7.7±0.03ms ? ?/sec 1.01 7.8±0.06ms ? ?/sec
arrow_reader_clickbench/async/Q12 1.00 14.3±0.04ms ? ?/sec 1.03 14.7±0.11ms ? ?/sec
arrow_reader_clickbench/async/Q13 1.00 16.9±0.06ms ? ?/sec 1.03 17.4±0.10ms ? ?/sec
arrow_reader_clickbench/async/Q14 1.00 15.8±0.04ms ? ?/sec 1.02 16.2±0.08ms ? ?/sec
arrow_reader_clickbench/async/Q19 1.00 1976.5±21.72µs ? ?/sec 1.55 3.1±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q20 1.07 82.6±9.34ms ? ?/sec 1.00 77.1±4.56ms ? ?/sec
arrow_reader_clickbench/async/Q21 1.00 80.2±0.19ms ? ?/sec 1.08 86.8±0.65ms ? ?/sec
arrow_reader_clickbench/async/Q22 1.00 122.8±4.28ms ? ?/sec 1.06 130.3±4.26ms ? ?/sec
arrow_reader_clickbench/async/Q23 1.00 237.4±0.83ms ? ?/sec 1.05 248.7±1.09ms ? ?/sec
arrow_reader_clickbench/async/Q24 1.00 19.3±0.11ms ? ?/sec 1.03 19.9±0.17ms ? ?/sec
arrow_reader_clickbench/async/Q27 1.00 57.1±0.19ms ? ?/sec 1.02 58.3±0.21ms ? ?/sec
arrow_reader_clickbench/async/Q28 1.00 57.0±0.18ms ? ?/sec 1.05 59.6±0.28ms ? ?/sec
arrow_reader_clickbench/async/Q30 1.00 18.4±0.07ms ? ?/sec 1.02 18.8±0.09ms ? ?/sec
arrow_reader_clickbench/async/Q36 1.00 14.6±0.16ms ? ?/sec 1.03 15.1±0.14ms ? ?/sec
arrow_reader_clickbench/async/Q37 1.12 6.1±0.03ms ? ?/sec 1.00 5.4±0.01ms ? ?/sec
arrow_reader_clickbench/async/Q38 1.00 12.9±0.13ms ? ?/sec 1.05 13.5±0.15ms ? ?/sec
arrow_reader_clickbench/async/Q39 1.00 23.8±0.22ms ? ?/sec 1.04 24.9±0.32ms ? ?/sec
arrow_reader_clickbench/async/Q40 1.00 5.3±0.03ms ? ?/sec 1.08 5.7±0.04ms ? ?/sec
arrow_reader_clickbench/async/Q41 1.00 4.6±0.02ms ? ?/sec 1.07 5.0±0.03ms ? ?/sec
arrow_reader_clickbench/async/Q42 1.00 3.2±0.02ms ? ?/sec 1.09 3.5±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q1 1.00 1067.9±6.11µs ? ?/sec 1.00 1068.9±3.07µs ? ?/sec
arrow_reader_clickbench/async_object_store/Q10 1.00 6.5±0.02ms ? ?/sec 1.01 6.6±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q11 1.00 7.5±0.05ms ? ?/sec 1.01 7.6±0.05ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q12 1.00 14.3±0.05ms ? ?/sec 1.02 14.6±0.12ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q13 1.00 16.8±0.10ms ? ?/sec 1.03 17.2±0.14ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q14 1.00 15.8±0.13ms ? ?/sec 1.02 16.1±0.09ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q19 1.00 1884.4±16.61µs ? ?/sec 1.55 2.9±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q20 1.00 70.7±0.22ms ? ?/sec 1.01 71.7±0.42ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q21 1.00 79.1±0.22ms ? ?/sec 1.02 80.8±0.54ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q22 1.00 98.2±1.92ms ? ?/sec 1.00 98.6±0.58ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q23 1.00 217.3±0.63ms ? ?/sec 1.08 234.1±1.15ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q24 1.00 19.1±0.11ms ? ?/sec 1.01 19.3±0.13ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q27 1.00 56.1±0.20ms ? ?/sec 1.02 57.1±0.46ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q28 1.00 56.1±0.17ms ? ?/sec 1.03 57.9±0.72ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q30 1.00 18.2±0.06ms ? ?/sec 1.02 18.5±0.11ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q36 1.00 14.6±2.10ms ? ?/sec 1.00 14.6±0.15ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q37 1.12 6.0±0.02ms ? ?/sec 1.00 5.4±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q38 1.00 12.5±0.15ms ? ?/sec 1.02 12.8±0.15ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q39 1.00 22.9±0.19ms ? ?/sec 1.02 23.4±0.30ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q40 1.00 5.1±0.03ms ? ?/sec 1.06 5.5±0.03ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q41 1.00 4.5±0.03ms ? ?/sec 1.06 4.8±0.02ms ? ?/sec
arrow_reader_clickbench/async_object_store/Q42 1.00 3.2±0.02ms ? ?/sec 1.08 3.4±0.01ms ? ?/sec
arrow_reader_clickbench/sync/Q1 1.00 872.1±2.46µs ? ?/sec 1.00 869.0±1.88µs ? ?/sec
arrow_reader_clickbench/sync/Q10 1.00 5.1±0.02ms ? ?/sec 1.02 5.1±0.04ms ? ?/sec
arrow_reader_clickbench/sync/Q11 1.00 6.0±0.03ms ? ?/sec 1.02 6.1±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q12 1.00 21.6±0.14ms ? ?/sec 1.04 22.6±0.64ms ? ?/sec
arrow_reader_clickbench/sync/Q13 1.02 29.5±0.12ms ? ?/sec 1.00 28.9±0.81ms ? ?/sec
arrow_reader_clickbench/sync/Q14 1.14 26.8±0.37ms ? ?/sec 1.00 23.4±0.08ms ? ?/sec
arrow_reader_clickbench/sync/Q19 1.00 2.7±0.04ms ? ?/sec 1.01 2.7±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q20 1.00 120.5±0.23ms ? ?/sec 1.01 122.0±1.54ms ? ?/sec
arrow_reader_clickbench/sync/Q21 1.00 96.4±0.11ms ? ?/sec 1.02 98.0±0.14ms ? ?/sec
arrow_reader_clickbench/sync/Q22 1.00 142.4±0.67ms ? ?/sec 1.02 145.1±0.34ms ? ?/sec
arrow_reader_clickbench/sync/Q23 1.00 276.9±14.50ms ? ?/sec 1.02 281.8±13.65ms ? ?/sec
arrow_reader_clickbench/sync/Q24 1.00 26.9±0.07ms ? ?/sec 1.02 27.6±0.10ms ? ?/sec
arrow_reader_clickbench/sync/Q27 1.00 106.1±0.11ms ? ?/sec 1.02 108.1±0.16ms ? ?/sec
arrow_reader_clickbench/sync/Q28 1.00 103.8±0.16ms ? ?/sec 1.04 107.8±0.27ms ? ?/sec
arrow_reader_clickbench/sync/Q30 1.00 18.5±0.06ms ? ?/sec 1.03 19.1±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q36 1.00 22.5±0.20ms ? ?/sec 1.00 22.4±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q37 1.00 6.9±0.01ms ? ?/sec 1.00 6.9±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q38 1.00 11.4±0.05ms ? ?/sec 1.00 11.5±0.05ms ? ?/sec
arrow_reader_clickbench/sync/Q39 1.00 20.9±0.05ms ? ?/sec 1.00 21.0±0.07ms ? ?/sec
arrow_reader_clickbench/sync/Q40 1.00 5.1±0.02ms ? ?/sec 1.02 5.2±0.03ms ? ?/sec
arrow_reader_clickbench/sync/Q41 1.00 5.5±0.02ms ? ?/sec 1.01 5.6±0.02ms ? ?/sec
arrow_reader_clickbench/sync/Q42 1.00 4.3±0.02ms ? ?/sec 1.00 4.3±0.02ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 786.3s |
| Peak memory | 3.1 GiB |
| Avg memory | 2.9 GiB |
| CPU user | 706.4s |
| CPU sys | 79.2s |
| Disk read | 0 B |
| Disk write | 846.8 MiB |
**branch**
| Metric | Value |
|--------|-------|
| Wall time | 776.9s |
| Peak memory | 3.2 GiB |
| Avg memory | 3.1 GiB |
| CPU user | 705.8s |
| CPU sys | 70.3s |
| Disk read | 0 B |
| Disk write | 171.2 MiB |
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085605561
> arrow_reader_clickbench/async_object_store/Q19 1.00 1884.4±16.61µs ? ?/sec 1.55 2.9±0.03ms ? ?/sec
:tada:
We could (at the cost of extra IO) also do this before loading the column chunks I think -
but probably needs to be some config.
(Perhaps it makes sense for other row filters as well to disable making IO requests small/sequential (especially object storage): don't try to save IO (for small/medium sized columns) but still try to prune to save CPU.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#discussion_r2967253381
##########
parquet/src/arrow/arrow_reader/dictionary_pruning.rs:
##########
@@ -0,0 +1,374 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Dictionary page pruning for row filters
+//!
+//! When a predicate targets a dictionary-encoded column, we can evaluate the
+//! predicate against the dictionary values before decoding any data pages.
+//! If no dictionary values match, the entire column chunk can be skipped.
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::array_reader::RowGroups;
+use crate::arrow::arrow_reader::ArrowPredicate;
+use crate::arrow::arrow_reader::filter::DictionaryPredicateResult;
+use crate::arrow::in_memory_row_group::InMemoryRowGroup;
+use crate::basic::{Encoding, Type as PhysicalType};
+use crate::errors::Result;
+use crate::file::metadata::ParquetMetaData;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use std::sync::Arc;
+
+/// Try to prune a predicate using dictionary pages from the in-memory row group.
+///
+/// Returns `Some(DictionaryPredicateResult)` if dictionary pruning was performed,
+/// `None` if the column is not dictionary-encoded or pruning is not applicable.
+pub(crate) fn try_dictionary_prune_in_memory(
+ predicate: &mut dyn ArrowPredicate,
+ row_group: &InMemoryRowGroup<'_>,
+ metadata: &ParquetMetaData,
+ fields: Option<&crate::arrow::schema::ParquetField>,
+) -> Result<Option<DictionaryPredicateResult>> {
+ let projection = predicate.projection();
+ let schema_descr = metadata.file_metadata().schema_descr();
+
+ // Only support single-column predicates
+ let col_idx = single_leaf_column(projection, schema_descr.num_columns());
+ let Some(col_idx) = col_idx else {
+ return Ok(None);
+ };
+
+ let row_group_meta = metadata.row_group(row_group.row_group_idx);
+ let col_meta = row_group_meta.column(col_idx);
+
+ // Only proceed if the column has a dictionary page
+ if col_meta.dictionary_page_offset().is_none() {
+ return Ok(None);
+ }
+
+ // Only safe to prune if ALL data pages are dictionary-encoded.
+ // If some pages fell back to plain encoding, the dictionary doesn't
+ // cover all values and we can't safely skip based on dictionary alone.
+ if !is_all_dictionary_encoded(col_meta) {
+ return Ok(None);
+ }
+
+ let physical_type = schema_descr.column(col_idx).physical_type();
+
+ // Only support BYTE_ARRAY and INT32/INT64 columns
+ if !matches!(
+ physical_type,
+ PhysicalType::BYTE_ARRAY | PhysicalType::INT32 | PhysicalType::INT64
+ ) {
+ return Ok(None);
+ }
+
+ // Get the arrow type for this column from the ParquetField tree.
+ // Only supports top-level primitive columns (not nested in structs/lists).
+ let Some(arrow_type) = fields.and_then(|f| find_top_level_leaf_arrow_type(f, col_idx)) else {
+ return Ok(None);
+ };
+
+ // Create a page reader for this column
+ let mut page_iter = row_group.column_chunks(col_idx)?;
+ let Some(page_reader) = page_iter.next() else {
+ return Ok(None);
+ };
+ let mut page_reader = page_reader?;
+
+ // Read the first page - should be the dictionary page
+ let first_page = page_reader.get_next_page()?;
+ let Some(page) = first_page else {
+ return Ok(None);
+ };
+
+ if !page.is_dictionary_page() {
+ return Ok(None);
+ }
+
+ let crate::column::page::Page::DictionaryPage {
+ buf, num_values, ..
+ } = page
+ else {
+ return Ok(None);
+ };
+
+ // Decode PLAIN-encoded dictionary values based on physical type,
+ // then cast to the target arrow type if needed
+ let array: ArrayRef = match physical_type {
+ PhysicalType::BYTE_ARRAY => {
+ decode_plain_byte_array(&buf, num_values as usize, &arrow_type)?
+ }
+ PhysicalType::INT32 => decode_plain_int32_as(&buf, num_values as usize, &arrow_type)?,
+ PhysicalType::INT64 => decode_plain_int64_as(&buf, num_values as usize, &arrow_type)?,
+ _ => return Ok(None),
+ };
+
+ // Build a RecordBatch with the dictionary values using a synthetic field
+ let col_name = schema_descr.column(col_idx).name().to_string();
+ let field = Field::new(&col_name, arrow_type, true);
+ let schema = Arc::new(Schema::new(vec![field]));
+ let batch = RecordBatch::try_new(schema, vec![array]).map_err(|e| {
+ crate::errors::ParquetError::General(format!("Failed to create dictionary batch: {}", e))
+ })?;
+
+ // Evaluate the predicate against dictionary values
+ let result = predicate.evaluate_dictionary(batch).map_err(|e| {
+ crate::errors::ParquetError::General(format!(
+ "Failed to evaluate dictionary predicate: {}",
+ e
+ ))
+ })?;
+
+ Ok(Some(result))
+}
+
+/// Decode PLAIN-encoded BYTE_ARRAY values into a string/binary array
+/// matching the target arrow type.
+fn decode_plain_byte_array(
+ buf: &[u8],
+ num_values: usize,
+ arrow_type: &DataType,
+) -> Result<ArrayRef> {
+ // Parse all byte array values
+ let mut values: Vec<&[u8]> = Vec::with_capacity(num_values);
+ let mut offset = 0;
+ for _ in 0..num_values {
+ if offset + 4 > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ let len = u32::from_le_bytes(buf[offset..offset + 4].try_into().unwrap()) as usize;
+ offset += 4;
+ if offset + len > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ values.push(&buf[offset..offset + len]);
+ offset += len;
+ }
+
+ match arrow_type {
+ DataType::Utf8View => {
+ let mut builder = arrow_array::builder::StringViewBuilder::with_capacity(num_values);
+ for v in &values {
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ let s = unsafe { std::str::from_utf8_unchecked(v) };
+ builder.append_value(s);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ let strs: Vec<&str> = values
+ .iter()
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ .map(|v| unsafe { std::str::from_utf8_unchecked(v) })
+ .collect();
+ Ok(Arc::new(arrow_array::StringArray::from(strs)))
+ }
+ DataType::BinaryView => {
+ let mut builder = arrow_array::builder::BinaryViewBuilder::with_capacity(num_values);
+ for v in &values {
+ builder.append_value(v);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ _ => {
+ // Default to BinaryArray for unknown types
+ let binary_values: Vec<&[u8]> = values;
+ Ok(Arc::new(arrow_array::BinaryArray::from(binary_values)))
+ }
+ }
+}
+
+/// Check if all data pages in a column chunk are dictionary-encoded.
+///
+/// Uses page encoding stats if available, otherwise falls back to checking
+/// column-level encodings. Returns false if we can't determine conclusively.
+fn is_all_dictionary_encoded(col_meta: &crate::file::metadata::ColumnChunkMetaData) -> bool {
+ // No dictionary page -> definitely not all dictionary encoded
+ if col_meta.dictionary_page_offset().is_none() {
+ return false;
+ }
+
+ // Method 1: Use page encoding stats mask if available (most reliable)
+ if let Some(mask) = col_meta.page_encoding_stats_mask() {
+ return mask.is_only(Encoding::PLAIN_DICTIONARY) || mask.is_only(Encoding::RLE_DICTIONARY);
+ }
+
+ // Method 1a: Use full page encoding stats if the mask form wasn't used
+ // (i.e. ParquetMetaDataOptions::with_encoding_stats_as_mask was set to false)
+ if let Some(stats) = col_meta.page_encoding_stats() {
+ return stats
+ .iter()
+ .filter(|s| {
+ matches!(
+ s.page_type,
+ crate::basic::PageType::DATA_PAGE | crate::basic::PageType::DATA_PAGE_V2
+ )
+ })
+ .all(|s| {
+ matches!(
+ s.encoding,
+ Encoding::PLAIN_DICTIONARY | Encoding::RLE_DICTIONARY
+ )
+ });
+ }
+
+ // Method 2: Check column-level encodings.
+ // Dictionary-encoded columns have PLAIN_DICTIONARY or RLE_DICTIONARY for data,
+ // plus PLAIN and/or RLE for definition/repetition levels.
+ // If PLAIN appears as a data encoding (fallback), the column also has
+ // PLAIN_DICTIONARY/RLE_DICTIONARY, so we'd see both.
+ // Safe heuristic: only dictionary + level encodings = all dict encoded.
+ let mut has_dict_data_encoding = false;
+ let mut has_non_dict_data_encoding = false;
+ for enc in col_meta.encodings() {
+ match enc {
+ Encoding::PLAIN_DICTIONARY | Encoding::RLE_DICTIONARY => {
+ has_dict_data_encoding = true;
+ }
+ // These are used for definition/repetition levels, not data
+ #[allow(deprecated)]
+ Encoding::RLE | Encoding::BIT_PACKED => {}
+ // PLAIN is ambiguous - used for def/rep levels in V1 pages AND
+ // as a fallback data encoding. We allow ONE PLAIN if there's also
+ // a dictionary encoding, since the PLAIN is likely for levels.
+ Encoding::PLAIN => {
+ // PLAIN is expected for def/rep levels. We can't distinguish
+ // from data fallback here, so we allow it if dict encoding exists.
+ // This is a heuristic that works for most files.
+ }
+ // Any other encoding (DELTA_*, etc.) means non-dictionary data
+ _ => {
+ has_non_dict_data_encoding = true;
+ }
+ }
+ }
+
+ has_dict_data_encoding && !has_non_dict_data_encoding
+}
+
+/// Decode PLAIN-encoded INT32 values, casting to the target arrow type
+fn decode_plain_int32_as(buf: &[u8], num_values: usize, arrow_type: &DataType) -> Result<ArrayRef> {
+ if buf.len() < num_values * 4 {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary int32".into(),
+ ));
+ }
+ let values: Vec<i32> = (0..num_values)
+ .map(|i| i32::from_le_bytes(buf[i * 4..(i + 1) * 4].try_into().unwrap()))
+ .collect();
+ match arrow_type {
+ DataType::Int8 => Ok(Arc::new(arrow_array::Int8Array::from(
+ values.into_iter().map(|v| v as i8).collect::<Vec<_>>(),
+ ))),
+ DataType::Int16 => Ok(Arc::new(arrow_array::Int16Array::from(
+ values.into_iter().map(|v| v as i16).collect::<Vec<_>>(),
+ ))),
+ DataType::UInt8 => Ok(Arc::new(arrow_array::UInt8Array::from(
+ values.into_iter().map(|v| v as u8).collect::<Vec<_>>(),
+ ))),
+ DataType::UInt16 => Ok(Arc::new(arrow_array::UInt16Array::from(
+ values.into_iter().map(|v| v as u16).collect::<Vec<_>>(),
+ ))),
+ DataType::UInt32 => Ok(Arc::new(arrow_array::UInt32Array::from(
+ values.into_iter().map(|v| v as u32).collect::<Vec<_>>(),
+ ))),
+ _ => Ok(Arc::new(arrow_array::Int32Array::from(values))),
+ }
+}
+
+/// Decode PLAIN-encoded INT64 values, casting to the target arrow type
+fn decode_plain_int64_as(buf: &[u8], num_values: usize, arrow_type: &DataType) -> Result<ArrayRef> {
+ if buf.len() < num_values * 8 {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary int64".into(),
+ ));
+ }
+ let values: Vec<i64> = (0..num_values)
+ .map(|i| i64::from_le_bytes(buf[i * 8..(i + 1) * 8].try_into().unwrap()))
+ .collect();
+ match arrow_type {
+ DataType::Int32 => Ok(Arc::new(arrow_array::Int32Array::from(
+ values.into_iter().map(|v| v as i32).collect::<Vec<_>>(),
+ ))),
+ DataType::Timestamp(unit, tz) => {
+ use arrow_array::TimestampMicrosecondArray;
+ use arrow_array::TimestampMillisecondArray;
+ use arrow_array::TimestampNanosecondArray;
+ use arrow_array::TimestampSecondArray;
+ use arrow_schema::TimeUnit;
+ match unit {
+ TimeUnit::Second => Ok(Arc::new(
+ TimestampSecondArray::from(values).with_timezone_opt(tz.clone()),
+ )),
+ TimeUnit::Millisecond => Ok(Arc::new(
+ TimestampMillisecondArray::from(values).with_timezone_opt(tz.clone()),
+ )),
+ TimeUnit::Microsecond => Ok(Arc::new(
+ TimestampMicrosecondArray::from(values).with_timezone_opt(tz.clone()),
+ )),
+ TimeUnit::Nanosecond => Ok(Arc::new(
+ TimestampNanosecondArray::from(values).with_timezone_opt(tz.clone()),
+ )),
+ }
+ }
+ _ => Ok(Arc::new(arrow_array::Int64Array::from(values))),
Review Comment:
UInt64Array?
##########
parquet/src/arrow/arrow_reader/dictionary_pruning.rs:
##########
@@ -0,0 +1,374 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Dictionary page pruning for row filters
+//!
+//! When a predicate targets a dictionary-encoded column, we can evaluate the
+//! predicate against the dictionary values before decoding any data pages.
+//! If no dictionary values match, the entire column chunk can be skipped.
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::array_reader::RowGroups;
+use crate::arrow::arrow_reader::ArrowPredicate;
+use crate::arrow::arrow_reader::filter::DictionaryPredicateResult;
+use crate::arrow::in_memory_row_group::InMemoryRowGroup;
+use crate::basic::{Encoding, Type as PhysicalType};
+use crate::errors::Result;
+use crate::file::metadata::ParquetMetaData;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use std::sync::Arc;
+
+/// Try to prune a predicate using dictionary pages from the in-memory row group.
+///
+/// Returns `Some(DictionaryPredicateResult)` if dictionary pruning was performed,
+/// `None` if the column is not dictionary-encoded or pruning is not applicable.
+pub(crate) fn try_dictionary_prune_in_memory(
+ predicate: &mut dyn ArrowPredicate,
+ row_group: &InMemoryRowGroup<'_>,
+ metadata: &ParquetMetaData,
+ fields: Option<&crate::arrow::schema::ParquetField>,
+) -> Result<Option<DictionaryPredicateResult>> {
+ let projection = predicate.projection();
+ let schema_descr = metadata.file_metadata().schema_descr();
+
+ // Only support single-column predicates
+ let col_idx = single_leaf_column(projection, schema_descr.num_columns());
+ let Some(col_idx) = col_idx else {
+ return Ok(None);
+ };
+
+ let row_group_meta = metadata.row_group(row_group.row_group_idx);
+ let col_meta = row_group_meta.column(col_idx);
+
+ // Only proceed if the column has a dictionary page
+ if col_meta.dictionary_page_offset().is_none() {
+ return Ok(None);
+ }
+
+ // Only safe to prune if ALL data pages are dictionary-encoded.
+ // If some pages fell back to plain encoding, the dictionary doesn't
+ // cover all values and we can't safely skip based on dictionary alone.
+ if !is_all_dictionary_encoded(col_meta) {
+ return Ok(None);
+ }
+
+ let physical_type = schema_descr.column(col_idx).physical_type();
+
+ // Only support BYTE_ARRAY and INT32/INT64 columns
Review Comment:
Just curious why not other physical types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#discussion_r2966837415
##########
parquet/src/arrow/arrow_reader/dictionary_pruning.rs:
##########
@@ -0,0 +1,374 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Dictionary page pruning for row filters
+//!
+//! When a predicate targets a dictionary-encoded column, we can evaluate the
+//! predicate against the dictionary values before decoding any data pages.
+//! If no dictionary values match, the entire column chunk can be skipped.
+
+use crate::arrow::ProjectionMask;
+use crate::arrow::array_reader::RowGroups;
+use crate::arrow::arrow_reader::ArrowPredicate;
+use crate::arrow::arrow_reader::filter::DictionaryPredicateResult;
+use crate::arrow::in_memory_row_group::InMemoryRowGroup;
+use crate::basic::{Encoding, Type as PhysicalType};
+use crate::errors::Result;
+use crate::file::metadata::ParquetMetaData;
+use arrow_array::{ArrayRef, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use std::sync::Arc;
+
+/// Try to prune a predicate using dictionary pages from the in-memory row group.
+///
+/// Returns `Some(DictionaryPredicateResult)` if dictionary pruning was performed,
+/// `None` if the column is not dictionary-encoded or pruning is not applicable.
+pub(crate) fn try_dictionary_prune_in_memory(
+ predicate: &mut dyn ArrowPredicate,
+ row_group: &InMemoryRowGroup<'_>,
+ metadata: &ParquetMetaData,
+ fields: Option<&crate::arrow::schema::ParquetField>,
+) -> Result<Option<DictionaryPredicateResult>> {
+ let projection = predicate.projection();
+ let schema_descr = metadata.file_metadata().schema_descr();
+
+ // Only support single-column predicates
+ let col_idx = single_leaf_column(projection, schema_descr.num_columns());
+ let Some(col_idx) = col_idx else {
+ return Ok(None);
+ };
+
+ let row_group_meta = metadata.row_group(row_group.row_group_idx);
+ let col_meta = row_group_meta.column(col_idx);
+
+ // Only proceed if the column has a dictionary page
+ if col_meta.dictionary_page_offset().is_none() {
+ return Ok(None);
+ }
+
+ // Only safe to prune if ALL data pages are dictionary-encoded.
+ // If some pages fell back to plain encoding, the dictionary doesn't
+ // cover all values and we can't safely skip based on dictionary alone.
+ if !is_all_dictionary_encoded(col_meta) {
+ return Ok(None);
+ }
+
+ let physical_type = schema_descr.column(col_idx).physical_type();
+
+ // Only support BYTE_ARRAY and INT32/INT64 columns
+ if !matches!(
+ physical_type,
+ PhysicalType::BYTE_ARRAY | PhysicalType::INT32 | PhysicalType::INT64
+ ) {
+ return Ok(None);
+ }
+
+ // Get the arrow type for this column from the ParquetField tree.
+ // Only supports top-level primitive columns (not nested in structs/lists).
+ let Some(arrow_type) = fields.and_then(|f| find_top_level_leaf_arrow_type(f, col_idx)) else {
+ return Ok(None);
+ };
+
+ // Create a page reader for this column
+ let mut page_iter = row_group.column_chunks(col_idx)?;
+ let Some(page_reader) = page_iter.next() else {
+ return Ok(None);
+ };
+ let mut page_reader = page_reader?;
+
+ // Read the first page - should be the dictionary page
+ let first_page = page_reader.get_next_page()?;
+ let Some(page) = first_page else {
+ return Ok(None);
+ };
+
+ if !page.is_dictionary_page() {
+ return Ok(None);
+ }
+
+ let crate::column::page::Page::DictionaryPage {
+ buf, num_values, ..
+ } = page
+ else {
+ return Ok(None);
+ };
+
+ // Decode PLAIN-encoded dictionary values based on physical type,
+ // then cast to the target arrow type if needed
+ let array: ArrayRef = match physical_type {
+ PhysicalType::BYTE_ARRAY => {
+ decode_plain_byte_array(&buf, num_values as usize, &arrow_type)?
+ }
+ PhysicalType::INT32 => decode_plain_int32_as(&buf, num_values as usize, &arrow_type)?,
+ PhysicalType::INT64 => decode_plain_int64_as(&buf, num_values as usize, &arrow_type)?,
+ _ => return Ok(None),
+ };
+
+ // Build a RecordBatch with the dictionary values using a synthetic field
+ let col_name = schema_descr.column(col_idx).name().to_string();
+ let field = Field::new(&col_name, arrow_type, true);
+ let schema = Arc::new(Schema::new(vec![field]));
+ let batch = RecordBatch::try_new(schema, vec![array]).map_err(|e| {
+ crate::errors::ParquetError::General(format!("Failed to create dictionary batch: {}", e))
+ })?;
+
+ // Evaluate the predicate against dictionary values
+ let result = predicate.evaluate_dictionary(batch).map_err(|e| {
+ crate::errors::ParquetError::General(format!(
+ "Failed to evaluate dictionary predicate: {}",
+ e
+ ))
+ })?;
+
+ Ok(Some(result))
+}
+
+/// Decode PLAIN-encoded BYTE_ARRAY values into a string/binary array
+/// matching the target arrow type.
+fn decode_plain_byte_array(
+ buf: &[u8],
+ num_values: usize,
+ arrow_type: &DataType,
+) -> Result<ArrayRef> {
+ // Parse all byte array values
+ let mut values: Vec<&[u8]> = Vec::with_capacity(num_values);
+ let mut offset = 0;
+ for _ in 0..num_values {
+ if offset + 4 > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ let len = u32::from_le_bytes(buf[offset..offset + 4].try_into().unwrap()) as usize;
+ offset += 4;
+ if offset + len > buf.len() {
+ return Err(crate::errors::ParquetError::EOF(
+ "eof decoding dictionary byte array".into(),
+ ));
+ }
+ values.push(&buf[offset..offset + len]);
+ offset += len;
+ }
+
+ match arrow_type {
+ DataType::Utf8View => {
+ let mut builder = arrow_array::builder::StringViewBuilder::with_capacity(num_values);
+ for v in &values {
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ let s = unsafe { std::str::from_utf8_unchecked(v) };
+ builder.append_value(s);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ DataType::Utf8 | DataType::LargeUtf8 => {
+ let strs: Vec<&str> = values
+ .iter()
+ // SAFETY: parquet BYTE_ARRAY dictionary values for string columns are valid UTF-8
+ .map(|v| unsafe { std::str::from_utf8_unchecked(v) })
+ .collect();
+ Ok(Arc::new(arrow_array::StringArray::from(strs)))
+ }
+ DataType::BinaryView => {
+ let mut builder = arrow_array::builder::BinaryViewBuilder::with_capacity(num_values);
+ for v in &values {
+ builder.append_value(v);
+ }
+ Ok(Arc::new(builder.finish()))
+ }
+ _ => {
+ // Default to BinaryArray for unknown types
+ let binary_values: Vec<&[u8]> = values;
+ Ok(Arc::new(arrow_array::BinaryArray::from(binary_values)))
+ }
+ }
+}
+
+/// Check if all data pages in a column chunk are dictionary-encoded.
+///
+/// Uses page encoding stats if available, otherwise falls back to checking
+/// column-level encodings. Returns false if we can't determine conclusively.
+fn is_all_dictionary_encoded(col_meta: &crate::file::metadata::ColumnChunkMetaData) -> bool {
+ // No dictionary page -> definitely not all dictionary encoded
+ if col_meta.dictionary_page_offset().is_none() {
+ return false;
+ }
+
+ // Method 1: Use page encoding stats mask if available (most reliable)
+ if let Some(mask) = col_meta.page_encoding_stats_mask() {
+ return mask.is_only(Encoding::PLAIN_DICTIONARY) || mask.is_only(Encoding::RLE_DICTIONARY);
+ }
+
+ // Method 1a: Use full page encoding stats if the mask form wasn't used
+ // (i.e. ParquetMetaDataOptions::with_encoding_stats_as_mask was set to false)
+ if let Some(stats) = col_meta.page_encoding_stats() {
+ return stats
+ .iter()
+ .filter(|s| {
+ matches!(
+ s.page_type,
+ crate::basic::PageType::DATA_PAGE | crate::basic::PageType::DATA_PAGE_V2
+ )
+ })
+ .all(|s| {
+ matches!(
+ s.encoding,
+ Encoding::PLAIN_DICTIONARY | Encoding::RLE_DICTIONARY
+ )
+ });
+ }
+
+ // Method 2: Check column-level encodings.
+ // Dictionary-encoded columns have PLAIN_DICTIONARY or RLE_DICTIONARY for data,
+ // plus PLAIN and/or RLE for definition/repetition levels.
+ // If PLAIN appears as a data encoding (fallback), the column also has
+ // PLAIN_DICTIONARY/RLE_DICTIONARY, so we'd see both.
+ // Safe heuristic: only dictionary + level encodings = all dict encoded.
+ let mut has_dict_data_encoding = false;
+ let mut has_non_dict_data_encoding = false;
+ for enc in col_meta.encodings() {
+ match enc {
+ Encoding::PLAIN_DICTIONARY | Encoding::RLE_DICTIONARY => {
+ has_dict_data_encoding = true;
+ }
+ // These are used for definition/repetition levels, not data
+ #[allow(deprecated)]
+ Encoding::RLE | Encoding::BIT_PACKED => {}
+ // PLAIN is ambiguous - used for def/rep levels in V1 pages AND
Review Comment:
I'm not so sure about this one. So the way the encodings work is:
- levels are RLE or BIT_PACKED (as noted above)
- V1 dictionary uses the PLAIN_DICTIONARY variant for both the dictionary and data
- V2 dictionary uses PLAIN for dictionary and RLE_DICTIONARY for data
- all other encodings are for data
So if we see PLAIN_DICTIONARY and PLAIN, we can know that fallback has definitely occurred. In the case of RLE_DICTIONARY, it's a coin flip, but I'd err on the side of caution and return `false` in that case. In my experience fallback is quite common.
```rust
let mut has_plain_dict_encoding = false;
let mut has_plain_encoding = false;
for enc in col_meta.encodings() {
match enc {
Encoding::PLAIN_DICTIONARY => has_plain_dict_encoding = true,
// for RLE_DICT we can't know if fallback has occurred...be pessimistic
Encoding::RLE_DICTIONARY => return false,
Encoding::PLAIN => has_plain_encoding = true,
// These are used for definition/repetition levels, not data
#[allow(deprecated)]
Encoding::RLE | Encoding::BIT_PACKED => {}
// Any other encoding (DELTA_*, etc.) means non-dictionary data
_ => return false,
}
}
has_plain_dict_encoding && !has_plain_encoding
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085456078
YAAAS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#discussion_r2961989180
##########
parquet/tests/arrow_reader/io/async_reader.rs:
##########
@@ -275,9 +275,7 @@ async fn test_read_multiple_row_filter() {
"Read Multi:",
" Row Group 1, column 'a': MultiPage(dictionary_page: true, data_pages: [0, 1]) (1856 bytes, 1 requests) [data]",
"Read Multi:",
- " Row Group 1, column 'b': DictionaryPage (1617 bytes, 1 requests) [data]",
- " Row Group 1, column 'b': DataPage(0) (113 bytes , 1 requests) [data]",
- " Row Group 1, column 'b': DataPage(1) (126 bytes , 1 requests) [data]",
+ " Row Group 1, column 'b': MultiPage(dictionary_page: true, data_pages: [0, 1]) (1856 bytes, 1 requests) [data]",
Review Comment:
For anyone following along, I think @Dandandan fixed it here:
- https://github.com/apache/arrow-rs/pull/9578
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #9574: feat: Dictionary page pruning for row filter predicates
URL: https://github.com/apache/arrow-rs/pull/9574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4785345176
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4172996494
> BTW Codex says it found issues with this PR; I haven't had time to review them carefully but I would like to before we merge this. I hope to find time for this PR over the next few days
FWIW, the second "High" finding is what I was worried about above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4092847608
> Is this similar to
>
> * [parquet: decoder-level dictionary filter pushdown for Parquet reader #9464](https://github.com/apache/arrow-rs/pull/9464)
> Is this similar to
>
> * [parquet: decoder-level dictionary filter pushdown for Parquet reader #9464](https://github.com/apache/arrow-rs/pull/9464)
No - this PR applies it to the dictionary (which is small thus fast) and avoids decompressing / decoding the data pages.
The linked PR https://github.com/apache/arrow-rs/pull/9464 as far as I see tries to reuse the booleans from the predicate by gathering them back onto the rows.
I think in case you only need to apply the rowfilter (and don't need the values) it could save materializing the values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085328605
🤖 Arrow criterion benchmark running (GKE) | [trigger](https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085313791)
`Linux bench-c4085313791-433-jqc52 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux`
Comparing dictionary-page-pruning (d2d1d28adf23dcb75dbb2ae0c01b02b9410d05ac) to 3931179 (merge-base) [diff](https://github.com/apache/arrow-rs/compare/393117979882e97a15125edd142c70a5e2c16386..d2d1d28adf23dcb75dbb2ae0c01b02b9410d05ac)
BENCH_NAME=arrow_reader_clickbench
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench arrow_reader_clickbench
BENCH_FILTER=
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Dictionary page pruning for row filter predicates [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9574:
URL: https://github.com/apache/arrow-rs/pull/9574#issuecomment-4085640982
> We could (at the cost of extra IO request, but saving for some) also do this before loading the column chunks I think -
but probably needs to be some config.
I agree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org