You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/16 20:47:14 UTC
(datafusion) branch gh-readonly-queue/main/pr-26405-37c954c639da635178d16f4f311d40cb60eefce7 created (now ca1561fa11)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-26405-37c954c639da635178d16f4f311d40cb60eefce7
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at ca1561fa11 perf: avoid building a DFSchema for every pruning comparison (#26405)
This branch includes the following new commits:
new ca1561fa11 perf: avoid building a DFSchema for every pruning comparison (#26405)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: perf: avoid building a DFSchema for every pruning comparison (#26405)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-26405-37c954c639da635178d16f4f311d40cb60eefce7
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit ca1561fa11c947c33aca7170431a300d932a703e
Author: Peter L <ce...@hotmail.com>
AuthorDate: Fri Oct 16 20:46:51 2026 +0000
perf: avoid building a DFSchema for every pruning comparison (#26405)
## Which issue does this PR close?
- Related to #21968. There is no dedicated issue for this change.
## Rationale for this change
Building a pruning predicate for a Parquet file with a wide schema takes
time proportional to the number of columns in the file, once for every
comparison in the predicate, even when the predicate only references one
column. A Parquet scan builds several pruning predicates for each file
it opens (for example for row group statistics, the page index and bloom
filters), so on wide tables this cost is paid many times per file.
The cause is in `PruningExpressionBuilder::try_new`. For every
comparison it converts the whole file schema into a `DFSchema`, which
runs `DFSchema::check_names` and inserts every field name into a
`BTreeSet`. The `DFSchema` is then only used through
`schema.as_arrow()`, to get the data type of the expression inside a
`CAST` or `TRY_CAST`. That is the Arrow schema the builder already has.
In a profile of `wide_schema` Q01 (1024 columns, 256 files),
`DFSchema::try_from` called from `PruningExpressionBuilder::try_new` was
about 5% of CPU samples. With a file metadata cache large enough to hold
every footer it was 9.8%.
The `DFSchema` dates from #3422, when pruning rewrote logical `Expr`s
and needed a `DFSchema` to resolve their types. #5419 moved pruning to
physical expressions, which only need the Arrow schema, but kept the
`DFSchema` parameter.
## What changes are included in this PR?
- `rewrite_expr_to_prunable` and `rewrite_cast_child_to_prunable` take
`&Schema` instead of a `DFSchema`.
- `PruningExpressionBuilder::try_new` passes the schema it already has,
rather than building a `DFSchema` for each comparison.
- Two unit tests that built a `DFSchema` only to call
`rewrite_expr_to_prunable` now pass the `Schema`.
## What is the testing strategy for this PR?
The change is covered by existing tests, so this PR adds none. Both
functions are private, and their behaviour is unchanged apart from the
duplicate-name case described below.
- `datafusion-pruning` tests, including `test_rewrite_expr_to_prunable`
and `test_rewrite_expr_to_prunable_error`
- `datafusion-datasource-parquet` lib tests
- `datafusion` `parquet_integration` tests, which cover row group and
page index pruning
## Are there any user-facing changes?
There are no API changes, because both functions are private.
There is one behaviour change. `DFSchema::try_from` fails when the
schema has duplicate field names, so previously no comparison against
such a schema could be used for pruning. Now pruning goes ahead and
resolves the column by name to the first field with that name. The
column lookup in the same function (`schema.column_with_name`) and the
Parquet statistics lookup (`parquet_column`) already resolve names this
way.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <no...@anthropic.com>
---
datafusion/pruning/src/pruning_predicate.rs | 56 +++++++++--------------------
1 file changed, 16 insertions(+), 40 deletions(-)
diff --git a/datafusion/pruning/src/pruning_predicate.rs b/datafusion/pruning/src/pruning_predicate.rs
index 07a24efb0d..6828c56fc8 100644
--- a/datafusion/pruning/src/pruning_predicate.rs
+++ b/datafusion/pruning/src/pruning_predicate.rs
@@ -41,9 +41,7 @@ use log::{debug, trace};
use datafusion_common::error::Result;
use datafusion_common::tree_node::{TransformedResult, TreeNodeRecursion};
-use datafusion_common::{
- _internal_datafusion_err, Column, DFSchema, assert_eq_or_internal_err,
-};
+use datafusion_common::{_internal_datafusion_err, Column, assert_eq_or_internal_err};
use datafusion_common::{
ScalarValue, internal_datafusion_err, plan_datafusion_err, plan_err,
tree_node::{Transformed, TreeNode},
@@ -1155,13 +1153,8 @@ impl<'a> PruningExpressionBuilder<'a> {
}
};
- let df_schema = DFSchema::try_from(Arc::clone(schema))?;
- let (column_expr, correct_operator, scalar_expr) = rewrite_expr_to_prunable(
- column_expr,
- correct_operator,
- scalar_expr,
- df_schema,
- )?;
+ let (column_expr, correct_operator, scalar_expr) =
+ rewrite_expr_to_prunable(column_expr, correct_operator, scalar_expr, schema)?;
let Some((_, field)) = schema.column_with_name(column.name()) else {
return plan_err!("Field not found in schema");
};
@@ -1251,7 +1244,7 @@ fn rewrite_expr_to_prunable(
column_expr: &PhysicalExprRef,
op: Operator,
scalar_expr: &PhysicalExprRef,
- schema: DFSchema,
+ schema: &Schema,
) -> Result<(PhysicalExprRef, Operator, PhysicalExprRef)> {
if !is_compare_op(op) {
return plan_err!("rewrite_expr_to_prunable only support compare expression");
@@ -1329,12 +1322,9 @@ fn rewrite_cast_child_to_prunable(
cast_type: &DataType,
op: Operator,
scalar_expr: &PhysicalExprRef,
- schema: DFSchema,
+ schema: &Schema,
) -> Result<(PhysicalExprRef, Operator, PhysicalExprRef)> {
- verify_support_type_for_prune(
- &cast_child_expr.data_type(schema.as_arrow())?,
- cast_type,
- )?;
+ verify_support_type_for_prune(&cast_child_expr.data_type(schema)?, cast_type)?;
rewrite_expr_to_prunable(cast_child_expr, op, scalar_expr, schema)
}
@@ -6753,20 +6743,15 @@ mod tests {
#[test]
fn test_rewrite_expr_to_prunable() {
let schema = Schema::new(vec![Field::new("a", DataType::Int32, true)]);
- let df_schema = DFSchema::try_from(schema.clone()).unwrap();
// column op lit
let left_input = col("a");
let left_input = logical2physical(&left_input, &schema);
let right_input = lit(ScalarValue::Int32(Some(12)));
let right_input = logical2physical(&right_input, &schema);
- let (result_left, _, result_right) = rewrite_expr_to_prunable(
- &left_input,
- Operator::Eq,
- &right_input,
- df_schema.clone(),
- )
- .unwrap();
+ let (result_left, _, result_right) =
+ rewrite_expr_to_prunable(&left_input, Operator::Eq, &right_input, &schema)
+ .unwrap();
assert_eq!(result_left.to_string(), left_input.to_string());
assert_eq!(result_right.to_string(), right_input.to_string());
@@ -6775,13 +6760,9 @@ mod tests {
let left_input = logical2physical(&left_input, &schema);
let right_input = lit(ScalarValue::Decimal128(Some(12), 20, 3));
let right_input = logical2physical(&right_input, &schema);
- let (result_left, _, result_right) = rewrite_expr_to_prunable(
- &left_input,
- Operator::Gt,
- &right_input,
- df_schema.clone(),
- )
- .unwrap();
+ let (result_left, _, result_right) =
+ rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, &schema)
+ .unwrap();
assert_eq!(result_left.to_string(), left_input.to_string());
assert_eq!(result_right.to_string(), right_input.to_string());
@@ -6791,7 +6772,7 @@ mod tests {
let right_input = lit(ScalarValue::Int64(Some(12)));
let right_input = logical2physical(&right_input, &schema);
let (result_left, _, result_right) =
- rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, df_schema)
+ rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, &schema)
.unwrap();
assert_eq!(result_left.to_string(), left_input.to_string());
assert_eq!(result_right.to_string(), right_input.to_string());
@@ -6872,17 +6853,12 @@ mod tests {
// cast string value to numeric value
// this cast is not supported
let schema = Schema::new(vec![Field::new("a", DataType::Utf8, true)]);
- let df_schema = DFSchema::try_from(schema.clone()).unwrap();
let left_input = cast(col("a"), DataType::Int64);
let left_input = logical2physical(&left_input, &schema);
let right_input = lit(ScalarValue::Int64(Some(12)));
let right_input = logical2physical(&right_input, &schema);
- let result = rewrite_expr_to_prunable(
- &left_input,
- Operator::Gt,
- &right_input,
- df_schema.clone(),
- );
+ let result =
+ rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, &schema);
assert!(result.is_err());
// other expr
@@ -6891,7 +6867,7 @@ mod tests {
let right_input = lit(ScalarValue::Int64(Some(12)));
let right_input = logical2physical(&right_input, &schema);
let result =
- rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, df_schema);
+ rewrite_expr_to_prunable(&left_input, Operator::Gt, &right_input, &schema);
assert!(result.is_err());
// TODO: add other negative test for other case and op
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org