You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ion-elgreco (via GitHub)" <gi...@apache.org> on 2025/02/28 18:05:27 UTC
[I] Should pruningpredicate coerce? [datafusion]
ion-elgreco opened a new issue, #14944:
URL: https://github.com/apache/datafusion/issues/14944
### Describe the bug
Currently when you pass a pruning predicate where the predicate has a different type as the targeted column it will not prune it, even though in theory the value is castable to the target column type.
The predicate looks like this: "month_id = '202502' AND date_id = '20250226'"
However the columns are `int` column not `utf8`. So in theory these string values can be casted to int but I don't believe this is happening. Is this something that should be added
This is our rust code btw
```
// delta-rs/crates/core/src/delta_datafusion/mod.rs
let pruning_predicate = PruningPredicate::try_new(predicate.clone(), logical_schema.clone())?;
let files_to_prune = pruning_predicate.prune(self.snapshot)?;
```
### To Reproduce
Related delta-rs issue: https://github.com/delta-io/delta-rs/issues/3278
### Expected behavior
Allow literal value coercion during pruning
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should pruningpredicate coerce? [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2694168148
> It's like cast(month_id, 'utf8') = '202502', see below:
So it seems like it would be a valuable thing to apply the type coercion rewriter in the expr simplifier then.
https://github.com/apache/datafusion/blob/57a122137a0b64ea523fe6c02b88423f92b9aa0f/datafusion/optimizer/src/unwrap_cast_in_comparison.rs#L115-L125
I'll file a ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should `PruningPredicate` coerce? [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2701725282
Also filed a ticket for unwrapping that particular comparison expressio
- https://github.com/apache/datafusion/issues/15035
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should `PruningPredicate` coerce? [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2701582743
So in other words, I think https://github.com/apache/datafusion/pull/15012 is necessary, but not sufficient, to fix this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should `PruningPredicate` coerce? [datafusion]
Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2695290435
@alamb thanks! 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should `PruningPredicate` coerce? [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2695282931
Filed
- https://github.com/apache/datafusion/issues/14987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should pruningpredicate coerce? [datafusion]
Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2691341269
@alamb this is actually already done like that, but it still doesn't prune properly:
` .map(|expr| context.create_physical_expr(expr, &df_schema).unwrap());`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should pruningpredicate coerce? [datafusion]
Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2692138942
It's like cast(month_id, 'utf8') = '202502', see below:
```
[crates/core/src/delta_datafusion/mod.rs:551:9] self.filter.clone() = Some(
BinaryExpr(
BinaryExpr {
left: BinaryExpr(
BinaryExpr {
left: Cast(
Cast {
expr: Column(
Column {
relation: None,
name: "month_id",
},
),
data_type: Utf8,
},
),
op: Eq,
right: Literal(
Utf8("202502"),
),
},
),
op: And,
right: BinaryExpr(
BinaryExpr {
left: Cast(
Cast {
expr: Column(
Column {
relation: None,
name: "date_id",
},
),
data_type: Utf8,
},
),
op: Eq,
right: Literal(
Utf8("20250226"),
),
},
),
},
),
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should pruningpredicate coerce? [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2691384188
🤔 What are the coerced expressions?
If it is like this:
```
cast(month_id, 'utf8') = '202502'
```
That is not going to prune because the cast is happening on `month_id`
The predicate needs to be
```
month_id = cast('202502')
```
This is done in this analyzer pass: https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/unwrap_cast_in_comparison.rs
Maybe we should do the same thing in `create_physical_expr` too 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should pruningpredicate coerce? [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2691336051
I think the conclusion is that PruningPredicaate expects a properly coerced expression already
You can use `SessionContext::create_physical_expr` to automatically perform coercsion as shown in this example
https://github.com/apache/datafusion/blob/5e49094c159ce110bebd2bb6f4858ff515cd1860/datafusion-examples/examples/expr_api.rs#L540-L543
There are a bunch of other examples there about how to apply coercion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Should `PruningPredicate` coerce? [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14944:
URL: https://github.com/apache/datafusion/issues/14944#issuecomment-2701581478
- So I was writing some tests for https://github.com/apache/datafusion/pull/15012
I found that a predicate like this:
```sql
month_id = '202502'
```
Is actually coerced so that it does string comparison rather than integer comparison (note the predicate in the plan below is `CAST(foo.i AS Utf8) = Utf8("2024")` rather than `foo.i = CAST("2024" as Utf8)
```
DataFusion CLI v46.0.0
> create table foo(i int) as values (1), (2), (3);
0 row(s) fetched.
Elapsed 0.033 seconds.
> explain select i from foo where i = '2024';
+---------------+-------------------------------------------------------+
| plan_type | plan |
+---------------+-------------------------------------------------------+
| logical_plan | Filter: CAST(foo.i AS Utf8) = Utf8("2024") |
| | TableScan: foo projection=[i] |
| physical_plan | CoalesceBatchesExec: target_batch_size=8192 |
| | FilterExec: CAST(i@0 AS Utf8) = 2024 |
| | DataSourceExec: partitions=1, partition_sizes=[1] |
| | |
+---------------+-------------------------------------------------------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org