You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "doki23 (via GitHub)" <gi...@apache.org> on 2023/04/02 07:11:46 UTC
[GitHub] [arrow-datafusion] doki23 opened a new issue, #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
doki23 opened a new issue, #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830
### Is your feature request related to a problem or challenge?
```sql
❯ explain select * from t where a > 1 and a < 1;
+---------------+-------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+-------------------------------------------------------------------------------+
| logical_plan | Filter: t.a > Int32(1) AND t.a < Int32(1) |
| | TableScan: t projection=[a, b] |
| physical_plan | CoalesceBatchesExec: target_batch_size=8192 |
| | FilterExec: a@0 > 1 AND a@0 < 1 |
| | MemoryExec: partitions=10, partition_sizes=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
| | |
+---------------+-------------------------------------------------------------------------------+
```
Datafusion needs to scan table t and execute the filter, although it's unnecessary.
### Describe the solution you'd like
Add an optimizer rule to simplify this predicate to a constant bool value and make the logical plan equal to 'select * from t where false' like
```sql
❯ explain select * from t where false;
+---------------+----------------------------------+
| plan_type | plan |
+---------------+----------------------------------+
| logical_plan | EmptyRelation |
| physical_plan | EmptyExec: produce_one_row=false |
| | |
+---------------+----------------------------------+
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1497466044
> A related idea: May we need a easy-to-use expression rewriting framework to add some rule to rewrite expression?
This is a neat idea @jackwener and people have suggested that or similar things (for example, I think https://github.com/apache/arrow-datafusion/pull/1485 was one such suggestion)
Having general purpose rewrite rules certainly sounds like a good idea, though I don't have much knowledge of how well the work in practice in query engines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1494512805
> And I'm not sure whether a > 1 and a < 1 can always fold to false, considering a might be NULL
Agree with it.
BTW, it's same with #1716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1495430901
Oh, I notice this issue already specify `Predicate`, so `a > 1 and a < 1` can be folded to `false`.
Because predicate contain implicit `not null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Simplify predicate expressions like 'a > 1 and a < 1' to constant false [datafusion]
Posted by "wudidapaopao (via GitHub)" <gi...@apache.org>.
wudidapaopao commented on issue #5830:
URL: https://github.com/apache/datafusion/issues/5830#issuecomment-5640019341
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Simplify predicate expressions like 'a > 1 and a < 1' to constant false [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #5830:
URL: https://github.com/apache/datafusion/issues/5830#issuecomment-5649799385
pub enum Endpoint {
Unbounded,
Open(ScalarValue), // exclusive
Closed(ScalarValue), // inclusive
}
pub struct Range { lower: Endpoint, upper: Endpoint }
pub struct Domain {
data_type: DataType,
ranges: Vec<Range>, // sorted, disjoint, non-adjacent, all non-empty
nulls: bool,
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] mingmwang commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "mingmwang (via GitHub)" <gi...@apache.org>.
mingmwang commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1494447713
> Consider a little bit more complex predicate like 'a > 1 and b > 1 and (a + b) < 2'
We can not simply fold the `a > 1 and b > 1 and (a + b) < 2` to always `false`, considering ` a + b` might overflow the data types of `Int`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] mingmwang commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "mingmwang (via GitHub)" <gi...@apache.org>.
mingmwang commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1494436387
> A related idea: May we need a easy-to-use expression rewriting framework to add some rule to rewrite expression?
>
> like Apache Doris : https://github.com/apache/doris/tree/master/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite
>
> this issue is resolved by a rule: https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/SimplifyRange.java
>
> cc @mingmwang @alamb
@jackwener
I think we already have enough rewriting utils for exprs to implement such rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1494340314
A related idea: we need a easy-to-use expression rewriting framework to add some rule to rewrite expression.
cc @mingmwang @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Simplify predicate expressions like 'a > 1 and a < 1' to constant false [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
URL: https://github.com/apache/datafusion/issues/5830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] mingmwang commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "mingmwang (via GitHub)" <gi...@apache.org>.
mingmwang commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1494441196
And I'm not sure whether `a > 1 and a < 1` can always fold to `false`, considering `a` might be `NULL`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Simplify predicate expressions like 'a > 1 and a < 1' to constant false [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #5830:
URL: https://github.com/apache/datafusion/issues/5830#issuecomment-5673026751
> I'm thinking we could have something like this in `datafusion-common`, and `simplify_predicates` forces `Domain → Expr` 🤔
This idea is very similar to a stalled PR (though they're targeting different problem)
- https://github.com/apache/datafusion/pull/19609
Would be amazing if we can figure out an API to be reusable by both problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] doki23 commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "doki23 (via GitHub)" <gi...@apache.org>.
doki23 commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1495320987
Alright, let's change the example to `a is not null and b is not null and a > 1 and a < 100 and b > 1 and b < 100 and a + b < 2`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Simplify predicate expressions like 'a > 1 and a < 1' to constant false [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #5830:
URL: https://github.com/apache/datafusion/issues/5830#issuecomment-5649823176
I'm thinking we could have something like this in `datafusion-common`, and `simplify_predicates` forces `Domain → Expr` 🤔
```rs
pub enum Endpoint {
Unbounded,
Open(ScalarValue), // exclusive
Closed(ScalarValue), // inclusive
}
pub struct Range { lower: Endpoint, upper: Endpoint }
pub struct Domain {
data_type: DataType,
ranges: Vec<Range>, // sorted, disjoint, non-adjacent, all non-empty
nulls: bool,
}
impl Domain {
fn all(dt: &DataType) -> Self; // ranges: [(-∞,∞)], nulls: true
fn none(dt: &DataType) -> Self;
fn null(dt: &DataType) -> Self; // ranges: [], nulls: true
fn from_range(dt, Range, nulls: bool) -> Result<Self>;
fn from_values(dt, impl IntoIterator<Item = ScalarValue>) -> Result<Self>; // IN lists
fn union(&self, &Self) -> Result<Self>;
fn intersect(&self, &Self) -> Result<Self>;
fn contains(&self, &ScalarValue) -> bool;
fn is_empty(&self) -> bool; // the refutation test
fn is_all(&self) -> bool;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Simplify predicate expressions like 'a > 1 and a < 1' to constant false [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5830:
URL: https://github.com/apache/datafusion/issues/5830#issuecomment-5687211772
> Would be amazing if we can figure out an API to be reusable by both problem.
I feel like the range analysis (input ranges of input columns, output is ranges of output columns) could be the building block for such an optimization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] doki23 commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "doki23 (via GitHub)" <gi...@apache.org>.
doki23 commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1493254059
Consider a little bit more complex predicate like 'a > 1 and b > 1 and (a + b) < 2'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #5830: Simplify predicate expressions like 'a > 1 and a < 1' to constant false
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5830:
URL: https://github.com/apache/arrow-datafusion/issues/5830#issuecomment-1497463640
I think we could probably add this specific type of simplification to the expression simplifier and it would be fine.
I think a more general approach for handling arbitrary predicates with constraints, such as @doki23 describes
```
a is not null and b is not null and a > 1 and a < 100 and b > 1 and b < 100 and a + b < 2
```
Is not rewrite rules but rather some form of range analysis (e.g. as described on https://github.com/apache/arrow-datafusion/issues/5535)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Simplify predicate expressions like 'a > 1 and a < 1' to constant false [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5830:
URL: https://github.com/apache/datafusion/issues/5830#issuecomment-3172527648
I think a more general solution to solving this type of issue is with the `cp_solver` -- https://docs.rs/datafusion/latest/datafusion/physical_expr/intervals/cp_solver/index.html
The idea is we propagate interval analysis for general expressions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org