You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "hiltontj (via GitHub)" <gi...@apache.org> on 2025/01/26 14:28:28 UTC
[I] Allow for bounds analysis on selective columns in a schema [datafusion]
hiltontj opened a new issue, #14309:
URL: https://github.com/apache/datafusion/issues/14309
### Is your feature request related to a problem or challenge?
When performing bounds analysis on a filter expression (`Expr`), I would like to only analyze bounds for specific columns in my schema, and ignore other columns that are in the `Expr`. However, the [`analyze`](https://github.com/apache/datafusion/blob/7c07948358eac81c4b297fa2400cba3c9ca55dc2/datafusion/physical-expr/src/analysis.rs#L159) and underlying [`ExprIntervalGraph`](https://github.com/apache/datafusion/blob/7c07948358eac81c4b297fa2400cba3c9ca55dc2/datafusion/physical-expr/src/intervals/cp_solver.rs#L123) analyzes bounds for all nodes in the `Expr` tree.
The problem is that sometimes the `Expr` tree can have a mismatch in the LHS/RHS data type, which leads to the analysis failing due to errors like [this one](https://github.com/apache/datafusion/blob/7c07948358eac81c4b297fa2400cba3c9ca55dc2/datafusion/expr-common/src/interval_arithmetic.rs#L1195-L1199).
### Describe the solution you'd like
When analyzing an expression like
```
WHERE a < 10 AND b > "aaa"
```
I would like there to be a way to only analyze the bounds imposed on `a`, which in the above should result in the interval [`None`, `9`].
### Describe alternatives you've considered
I may not be using the correct APIs, or am using the correct APIs incorrectly; if either are the case, then perhaps I can learn what is correct and create an example for my use case.
There may be other solutions to this problem that I am not aware of. DataFusion may also be doing expression simplification with respect to the schema that make this less of a problem than it appears to be in my contrived [reproducer](https://github.com/apache/datafusion/pull/14308).
### Additional context
I opened a reproducer for this here: https://github.com/apache/datafusion/pull/14308
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow for bounds analysis on selective columns in a schema [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14309:
URL: https://github.com/apache/datafusion/issues/14309#issuecomment-2616061681
FYI @metesynnada and @ozankabak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow for bounds analysis on selective columns in a schema [datafusion]
Posted by "hiltontj (via GitHub)" <gi...@apache.org>.
hiltontj commented on issue #14309:
URL: https://github.com/apache/datafusion/issues/14309#issuecomment-2614450399
Another thing worth noting is that from what I've seen, DataFusion splits filter expressions on `AND` conjunctions. So in practice, _I think_ this is less of an issue, because the expression,
```
a < 10 AND b > "aaa"
```
would be split into two `Expr`s when pushed down, e.g.,
```rust
// this:
assert_eq!(
filters,
&[col("a").lt(10), col("b").gt(lit("aaa"))]
);
// NOT this:
assert_eq!(
filters,
&[col("a").lt(10).and(col("b").gt(lit("aaa")))]
);
```
And because analysis is done _per_ `Expr` in the list, then we wouldn't run into the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org