You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/24 21:18:26 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue #420: Support for `!=` predicate in pruning predicates
alamb opened a new issue #420:
URL: https://github.com/apache/arrow-datafusion/issues/420
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While evaluating queries against data stored in containers / multiple files, it is helpful to prune entire files using statistics (see #363 for more details). DataFusion has this logic already for `==`, `<`, `<=`, `>` and `>=` operators in the pruning predicate.
However, as @NGA-TRAN noticed, there is no support for `!=` at the moment.
https://github.com/apache/arrow-datafusion/blob/14f1eebef068a9e65f556ed74d2b6d98376c97f4/datafusion/src/physical_plan/parquet.rs#L683
**Describe the solution you'd like**
Add support and tests for `!=` in predicate pruning logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #420: Support for `!=` predicate in pruning predicates
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #420:
URL: https://github.com/apache/arrow-datafusion/issues/420#issuecomment-859114509
Thanks @jgoday ! That would be great
For `!=` I think we would prune the container (aka return false or NULL) from the predicate if the constant value *did* fall within the min/max bounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb edited a comment on issue #420: Support for `!=` predicate in pruning predicates
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on issue #420:
URL: https://github.com/apache/arrow-datafusion/issues/420#issuecomment-859114509
Thanks @jgoday ! That would be great
For `!=` I think we would prune the container (aka return false or NULL) from the predicate if the constant value *did* fall within the min/max bounds but I might be misunderstanding what you are saying; I'll love to check out the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #420: Support for `!=` predicate in pruning predicates
Posted by GitBox <gi...@apache.org>.
alamb closed issue #420:
URL: https://github.com/apache/arrow-datafusion/issues/420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jgoday commented on issue #420: Support for `!=` predicate in pruning predicates
Posted by GitBox <gi...@apache.org>.
jgoday commented on issue #420:
URL: https://github.com/apache/arrow-datafusion/issues/420#issuecomment-859926283
@alamb I think that I express myself incorrectly :)
Created #544 PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jgoday commented on issue #420: Support for `!=` predicate in pruning predicates
Posted by GitBox <gi...@apache.org>.
jgoday commented on issue #420:
URL: https://github.com/apache/arrow-datafusion/issues/420#issuecomment-858776910
Can I try to solve this issue ?
If I understand it correctly, for the non equal predicate the expression should be pruned if the literal value does not fall between the min and max values, Am I right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jgoday removed a comment on issue #420: Support for `!=` predicate in pruning predicates
Posted by GitBox <gi...@apache.org>.
jgoday removed a comment on issue #420:
URL: https://github.com/apache/arrow-datafusion/issues/420#issuecomment-858776910
Can I try to solve this issue ?
If I understand it correctly, for the non equal predicate the expression should be pruned if the literal value does not fall between the min and max values, Am I right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jgoday commented on issue #420: Support for `!=` predicate in pruning predicates
Posted by GitBox <gi...@apache.org>.
jgoday commented on issue #420:
URL: https://github.com/apache/arrow-datafusion/issues/420#issuecomment-858780000
@alamb Can I try to solve this issue ?
If I understand it correctly, for the non equal predicate the expression should be pruned if the literal value does not fall between the min and max values, Am I right ?
(I have already tried to implement it here (https://github.com/jgoday/arrow-datafusion/commit/3bb55a4464d23e2a4eb319aac1b51361f31c7485), I can make a PR if you approve it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org