You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zach Amsden (Code Review)" <ge...@cloudera.org> on 2017/04/15 00:04:57 UTC
[Impala-ASF-CR] IMPALA-5159: Optimze predicates
Zach Amsden has uploaded a new change for review.
http://gerrit.cloudera.org:8080/6650
Change subject: IMPALA-5159: Optimze predicates
......................................................................
IMPALA-5159: Optimze predicates
In addition to propagating constants, we can infer bounds on slotRefs
which are implied by other constant bounds. Such conjuncts may help
us create min/max filters and/or restrict partitions. In addition,
there is no reason not to support null-safe operators for use in joins.
In many cases, we may derive contradictions from empty ranges, and
also eliminate tests which are outside of the range.
Testing: manual testing. This will have to be staged as a change to
the normalization rule, along with all the planner tests that get
changed, then an additional checkin to support this optimization.
For example:
int_col1 >= int_col2 and int_col2 >= int_col1
predicates: int_col1 = int_col2
int_col1 >= 10 and int_col2 < int_col1 and int_col2 > 15
and int_col1 != -100
predicates: int_col1 > 15, int_col1 > int_col2, int_col2 > 15
int_col1 <=> int_col2 and int_col1 >= int_col2
predicates: int_col1 = int_col2
int_col1 <= int_col2 and int_col2 <= int_col3 and int_col1 >= int_col3
predicate: int_col1 = int_col2, int_col1 = int_col3
Change-Id: I708ce5aa2e04f8d572af6dbc9b4e49b662843f34
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/planner/ValueRange.java
M fe/src/main/java/org/apache/impala/rewrite/NormalizeBinaryPredicatesRule.java
5 files changed, 701 insertions(+), 61 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/6650/1
--
To view, visit http://gerrit.cloudera.org:8080/6650
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I708ce5aa2e04f8d572af6dbc9b4e49b662843f34
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
[Impala-ASF-CR] IMPALA-5159: Optimze predicates [DRAFT]
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has uploaded a new patch set (#2).
Change subject: IMPALA-5159: Optimze predicates [DRAFT]
......................................................................
IMPALA-5159: Optimze predicates [DRAFT]
In addition to propagating constants, we can infer bounds on slotRefs
which are implied by other constant bounds. Such conjuncts may help
us create min/max filters and/or restrict partitions. In addition,
there is no reason not to support null-safe operators for use in joins.
In many cases, we may derive contradictions from empty ranges, and
also eliminate tests which are outside of the range.
Testing: Manual testing. No currently known bugs.
This will have to be staged as a change to
the normalization rule, along with all the planner tests that get
changed, then an additional checkin to support this optimization.
For example:
int_col1 >= int_col2 and int_col2 >= int_col1
predicates: int_col1 = int_col2
int_col1 >= 10 and int_col2 < int_col1 and int_col2 > 15
and int_col1 != -100
predicates: int_col1 > 15, int_col1 > int_col2, int_col2 > 15
int_col1 <=> int_col2 and int_col1 >= int_col2
predicates: int_col1 = int_col2
int_col1 <= int_col2 and int_col2 <= int_col3 and int_col1 >= int_col3
predicate: int_col1 = int_col2, int_col1 = int_col3
Change-Id: I708ce5aa2e04f8d572af6dbc9b4e49b662843f34
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/planner/ValueRange.java
M fe/src/main/java/org/apache/impala/rewrite/NormalizeBinaryPredicatesRule.java
5 files changed, 722 insertions(+), 61 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/6650/2
--
To view, visit http://gerrit.cloudera.org:8080/6650
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I708ce5aa2e04f8d572af6dbc9b4e49b662843f34
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
[Impala-ASF-CR] IMPALA-5159: Optimze predicates [DRAFT]
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has abandoned this change.
Change subject: IMPALA-5159: Optimze predicates [DRAFT]
......................................................................
Abandoned
Shelving this for now. Let's work on more important tasks and come back to inequality propagation in cases where we can see benefits with column statistics or partition elimination.
--
To view, visit http://gerrit.cloudera.org:8080/6650
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: abandon
Gerrit-Change-Id: I708ce5aa2e04f8d572af6dbc9b4e49b662843f34
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
[Impala-ASF-CR] IMPALA-5159: Optimze predicates
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has posted comments on this change.
Change subject: IMPALA-5159: Optimze predicates
......................................................................
Patch Set 1:
So I don't think our JVM implementation likes calling .add and .remove on the same ListIterator, despite that being the advertised purpose. Consider this change a draft until I've attached the requisite tests.
--
To view, visit http://gerrit.cloudera.org:8080/6650
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I708ce5aa2e04f8d572af6dbc9b4e49b662843f34
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-HasComments: No