You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2020/04/14 22:32:09 UTC
[impala] branch branch-3.4.0 updated (9dde3c1 -> 9f1c31c)
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a change to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 9dde3c1 IMPALA-9571: Fix Impala crash from unexpected boost filesystem_error exception
new 74e55ed IMPALA-8361: Propagate predicates of outer-joined InlineView
new 9f1c31c IMPALA-9529: Fix multi-tuple predicates not assigned in column masking
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../java/org/apache/impala/analysis/Analyzer.java | 142 +++++++----
.../apache/impala/analysis/DescriptorTable.java | 12 +-
.../org/apache/impala/analysis/InlineViewRef.java | 16 +-
.../apache/impala/analysis/MultiAggregateInfo.java | 1 +
.../org/apache/impala/analysis/SelectStmt.java | 3 +-
.../org/apache/impala/analysis/SlotDescriptor.java | 10 +-
.../java/org/apache/impala/analysis/TableRef.java | 11 +
.../apache/impala/analysis/TupleDescriptor.java | 20 +-
.../java/org/apache/impala/planner/JoinNode.java | 11 +
.../apache/impala/planner/SingleNodePlanner.java | 71 +++++-
.../apache/impala/analysis/AnalyzeStmtsTest.java | 2 +-
.../queries/PlannerTest/inline-view.test | 175 +++++++++++++-
.../queries/PlannerTest/nested-collections.test | 1 +
.../queries/PlannerTest/predicate-propagation.test | 34 +--
.../queries/QueryTest/ranger_column_masking.test | 264 +++++++++++++++++++++
15 files changed, 678 insertions(+), 95 deletions(-)
[impala] 01/02: IMPALA-8361: Propagate predicates of outer-joined
InlineView
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 74e55edd2c1a814e5ca4bfd7a60017e142f6ee98
Author: xqhe <he...@126.com>
AuthorDate: Wed Jan 15 21:31:24 2020 +0800
IMPALA-8361: Propagate predicates of outer-joined InlineView
This is an improvement that tries to propagate predicates of the
nullable side of the outer join into inline view.
For example:
SELECT *
FROM functional.alltypessmall a
LEFT JOIN (
SELECT id, upper(string_col) AS upper_val,
length(string_col) AS len
FROM functional.alltypestiny
) b ON a.id = b.id
WHERE b.upper_val is NULL and b.len = 0
Before this change, the predicate b.len=0 can't be migrated into inline
view since that is on the nullable side of an outer join if the
predicate evaluates in the inline view nulls will not be rejected.
However, we can be more aggressive. In particular, some predicates that
must be evaluted at a join node can also be safely evaluted by the
outer-joined inline view. Such predicates are not marked as assigned.
The predicates propagate into the inline view and also be evaluated at
a join node.
We can divide predicates into two types. One that satisfies the condition
that same as Analyzer#canEvalPredicate can be migrated into inline view，
and one that satisfies the below three conditions is safe to be propagated
into the nullable side of an outer join.
1) The predicate needs to be bound by tupleIds.
2) The predicate is not on-clause.
3) The predicate evaluates to false when all its referenced tuples are NULL.
Therefore, 'b.upper_val is NULL' cannot be propagated to inline view but
‘b.len = 0’ can be propagated to inline view.
Tests:
* Add plan tests in inline-view.test
* One baseline plan in inline-view.test, one in nested-collections.test
and two in predicate-propagation.test had to be updated
* Ran the full set of verifications in Impala Public Jenkins
Change-Id: I6c23a45aeb5dd1aa06a95c9aa8628ecbe37ef2c1
Reviewed-on: http://gerrit.cloudera.org:8080/15047
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
(cherry picked from commit 9cb6dabe10da0aa29d61b6bba0f82ec3cefc4e0d)
---
.../java/org/apache/impala/analysis/Analyzer.java | 112 +++++++------
.../apache/impala/analysis/MultiAggregateInfo.java | 1 +
.../org/apache/impala/analysis/SelectStmt.java | 3 +-
.../apache/impala/planner/SingleNodePlanner.java | 64 ++++++--
.../queries/PlannerTest/inline-view.test | 175 ++++++++++++++++++++-
.../queries/PlannerTest/nested-collections.test | 1 +
.../queries/PlannerTest/predicate-propagation.test | 34 ++--
7 files changed, 317 insertions(+), 73 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
index c842d64..be26b3e 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
@@ -1656,21 +1656,7 @@ public class Analyzer {
* Returns true if predicate 'e' can be correctly evaluated by a tree materializing
* 'tupleIds', otherwise false:
* - The predicate needs to be bound by tupleIds.
- * - For On-clause predicates:
- * - If the predicate is from an anti-join On-clause it must be evaluated by the
- * corresponding anti-join node.
- * - Predicates from the On-clause of an inner or semi join are evaluated at the
- * node that materializes the required tuple ids, unless they reference outer
- * joined tuple ids. In that case, the predicates are evaluated at the join node
- * of the corresponding On-clause.
- * - Predicates referencing full-outer joined tuples are assigned at the originating
- * join if it is a full-outer join, otherwise at the last full-outer join that does
- * not materialize the table ref ids of the originating join.
- * - Predicates from the On-clause of a left/right outer join are assigned at
- * the corresponding outer join node with the exception of simple predicates
- * that only reference a single tuple id. Those may be assigned below the
- * outer join node if they are from the same On-clause that makes the tuple id
- * nullable.
+ * - For On-clause predicates, see canEvalOnClauseConjunct() for more info.
* - Otherwise, a predicate can only be correctly evaluated if for all outer-joined
* referenced tids the last join to outer-join this tid has been materialized.
*/
@@ -1681,38 +1667,18 @@ public class Analyzer {
if (tids.isEmpty()) return true;
if (e.isOnClauseConjunct()) {
- if (isAntiJoinedConjunct(e)) return canEvalAntiJoinedConjunct(e, tupleIds);
-
- if (isIjConjunct(e) || isSjConjunct(e)) {
- if (!containsOuterJoinedTid(tids)) return true;
- // If the predicate references an outer-joined tuple, then evaluate it at
- // the join that the On-clause belongs to.
- TableRef onClauseTableRef = null;
- if (isIjConjunct(e)) {
- onClauseTableRef = globalState_.ijClauseByConjunct.get(e.getId());
- } else {
- onClauseTableRef = globalState_.sjClauseByConjunct.get(e.getId());
- }
- Preconditions.checkNotNull(onClauseTableRef);
- return tupleIds.containsAll(onClauseTableRef.getAllTableRefIds());
- }
-
- if (isFullOuterJoined(e)) return canEvalFullOuterJoinedConjunct(e, tupleIds);
- if (isOjConjunct(e)) {
- // Force this predicate to be evaluated by the corresponding outer join node.
- // The join node will pick up the predicate later via getUnassignedOjConjuncts().
- if (tids.size() > 1) return false;
- // Optimization for single-tid predicates: Legal to assign below the outer join
- // if the predicate is from the same On-clause that makes tid nullable
- // (otherwise e needn't be true when that tuple is set).
- TupleId tid = tids.get(0);
- return globalState_.ojClauseByConjunct.get(e.getId()) == getLastOjClause(tid);
- }
-
- // Should have returned in one of the cases above.
- Preconditions.checkState(false);
+ return canEvalOnClauseConjunct(tupleIds, e);
}
+ return isLastOjMaterializedByTupleIds(tupleIds, e);
+ }
+ /**
+ * Checks if for all outer-joined referenced tids of the predicate the last join
+ * to outer-join this tid has been materialized by tupleIds.
+ */
+ public boolean isLastOjMaterializedByTupleIds(List<TupleId> tupleIds, Expr e) {
+ List<TupleId> tids = new ArrayList<>();
+ e.getIds(tids, null);
for (TupleId tid: tids) {
TableRef rhsRef = getLastOjClause(tid);
// Ignore 'tid' because it is not outer-joined.
@@ -1724,6 +1690,62 @@ public class Analyzer {
}
/**
+ * Checks if a conjunct from the On-clause can be evaluated in a node that materializes
+ * a given list of tuple ids.
+ * - If the predicate is from an anti-join On-clause it must be evaluated by the
+ * corresponding anti-join node.
+ * - Predicates from the On-clause of an inner or semi join are evaluated at the node
+ * that materializes the required tuple ids, unless they reference outer joined tuple
+ * ids. In that case, the predicates are evaluated at the join node of the corresponding
+ * On-clause.
+ * - Predicates referencing full-outer joined tuples are assigned at the originating
+ * join if it is a full-outer join, otherwise at the last full-outer join that does not
+ * materialize the table ref ids of the originating join.
+ * - Predicates from the On-clause of a left/right outer join are assigned at the
+ * corresponding outer join node with the exception of simple predicates that only
+ * reference a single tuple id. Those may be assigned below the outer join node if they
+ * are from the same On-clause that makes the tuple id nullable.
+ */
+ public boolean canEvalOnClauseConjunct(List<TupleId> tupleIds, Expr e) {
+ Preconditions.checkState(e.isOnClauseConjunct());
+ if (isAntiJoinedConjunct(e)) return canEvalAntiJoinedConjunct(e, tupleIds);
+
+ List<TupleId> exprTids = new ArrayList<>();
+ e.getIds(exprTids, null);
+ if (exprTids.isEmpty()) return true;
+
+ if (isIjConjunct(e) || isSjConjunct(e)) {
+ if (!containsOuterJoinedTid(exprTids)) return true;
+ // If the predicate references an outer-joined tuple, then evaluate it at
+ // the join that the On-clause belongs to.
+ TableRef onClauseTableRef = null;
+ if (isIjConjunct(e)) {
+ onClauseTableRef = globalState_.ijClauseByConjunct.get(e.getId());
+ } else {
+ onClauseTableRef = globalState_.sjClauseByConjunct.get(e.getId());
+ }
+ Preconditions.checkNotNull(onClauseTableRef);
+ return tupleIds.containsAll(onClauseTableRef.getAllTableRefIds());
+ }
+
+ if (isFullOuterJoined(e)) return canEvalFullOuterJoinedConjunct(e, tupleIds);
+ if (isOjConjunct(e)) {
+ // Force this predicate to be evaluated by the corresponding outer join node.
+ // The join node will pick up the predicate later via getUnassignedOjConjuncts().
+ if (exprTids.size() > 1) return false;
+ // Optimization for single-tid predicates: Legal to assign below the outer join
+ // if the predicate is from the same On-clause that makes tid nullable
+ // (otherwise e needn't be true when that tuple is set).
+ TupleId tid = exprTids.get(0);
+ return globalState_.ojClauseByConjunct.get(e.getId()) == getLastOjClause(tid);
+ }
+
+ // Should have returned in one of the cases above.
+ Preconditions.checkState(false);
+ return false;
+ }
+
+ /**
* Checks if a conjunct from the On-clause of an anti join can be evaluated in a node
* that materializes a given list of tuple ids.
*/
diff --git a/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java b/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
index f054e04..436d607 100644
--- a/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
+++ b/fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
@@ -642,6 +642,7 @@ public class MultiAggregateInfo {
result.addAll(bindingPredicates);
result.addAll(unassignedConjuncts);
analyzer.createEquivConjuncts(tid, result, groupBySids);
+ Expr.removeDuplicates(result);
return result;
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java b/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
index 456ac0a..6f174d7 100644
--- a/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
@@ -982,7 +982,8 @@ public class SelectStmt extends QueryStmt {
@Override
public void materializeRequiredSlots(Analyzer analyzer) {
// Mark unassigned join predicates. Some predicates that must be evaluated by a join
- // can also be safely evaluated below the join (picked up by getBoundPredicates()).
+ // can also be safely evaluated below the join (picked up by getBoundPredicates() and
+ // migrateConjunctsToInlineView()).
// Such predicates will be marked twice and that is ok.
List<Expr> unassigned =
analyzer.getUnassignedConjuncts(getTableRefIds(), true);
diff --git a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java b/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
index 01921f2..2369142 100644
--- a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
+++ b/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
@@ -1165,6 +1165,56 @@ public class SingleNodePlanner {
}
/**
+ * Get the unassigned conjuncts that can be evaluated in inline view and return them
+ * in 'evalInInlineViewPreds'.
+ * If a conjunct is not an On-clause predicate and is safe to propagate it inside the
+ * inline view, add it to 'evalAfterJoinPreds'.
+ */
+ private void getConjunctsToInlineView(final Analyzer analyzer,
+ final InlineViewRef inlineViewRef, List<Expr> evalInInlineViewPreds,
+ List<Expr> evalAfterJoinPreds) {
+ List<Expr> unassignedConjuncts =
+ analyzer.getUnassignedConjuncts(inlineViewRef.getId().asList(), true);
+ List<TupleId> tupleIds = inlineViewRef.getId().asList();
+ for (Expr e: unassignedConjuncts) {
+ if (!e.isBoundByTupleIds(tupleIds)) continue;
+ List<TupleId> tids = new ArrayList<>();
+ e.getIds(tids, null);
+ if (tids.isEmpty()) {
+ evalInInlineViewPreds.add(e);
+ } else if (e.isOnClauseConjunct()) {
+ if (!analyzer.canEvalOnClauseConjunct(tupleIds, e)) continue;
+ evalInInlineViewPreds.add(e);
+ } else if (analyzer.isLastOjMaterializedByTupleIds(tupleIds, e)) {
+ evalInInlineViewPreds.add(e);
+ } else {
+ // The predicate belong to an outer-joined tuple, it's correct to duplicate(not
+ // migrate) this predicate inside the inline view since it does not evalute to
+ // true with null slots
+ try {
+ if (!analyzer.isTrueWithNullSlots(e)) {
+ evalAfterJoinPreds.add(e);
+ if (LOG.isTraceEnabled()) {
+ LOG.trace(String.format("Can propagate %s to inline view %s",
+ e.debugString(), inlineViewRef.getExplicitAlias()));
+ }
+ }
+ } catch (InternalException ex) {
+ // Expr evaluation failed in the backend. Skip 'e' since we cannot
+ // determine whether propagation is safe.
+ LOG.warn("Skipping propagation of conjunct because backend evaluation failed: "
+ + e.toSql(), ex);
+ }
+ continue;
+ }
+ if (LOG.isTraceEnabled()) {
+ LOG.trace(String.format("Can evaluate %s in inline view %s", e.debugString(),
+ inlineViewRef.getExplicitAlias()));
+ }
+ }
+ }
+
+ /**
* Migrates unassigned conjuncts into an inline view. Conjuncts are not
* migrated into the inline view if the view has a LIMIT/OFFSET clause or if the
* view's stmt computes analytic functions (see IMPALA-1243/IMPALA-1900).
@@ -1191,20 +1241,16 @@ public class SingleNodePlanner {
}
List<Expr> preds = new ArrayList<>();
- for (Expr e: unassignedConjuncts) {
- if (analyzer.canEvalPredicate(inlineViewRef.getId().asList(), e)) {
- preds.add(e);
- if (LOG. isTraceEnabled()) {
- LOG.trace(String.format("Can evaluate %s in inline view %s", e.debugString(),
- inlineViewRef.getExplicitAlias()));
- }
- }
- }
+ List<Expr> evalAfterJoinPreds = new ArrayList<>();
+ getConjunctsToInlineView(analyzer, inlineViewRef, preds, evalAfterJoinPreds);
unassignedConjuncts.removeAll(preds);
// Migrate the conjuncts by marking the original ones as assigned. They will either
// be ignored if they are identity predicates (e.g. a = a), or be substituted into
// new ones (viewPredicates below). The substituted ones will be re-registered.
analyzer.markConjunctsAssigned(preds);
+ // Propagate the conjuncts evaluating the nullable side of outer-join.
+ // Don't mark them as assigned so they would be assigned at the JOIN node.
+ preds.addAll(evalAfterJoinPreds);
// Generate predicates to enforce equivalences among slots of the inline view
// tuple. These predicates are also migrated into the inline view.
analyzer.createEquivConjuncts(inlineViewRef.getId(), preds);
diff --git a/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test b/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
index 4d2ba6d..60e0b78 100644
--- a/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
+++ b/testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
@@ -2160,6 +2160,7 @@ PLAN-ROOT SINK
====
# IMPALA-7957: Same thing except with a predicate on a different column (i.e. adding it
# to the equivalence class)
+# IMPALA-8361: Bound predicates optimization doesn't work for InlineView
SELECT t.id, t2.id
FROM functional.alltypestiny t
LEFT JOIN
@@ -2183,7 +2184,7 @@ PLAN-ROOT SINK
|
|--02:SCAN HDFS [functional.alltypestiny]
| HDFS partitions=4/4 files=4 size=460B
-| predicates: int_col = id, id = smallint_col
+| predicates: int_col = id, int_col = smallint_col
| row-size=10B cardinality=1
|
01:SCAN HDFS [functional.alltypestiny t]
@@ -2311,3 +2312,175 @@ PLAN-ROOT SINK
runtime filters: RF000 -> functional.alltypessmall.bigint_col
row-size=10B cardinality=81
====
+# IMPALA-8361: Bound predicates optimization doesn't work for InlineView
+SELECT *
+FROM functional.alltypessmall a
+ LEFT JOIN (
+ SELECT id, upper(string_col) AS upper_val
+ FROM functional.alltypestiny
+ ) b ON a.id = b.id
+WHERE b.upper_val = '1'
+---- PLAN
+PLAN-ROOT SINK
+|
+02:HASH JOIN [LEFT OUTER JOIN]
+| hash predicates: a.id = id
+| other predicates: upper(string_col) = '1'
+| row-size=106B cardinality=100
+|
+|--01:SCAN HDFS [functional.alltypestiny]
+| HDFS partitions=4/4 files=4 size=460B
+| predicates: upper(string_col) = '1'
+| row-size=17B cardinality=4
+|
+00:SCAN HDFS [functional.alltypessmall a]
+ HDFS partitions=4/4 files=4 size=6.32KB
+ row-size=89B cardinality=100
+====
+# IMPALA-8361: Bound predicates optimization doesn't work for InlineView
+# For local view
+WITH b as (SELECT id, upper(string_col) AS upper_val FROM functional.alltypestiny)
+SELECT * FROM functional.alltypessmall a LEFT JOIN b ON a.id = b.id
+WHERE b.upper_val = '1';
+---- PLAN
+PLAN-ROOT SINK
+|
+02:HASH JOIN [LEFT OUTER JOIN]
+| hash predicates: a.id = id
+| other predicates: upper(string_col) = '1'
+| row-size=106B cardinality=100
+|
+|--01:SCAN HDFS [functional.alltypestiny]
+| HDFS partitions=4/4 files=4 size=460B
+| predicates: upper(string_col) = '1'
+| row-size=17B cardinality=4
+|
+00:SCAN HDFS [functional.alltypessmall a]
+ HDFS partitions=4/4 files=4 size=6.32KB
+ row-size=89B cardinality=100
+====
+# IMPALA-8361: Bound predicates optimization doesn't work for InlineView
+# Outer-joined inline view is union statement
+SELECT * FROM functional.alltypestiny a
+LEFT JOIN
+(SELECT upper(a.string_col) upper_val, a.id FROM functional.alltypestiny a
+UNION SELECT b.string_col,b.id FROM functional.alltypestiny b) b
+ON a.id=b.id
+WHERE b.upper_val='123';
+---- PLAN
+PLAN-ROOT SINK
+|
+05:HASH JOIN [LEFT OUTER JOIN]
+| hash predicates: a.id = id
+| other predicates: upper_val = '123'
+| row-size=105B cardinality=8
+|
+|--04:AGGREGATE [FINALIZE]
+| | group by: upper_val, id
+| | row-size=16B cardinality=6
+| |
+| 01:UNION
+| | pass-through-operands: 03
+| | row-size=16B cardinality=6
+| |
+| |--02:SCAN HDFS [functional.alltypestiny a]
+| | HDFS partitions=4/4 files=4 size=460B
+| | predicates: upper(a.string_col) = '123'
+| | row-size=17B cardinality=2
+| |
+| 03:SCAN HDFS [functional.alltypestiny b]
+| HDFS partitions=4/4 files=4 size=460B
+| predicates: b.string_col = '123'
+| row-size=17B cardinality=4
+|
+00:SCAN HDFS [functional.alltypestiny a]
+ HDFS partitions=4/4 files=4 size=460B
+ row-size=89B cardinality=8
+====
+# IMPALA-8361: Bound predicates optimization doesn't work for InlineView
+# Inline view contains outer join
+SELECT * FROM functional.alltypestiny a
+LEFT JOIN
+(SELECT upper(b.string_col) string_col, b.id FROM functional.alltypestiny a
+LEFT JOIN
+functional.alltypestiny b ON a.id=b.id) b ON a.id=b.id
+WHERE b.string_col='1';
+---- PLAN
+PLAN-ROOT SINK
+|
+04:HASH JOIN [LEFT OUTER JOIN]
+| hash predicates: a.id = b.id
+| other predicates: upper(b.string_col) = '1'
+| row-size=110B cardinality=8
+|
+|--03:HASH JOIN [RIGHT OUTER JOIN]
+| | hash predicates: b.id = a.id
+| | other predicates: upper(b.string_col) = '1'
+| | runtime filters: RF000 <- a.id
+| | row-size=21B cardinality=8
+| |
+| |--01:SCAN HDFS [functional.alltypestiny a]
+| | HDFS partitions=4/4 files=4 size=460B
+| | row-size=4B cardinality=8
+| |
+| 02:SCAN HDFS [functional.alltypestiny b]
+| HDFS partitions=4/4 files=4 size=460B
+| predicates: upper(b.string_col) = '1'
+| runtime filters: RF000 -> b.id
+| row-size=17B cardinality=4
+|
+00:SCAN HDFS [functional.alltypestiny a]
+ HDFS partitions=4/4 files=4 size=460B
+ row-size=89B cardinality=8
+====
+# IMPALA-8361: Bound predicates optimization doesn't work for InlineView
+# More predicates
+SELECT *
+FROM functional.alltypessmall a
+ LEFT JOIN (
+ SELECT id + 1 as id, upper(string_col) AS upper_val, length(string_col) AS len
+ FROM functional.alltypestiny
+ ) b ON a.id = b.id
+WHERE b.upper_val is NULL and b.len = 0 and b.id > 0;
+---- PLAN
+PLAN-ROOT SINK
+|
+02:HASH JOIN [LEFT OUTER JOIN]
+| hash predicates: a.id = id + 1
+| other predicates: id + 1 > 0, upper(string_col) IS NULL, length(string_col) = 0
+| row-size=106B cardinality=100
+|
+|--01:SCAN HDFS [functional.alltypestiny]
+| HDFS partitions=4/4 files=4 size=460B
+| predicates: id + 1 > 0, length(string_col) = 0
+| row-size=17B cardinality=1
+|
+00:SCAN HDFS [functional.alltypessmall a]
+ HDFS partitions=4/4 files=4 size=6.32KB
+ row-size=89B cardinality=100
+====
+# IMPALA-8361: Bound predicates optimization doesn't work for InlineView
+# Where-clause predicate that can't be propagated
+SELECT *
+FROM functional.alltypessmall a
+ LEFT JOIN (
+ SELECT id, upper(string_col) AS upper_val
+ FROM functional.alltypestiny
+ ) b ON a.id = b.id
+WHERE b.upper_val is NULL;
+---- PLAN
+PLAN-ROOT SINK
+|
+02:HASH JOIN [LEFT OUTER JOIN]
+| hash predicates: a.id = id
+| other predicates: upper(string_col) IS NULL
+| row-size=106B cardinality=100
+|
+|--01:SCAN HDFS [functional.alltypestiny]
+| HDFS partitions=4/4 files=4 size=460B
+| row-size=17B cardinality=8
+|
+00:SCAN HDFS [functional.alltypessmall a]
+ HDFS partitions=4/4 files=4 size=6.32KB
+ row-size=89B cardinality=100
+====
diff --git a/testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test b/testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
index eaa167b..ff61101 100644
--- a/testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
+++ b/testdata/workloads/functional-planner/queries/PlannerTest/nested-collections.test
@@ -2290,6 +2290,7 @@ PLAN-ROOT SINK
| | row-size=16B cardinality=1
| |
| 03:UNNEST [t1.int_array_col t2]
+| predicates: item = 1
| row-size=12B cardinality=10
|
00:SCAN HDFS [functional.allcomplextypes t1]
diff --git a/testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test b/testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
index 7780a14..a969846 100644
--- a/testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
+++ b/testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
@@ -998,7 +998,8 @@ PLAN-ROOT SINK
|
|--02:AGGREGATE [FINALIZE]
| | group by: id, int_col
-| | row-size=8B cardinality=730
+| | having: int_col = 17
+| | row-size=8B cardinality=73
| |
| 01:SCAN HDFS [functional.alltypes]
| partitions=24/24 files=24 size=478.45KB
@@ -1059,26 +1060,25 @@ where a.id is null and isnull(b.id, 0) = 0 and b.int_col = 17
---- PLAN
PLAN-ROOT SINK
|
-03:HASH JOIN [RIGHT OUTER JOIN]
-| hash predicates: id = a.id
+03:HASH JOIN [LEFT OUTER JOIN]
+| hash predicates: a.id = id
| other predicates: int_col = 17, isnull(id, 0) = 0
-| runtime filters: RF000 <- a.id
| row-size=12B cardinality=730
|
-|--02:SCAN HDFS [functional.alltypes a]
-| partitions=24/24 files=24 size=478.45KB
-| predicates: a.id IS NULL
-| row-size=4B cardinality=730
-|
-01:AGGREGATE [FINALIZE]
-| group by: id, int_col
-| row-size=8B cardinality=730
+|--01:AGGREGATE [FINALIZE]
+| | group by: id, int_col
+| | having: int_col = 17
+| | row-size=8B cardinality=73
+| |
+| 00:SCAN HDFS [functional.alltypes]
+| HDFS partitions=24/24 files=24 size=478.45KB
+| predicates: functional.alltypes.int_col = 17
+| row-size=8B cardinality=730
|
-00:SCAN HDFS [functional.alltypes]
- partitions=24/24 files=24 size=478.45KB
- predicates: functional.alltypes.int_col = 17
- runtime filters: RF000 -> functional.alltypes.id
- row-size=8B cardinality=730
+02:SCAN HDFS [functional.alltypes a]
+ HDFS partitions=24/24 files=24 size=478.45KB
+ predicates: a.id IS NULL
+ row-size=4B cardinality=730
====
select straight_join a.id, b.id
from
[impala] 02/02: IMPALA-9529: Fix multi-tuple predicates not
assigned in column masking
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch branch-3.4.0
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 9f1c31c597b73662b76eb09a102bdeead58a96ed
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Mon Apr 6 10:14:38 2020 +0800
IMPALA-9529: Fix multi-tuple predicates not assigned in column masking
Column masking is implemented by replacing the masked table with a table
masking view which has masked expressions in its SelectList. However,
nested columns can't be exposed in the SelectList, so we expose them
in the output field of the view in IMPALA-9330. As a result, predicates
that reference both primitive and nested columns of the masked table
become multi-tuple predicates (referencing tuples of the view and the
masked table). Such kinds of predicates are not assigned since they no
longer bound to the view's tuple or the masked table's tuple.
We need to pick up the masked table's tuple id when getting unassigned
predicates for the table masking view. Also need to do this for
assigning predicates to the JoinNode which is the only place that
introduces multi-tuple predicates.
Tests:
- Add tests with multi-tuple predicates referencing nested columns.
- Run CORE tests.
Change-Id: I12f1b59733db5a88324bb0c16085f565edc306b3
Reviewed-on: http://gerrit.cloudera.org:8080/15654
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
(cherry picked from commit 5c2cae89f2ba1de55131a261dce7c8e284bb6c0e)
---
.../java/org/apache/impala/analysis/Analyzer.java | 30 ++-
.../apache/impala/analysis/DescriptorTable.java | 12 +-
.../org/apache/impala/analysis/InlineViewRef.java | 16 +-
.../org/apache/impala/analysis/SlotDescriptor.java | 10 +-
.../java/org/apache/impala/analysis/TableRef.java | 11 +
.../apache/impala/analysis/TupleDescriptor.java | 20 +-
.../java/org/apache/impala/planner/JoinNode.java | 11 +
.../apache/impala/planner/SingleNodePlanner.java | 27 ++-
.../apache/impala/analysis/AnalyzeStmtsTest.java | 2 +-
.../queries/QueryTest/ranger_column_masking.test | 264 +++++++++++++++++++++
10 files changed, 371 insertions(+), 32 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
index be26b3e..b2dc311 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
@@ -63,6 +63,7 @@ import org.apache.impala.common.ImpalaException;
import org.apache.impala.common.InternalException;
import org.apache.impala.common.Pair;
import org.apache.impala.common.RuntimeEnv;
+import org.apache.impala.planner.JoinNode;
import org.apache.impala.planner.PlanNode;
import org.apache.impala.rewrite.BetweenToCompoundRule;
import org.apache.impala.rewrite.EqualityDisjunctsToInRule;
@@ -1493,8 +1494,24 @@ public class Analyzer {
* Return all unassigned registered conjuncts for node's table ref ids.
* Wrapper around getUnassignedConjuncts(List<TupleId> tupleIds).
*/
- public List<Expr> getUnassignedConjuncts(PlanNode node) {
- return getUnassignedConjuncts(node.getTblRefIds());
+ public final List<Expr> getUnassignedConjuncts(PlanNode node) {
+ List<TupleId> tupleIds = Lists.newCopyOnWriteArrayList(node.getTblRefIds());
+ if (node instanceof JoinNode) {
+ for (TupleId tid : node.getTblRefIds()) {
+ // Pick up TupleId of the masked table since the table masking view and the masked
+ // table are the same in the original query. SlotRefs of table's nested columns
+ // are resolved into table's tuple, but not the table masking view's tuple. As a
+ // result, predicates referencing such nested columns also reference the masked
+ // table's tuple id.
+ TupleDescriptor tuple = getTupleDesc(tid);
+ Preconditions.checkNotNull(tuple);
+ BaseTableRef maskedTable = tuple.getMaskedTable();
+ if (maskedTable != null && maskedTable.exposeNestedColumnsByTableMaskView()) {
+ tupleIds.add(maskedTable.getId());
+ }
+ }
+ }
+ return getUnassignedConjuncts(tupleIds);
}
/**
@@ -2574,21 +2591,22 @@ public class Analyzer {
/**
* Mark all slots that are referenced in exprs as materialized.
+ * Return the affected Tuples.
*/
- public void materializeSlots(List<Expr> exprs) {
+ public Set<TupleDescriptor> materializeSlots(List<Expr> exprs) {
List<SlotId> slotIds = new ArrayList<>();
for (Expr e: exprs) {
Preconditions.checkState(e.isAnalyzed());
e.getIds(null, slotIds);
}
- globalState_.descTbl.markSlotsMaterialized(slotIds);
+ return globalState_.descTbl.markSlotsMaterialized(slotIds);
}
- public void materializeSlots(Expr e) {
+ public Set<TupleDescriptor> materializeSlots(Expr e) {
List<SlotId> slotIds = new ArrayList<>();
Preconditions.checkState(e.isAnalyzed());
e.getIds(null, slotIds);
- globalState_.descTbl.markSlotsMaterialized(slotIds);
+ return globalState_.descTbl.markSlotsMaterialized(slotIds);
}
/**
diff --git a/fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java b/fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
index ef06c76..7e41e1a 100644
--- a/fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
+++ b/fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
@@ -39,6 +39,7 @@ import org.apache.impala.thrift.TDescriptorTableSerialized;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
/**
* Repository for tuple (and slot) descriptors.
@@ -131,12 +132,17 @@ public class DescriptorTable {
}
/**
- * Marks all slots in list as materialized.
+ * Marks all slots in list as materialized and return the affected Tuples.
*/
- public void markSlotsMaterialized(List<SlotId> ids) {
+ public Set<TupleDescriptor> markSlotsMaterialized(List<SlotId> ids) {
+ Set<TupleDescriptor> affectedTuples = Sets.newHashSet();
for (SlotId id: ids) {
- getSlotDesc(id).setIsMaterialized(true);
+ SlotDescriptor slotDesc = getSlotDesc(id);
+ if (slotDesc.isMaterialized()) continue;
+ slotDesc.setIsMaterialized(true);
+ affectedTuples.add(slotDesc.getParent());
}
+ return affectedTuples;
}
/**
diff --git a/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java b/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
index 41106f0..f69f0e8 100644
--- a/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
+++ b/fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
@@ -309,6 +309,11 @@ public class InlineViewRef extends TableRef {
}
fields.add(new StructField(colAlias, selectItemExpr.getType(), null));
}
+
+ // Create the non-materialized tuple and set its type.
+ TupleDescriptor result = analyzer.getDescTbl().createTupleDescriptor(
+ getClass().getSimpleName() + " " + getUniqueAlias());
+ result.setIsMaterialized(false);
// If this is a table masking view, the underlying table is wrapped by this so its
// nested columns are not visible to the original query block, because we can't
// expose nested columns in the SelectList. However, we can expose nested columns
@@ -321,6 +326,7 @@ public class InlineViewRef extends TableRef {
if (tblRef instanceof BaseTableRef) {
BaseTableRef baseTbl = (BaseTableRef) tblRef;
FeTable tbl = baseTbl.resolvedPath_.getRootTable();
+ boolean exposeNestedColumn = false;
for (Column col : tbl.getColumnsInHiveOrder()) {
if (!col.getType().isComplexType()) continue;
if (LOG.isTraceEnabled()) {
@@ -328,14 +334,14 @@ public class InlineViewRef extends TableRef {
col.getName(), col.getType().toSql());
}
fields.add(new StructField(col.getName(), col.getType(), null));
+ exposeNestedColumn = true;
+ }
+ if (exposeNestedColumn) {
+ baseTbl.setExposeNestedColumnsByTableMaskView();
}
+ result.setMaskedTable(baseTbl);
}
}
-
- // Create the non-materialized tuple and set its type.
- TupleDescriptor result = analyzer.getDescTbl().createTupleDescriptor(
- getClass().getSimpleName() + " " + getUniqueAlias());
- result.setIsMaterialized(false);
result.setType(new StructType(fields));
return result;
}
diff --git a/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java b/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
index 4495f19..15f191c 100644
--- a/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
+++ b/fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
@@ -28,6 +28,8 @@ import org.apache.impala.catalog.FeKuduTable;
import org.apache.impala.catalog.KuduColumn;
import org.apache.impala.catalog.Type;
import org.apache.impala.thrift.TSlotDescriptor;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import com.google.common.base.Joiner;
import com.google.common.base.Objects;
@@ -35,6 +37,7 @@ import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
public class SlotDescriptor {
+ private final static Logger LOG = LoggerFactory.getLogger(SlotDescriptor.class);
private final SlotId id_;
private final TupleDescriptor parent_;
@@ -118,7 +121,12 @@ public class SlotDescriptor {
itemTupleDesc_ = t;
}
public boolean isMaterialized() { return isMaterialized_; }
- public void setIsMaterialized(boolean value) { isMaterialized_ = value; }
+ public void setIsMaterialized(boolean value) {
+ if (isMaterialized_ == value) return;
+ isMaterialized_ = value;
+ LOG.trace("Mark slot(sid={}) of tuple(tid={}) as {}materialized",
+ id_, parent_.getId(), isMaterialized_ ? "" : "non-");
+ }
public boolean getIsNullable() { return isNullable_; }
public void setIsNullable(boolean value) { isNullable_ = value; }
public int getByteSize() { return byteSize_; }
diff --git a/fe/src/main/java/org/apache/impala/analysis/TableRef.java b/fe/src/main/java/org/apache/impala/analysis/TableRef.java
index 47be874..3290edd 100644
--- a/fe/src/main/java/org/apache/impala/analysis/TableRef.java
+++ b/fe/src/main/java/org/apache/impala/analysis/TableRef.java
@@ -132,6 +132,10 @@ public class TableRef extends StmtNode {
// analysis output
protected TupleDescriptor desc_;
+ // true if this table is masked by a table masking view and need to expose its nested
+ // columns via the view.
+ protected boolean exposeNestedColumnsByTableMaskView_ = false;
+
// END: Members that need to be reset()
/////////////////////////////////////////
@@ -197,6 +201,7 @@ public class TableRef extends StmtNode {
allMaterializedTupleIds_ = Lists.newArrayList(other.allMaterializedTupleIds_);
correlatedTupleIds_ = Lists.newArrayList(other.correlatedTupleIds_);
desc_ = other.desc_;
+ exposeNestedColumnsByTableMaskView_ = other.exposeNestedColumnsByTableMaskView_;
}
@Override
@@ -295,6 +300,12 @@ public class TableRef extends StmtNode {
public void setUsingClause(List<String> colNames) { this.usingColNames_ = colNames; }
public TableRef getLeftTblRef() { return leftTblRef_; }
public void setLeftTblRef(TableRef leftTblRef) { this.leftTblRef_ = leftTblRef; }
+ public void setExposeNestedColumnsByTableMaskView() {
+ exposeNestedColumnsByTableMaskView_ = true;
+ }
+ public boolean exposeNestedColumnsByTableMaskView() {
+ return exposeNestedColumnsByTableMaskView_;
+ }
public void setJoinHints(List<PlanHint> hints) {
Preconditions.checkNotNull(hints);
diff --git a/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java b/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
index 557874d..3df980d 100644
--- a/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
+++ b/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
@@ -104,6 +104,11 @@ public class TupleDescriptor {
private int numNullBytes_;
private float avgSerializedSize_; // in bytes; includes serialization overhead
+ // Underlying masked table if this is the tuple of a table masking view.
+ private BaseTableRef maskedTable_ = null;
+ // Tuple of the table masking view that masks this tuple's table.
+ private TupleDescriptor maskedByTuple_ = null;
+
public TupleDescriptor(TupleId id, String debugName) {
id_ = id;
path_ = null;
@@ -190,20 +195,29 @@ public class TupleDescriptor {
return path_.getRootDesc();
}
+ public TupleDescriptor getMaskedByTuple() { return maskedByTuple_; }
+ public BaseTableRef getMaskedTable() { return maskedTable_; }
+ public void setMaskedTable(BaseTableRef table) {
+ Preconditions.checkState(maskedTable_ == null);
+ maskedTable_ = table;
+ table.getDesc().maskedByTuple_ = this;
+ }
+
public String debugString() {
String tblStr = (getTable() == null ? "null" : getTable().getFullName());
List<String> slotStrings = new ArrayList<>();
for (SlotDescriptor slot : slots_) {
slotStrings.add(slot.debugString());
}
- return Objects.toStringHelper(this)
+ Objects.ToStringHelper toStrHelper = Objects.toStringHelper(this)
.add("id", id_.asInt())
.add("name", debugName_)
.add("tbl", tblStr)
.add("byte_size", byteSize_)
.add("is_materialized", isMaterialized_)
- .add("slots", "[" + Joiner.on(", ").join(slotStrings) + "]")
- .toString();
+ .add("slots", "[" + Joiner.on(", ").join(slotStrings) + "]");
+ if (maskedTable_ != null) toStrHelper.add("masks", maskedTable_.getId());
+ return toStrHelper.toString();
}
@Override
diff --git a/fe/src/main/java/org/apache/impala/planner/JoinNode.java b/fe/src/main/java/org/apache/impala/planner/JoinNode.java
index a3d6042..5eaf870 100644
--- a/fe/src/main/java/org/apache/impala/planner/JoinNode.java
+++ b/fe/src/main/java/org/apache/impala/planner/JoinNode.java
@@ -29,6 +29,7 @@ import org.apache.impala.analysis.Expr;
import org.apache.impala.analysis.JoinOperator;
import org.apache.impala.analysis.SlotDescriptor;
import org.apache.impala.analysis.SlotRef;
+import org.apache.impala.analysis.TupleDescriptor;
import org.apache.impala.analysis.TupleId;
import org.apache.impala.catalog.ColumnStats;
import org.apache.impala.catalog.FeTable;
@@ -209,6 +210,16 @@ public abstract class JoinNode extends PlanNode {
// have been collected.
assignConjuncts(analyzer);
createDefaultSmap(analyzer);
+ // Mark slots used by 'conjuncts_' as materialized after substitution. Recompute
+ // memory layout for affected tuples. Note: only tuples of the masked tables could
+ // be affected if they are referenced by multi-tuple predicates.
+ for (TupleDescriptor tuple : analyzer.materializeSlots(conjuncts_)) {
+ if (LOG.isTraceEnabled()) {
+ LOG.trace("Recompute mem layout for " + tuple.debugString());
+ }
+ Preconditions.checkNotNull(tuple.getMaskedByTuple());
+ tuple.recomputeMemLayout();
+ }
assignedConjuncts_ = analyzer.getAssignedConjuncts();
otherJoinConjuncts_ = Expr.substituteList(otherJoinConjuncts_,
getCombinedChildSmap(), analyzer, false);
diff --git a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java b/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
index 2369142..0eb3c61 100644
--- a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
+++ b/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
@@ -1170,12 +1170,10 @@ public class SingleNodePlanner {
* If a conjunct is not an On-clause predicate and is safe to propagate it inside the
* inline view, add it to 'evalAfterJoinPreds'.
*/
- private void getConjunctsToInlineView(final Analyzer analyzer,
- final InlineViewRef inlineViewRef, List<Expr> evalInInlineViewPreds,
+ private void getConjunctsToInlineView(final Analyzer analyzer, final String alias,
+ final List<TupleId> tupleIds, List<Expr> evalInInlineViewPreds,
List<Expr> evalAfterJoinPreds) {
- List<Expr> unassignedConjuncts =
- analyzer.getUnassignedConjuncts(inlineViewRef.getId().asList(), true);
- List<TupleId> tupleIds = inlineViewRef.getId().asList();
+ List<Expr> unassignedConjuncts = analyzer.getUnassignedConjuncts(tupleIds, true);
for (Expr e: unassignedConjuncts) {
if (!e.isBoundByTupleIds(tupleIds)) continue;
List<TupleId> tids = new ArrayList<>();
@@ -1195,8 +1193,7 @@ public class SingleNodePlanner {
if (!analyzer.isTrueWithNullSlots(e)) {
evalAfterJoinPreds.add(e);
if (LOG.isTraceEnabled()) {
- LOG.trace(String.format("Can propagate %s to inline view %s",
- e.debugString(), inlineViewRef.getExplicitAlias()));
+ LOG.trace("Can propagate {} to inline view {}", e.debugString(), alias);
}
}
} catch (InternalException ex) {
@@ -1208,8 +1205,7 @@ public class SingleNodePlanner {
continue;
}
if (LOG.isTraceEnabled()) {
- LOG.trace(String.format("Can evaluate %s in inline view %s", e.debugString(),
- inlineViewRef.getExplicitAlias()));
+ LOG.trace("Can evaluate {} in inline view {}", e.debugString(), alias);
}
}
}
@@ -1226,9 +1222,13 @@ public class SingleNodePlanner {
*/
public void migrateConjunctsToInlineView(final Analyzer analyzer,
final InlineViewRef inlineViewRef) throws ImpalaException {
- List<Expr> unassignedConjuncts =
- analyzer.getUnassignedConjuncts(inlineViewRef.getId().asList(), true);
- if (LOG. isTraceEnabled()) {
+ List<TupleId> tids = inlineViewRef.getId().asList();
+ if (inlineViewRef.isTableMaskingView()
+ && inlineViewRef.getUnMaskedTableRef().exposeNestedColumnsByTableMaskView()) {
+ tids.add(inlineViewRef.getUnMaskedTableRef().getId());
+ }
+ List<Expr> unassignedConjuncts = analyzer.getUnassignedConjuncts(tids, true);
+ if (LOG.isTraceEnabled()) {
LOG.trace("unassignedConjuncts: " + Expr.debugString(unassignedConjuncts));
}
if (!canMigrateConjuncts(inlineViewRef)) {
@@ -1242,7 +1242,8 @@ public class SingleNodePlanner {
List<Expr> preds = new ArrayList<>();
List<Expr> evalAfterJoinPreds = new ArrayList<>();
- getConjunctsToInlineView(analyzer, inlineViewRef, preds, evalAfterJoinPreds);
+ getConjunctsToInlineView(analyzer, inlineViewRef.getExplicitAlias(), tids, preds,
+ evalAfterJoinPreds);
unassignedConjuncts.removeAll(preds);
// Migrate the conjuncts by marking the original ones as assigned. They will either
// be ignored if they are identity predicates (e.g. a = a), or be substituted into
diff --git a/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java b/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
index 7f52760..dba2a85 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
@@ -4043,7 +4043,7 @@ public class AnalyzeStmtsTest extends AnalyzerTest {
testNumberOfMembers(ValuesStmt.class, 0);
// Also check TableRefs.
- testNumberOfMembers(TableRef.class, 21);
+ testNumberOfMembers(TableRef.class, 22);
testNumberOfMembers(BaseTableRef.class, 0);
testNumberOfMembers(InlineViewRef.class, 9);
}
diff --git a/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test b/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test
index 02451b3..48467c5 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test
@@ -751,3 +751,267 @@ where a2.item = 1 and a3.item = 2 and d2.e = 10 and d3.e = -10
---- TYPES
BIGINT,INT,INT,INT,INT,INT,INT,STRING,INT,STRING
====
+---- QUERY
+# IMPALA-9529: Test predicates that can be resolved to have different tuple ids.
+select id, nested_struct.a from functional_parquet.complextypestbl t
+where id = 100 or nested_struct.a = 1;
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates that can be resolved to have different tuple ids.
+select id, nested_struct.a from functional_parquet.complextypestbl t
+where id + nested_struct.a = 101;
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates that can be resolved to have different tuple ids.
+select id, id2 from (
+ select id, id as id2 from functional.alltypestiny
+ union all
+ select id, nested_struct.a as id2 from functional_parquet.complextypestbl
+) t
+where id + id2 = 101;
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates that can be resolved to have different tuple ids.
+with v as (
+ select id, nested_struct.a as id2 from functional_parquet.complextypestbl
+)
+select id, id2 from v
+where id + id2 = 101 or (id = 200 and id2 is null);
+---- RESULTS
+100,1
+200,NULL
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids.
+select t.id, t.nested_struct.a
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+where tiny.id + t.id + t.nested_struct.a = 201;
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids.
+select t.id, t.nested_struct.a
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id + t.id + t.nested_struct.a = 201;
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids. Make sure nested columns inside
+# the column masking view are materialized.
+select count(1)
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+where (tiny.id + t.id + t.nested_struct.a) is null;
+---- RESULTS
+5
+---- TYPES
+BIGINT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids in GroupBy clause.
+select count(1)
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+group by tiny.id + t.id + t.nested_struct.a;
+---- RESULTS
+1
+5
+1
+---- TYPES
+BIGINT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids in Having clause.
+select count(tiny.id + t.id + t.nested_struct.a)
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+group by t.id
+having count(tiny.id + t.id + t.nested_struct.a) = 1
+---- RESULTS
+1
+1
+---- TYPES
+BIGINT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids in OrderBy clause.
+select t.id
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+order by tiny.id + t.id + t.nested_struct.a, t.id;
+---- RESULTS: VERIFY_IS_EQUAL
+100
+700
+200
+300
+400
+500
+600
+---- TYPES
+BIGINT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids in analytic query.
+select t.id, rank() over(order by t.id)
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+where tiny.id + t.id + t.nested_struct.a is null;
+---- RESULTS
+200,1
+300,2
+400,3
+500,4
+600,5
+---- TYPES
+BIGINT,BIGINT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids.
+select t.id, t.nested_struct.a
+from functional.alltypestiny tiny
+ left join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+where tiny.id + t.id + t.nested_struct.a = 201;
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids.
+select t.id, t.nested_struct.a
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+ join functional.alltypestiny tiny2
+ on t.id = tiny2.id
+where tiny.id + t.id + t.nested_struct.a + tiny2.id = 301;
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids.
+select t.id, t.nested_struct.a
+from functional.alltypestiny tiny
+ left join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+ join functional.alltypestiny tiny2
+ on t.id = tiny2.id
+where tiny.id + t.id + t.nested_struct.a + tiny2.id = 301;
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+# IMPALA-9529: Test predicates with multiple tuple ids.
+with v as (
+ select t.id, t.nested_struct.a
+ from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+ where tiny.id + t.id + t.nested_struct.a = 201
+)
+select t.id, t.nested_struct.a from v
+ join functional_parquet.complextypestbl t
+ on v.id = t.id
+where v.id + t.id + t.nested_struct.a = 201
+---- RESULTS
+100,1
+---- TYPES
+BIGINT,INT
+====
+---- QUERY
+select count(distinct id), count(distinct nested_struct.a) from functional_parquet.complextypestbl
+---- RESULTS
+8,3
+---- TYPES
+BIGINT,BIGINT
+====
+---- QUERY
+select count(distinct id), count(distinct nested_struct.a) from functional_parquet.complextypestbl
+where id + nested_struct.a = 101;
+---- RESULTS
+1,1
+---- TYPES
+BIGINT,BIGINT
+====
+---- QUERY
+select count(distinct tiny.int_col), count(distinct t.id), count(distinct t.nested_struct.a)
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+---- RESULTS
+2,7,2
+---- TYPES
+BIGINT,BIGINT,BIGINT
+====
+---- QUERY
+select count(distinct tiny.int_col), count(distinct t.id), count(distinct t.nested_struct.a)
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t
+ on tiny.id = t.id
+where tiny.int_col + t.id + t.nested_struct.a = 102
+---- RESULTS
+1,1,1
+---- TYPES
+BIGINT,BIGINT,BIGINT
+====
+---- QUERY
+select tiny.id, t0.nested_struct.a
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t0 on tiny.id = t0.id
+ join functional_parquet.complextypestbl t1 on tiny.id = t1.id
+ join functional_parquet.complextypestbl t2 on tiny.id = t2.id
+ join functional_parquet.complextypestbl t3 on tiny.id = t3.id
+where tiny.id + t2.id + t3.nested_struct.a >= 201
+---- RESULTS
+100,1
+700,7
+---- TYPES
+INT,INT
+====
+---- QUERY
+select tiny.id, t0.nested_struct.a
+from functional.alltypestiny tiny
+ join functional_parquet.complextypestbl t0 on tiny.id = t0.id
+ join functional_parquet.complextypestbl t1 on tiny.id = t1.id
+ join functional_parquet.complextypestbl t2 on tiny.id = t2.id
+ join functional_parquet.complextypestbl t3 on tiny.id = t3.id
+where (t2.id + t3.id + t3.nested_struct.a = 201 or t2.nested_struct.a is null)
+ and tiny.id + t0.id + t3.nested_struct.a >= 201
+---- RESULTS
+100,1
+---- TYPES
+INT,INT
+====
\ No newline at end of file