You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by bi...@apache.org on 2021/02/17 02:13:18 UTC
[impala] branch master updated (4721978 -> 9736e4c)
This is an automated email from the ASF dual-hosted git repository.
bikram pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 4721978 IMPALA-4805: Avoid hash exchange before analytic function if appropriate
new baf81de IMPALA-9745: Propagate source type when doing constant propagation
new dc133d9 IMPALA-10499: Fix failing test_misc
new 9736e4c IMPALA-10389: (supplement) Add impala-profile-tool to build by default
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CMakeLists.txt | 2 +-
.../apache/impala/analysis/ConstantPredicateHandler.java | 13 +++++++++++--
.../queries/PlannerTest/constant-propagation.test | 13 +++++++++++++
.../workloads/functional-query/queries/QueryTest/misc.test | 2 +-
.../queries/QueryTest/range-constant-propagation.test | 14 ++++++++++++++
5 files changed, 40 insertions(+), 4 deletions(-)
[impala] 02/03: IMPALA-10499: Fix failing test_misc
Posted by bi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bikram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit dc133d951389924e1f38856db1777d2b334dc7a2
Author: Tamas Mate <tm...@cloudera.com>
AuthorDate: Sun Feb 14 19:49:15 2021 +0100
IMPALA-10499: Fix failing test_misc
This change modifies the result type of the misc test which was failing.
Testing:
- executed the misc tests with exhaustive exploration strategy
Change-Id: Ibe95f4bc3521f49d19e6da53deb904a25ac30982
Reviewed-on: http://gerrit.cloudera.org:8080/17066
Reviewed-by: Aman Sinha <am...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
testdata/workloads/functional-query/queries/QueryTest/misc.test | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/testdata/workloads/functional-query/queries/QueryTest/misc.test b/testdata/workloads/functional-query/queries/QueryTest/misc.test
index 2f48788..6e8d408 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/misc.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/misc.test
@@ -115,7 +115,7 @@ SELECT 1 as "``"
---- RESULTS
1
---- TYPES
-int
+tinyint
====
---- QUERY
# Select from table that contains unsupported primitive types
[impala] 01/03: IMPALA-9745: Propagate source type when doing
constant propagation
Posted by bi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bikram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit baf81dea6d752d92de27165509d39c345e4f6503
Author: Aman Sinha <am...@cloudera.com>
AuthorDate: Fri Feb 12 11:31:13 2021 -0800
IMPALA-9745: Propagate source type when doing constant propagation
When doing constant propagation the source type was not being
propagated to the target expression leading to an analysis failure.
The behavior is most easily reproducible with STRING to TIMESTAMP
conversion in the presence of other predicates.
This patch fixes this by adding an implicit cast if needed for such
cases.
Testing:
- Added planner test and ran other planner tests
- Added end-to-end test
Change-Id: Ic3853478945229440f733c256ea225639f9178ff
Reviewed-on: http://gerrit.cloudera.org:8080/17064
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Aman Sinha <am...@cloudera.com>
---
.../apache/impala/analysis/ConstantPredicateHandler.java | 13 +++++++++++--
.../queries/PlannerTest/constant-propagation.test | 13 +++++++++++++
.../queries/QueryTest/range-constant-propagation.test | 14 ++++++++++++++
3 files changed, 38 insertions(+), 2 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java b/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java
index ac48f67..36ae913 100644
--- a/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java
+++ b/fe/src/main/java/org/apache/impala/analysis/ConstantPredicateHandler.java
@@ -107,7 +107,7 @@ public class ConstantPredicateHandler {
Preconditions.checkNotNull(slotRef);
Expr subst = bp.getSlotBinding(slotRef.getSlotId());
ExprSubstitutionMap smap = new ExprSubstitutionMap();
- smap.put(slotRef, subst);
+ smap.put(slotRef, getTargetExpr(slotRef, subst));
for (int j = 0; j < conjuncts.size(); ++j) {
Expr toRewrite = conjuncts.get(j);
// Don't rewrite with our own substitution!
@@ -141,7 +141,7 @@ public class ConstantPredicateHandler {
BinaryPredicate bp = (BinaryPredicate) rangePred;
Expr subst = bp.getSlotBinding(slotRef.getSlotId());
ExprSubstitutionMap smap = new ExprSubstitutionMap();
- smap.put(slotRef, subst);
+ smap.put(slotRef, getTargetExpr(slotRef, subst));
// make a copy since we may need to re-use the original
Expr toRewrite = dtExpr.clone();
toRewrite = rewriteWithOp((BinaryPredicate) toRewrite, bp.getOp(), analyzer);
@@ -179,4 +179,13 @@ public class ConstantPredicateHandler {
return newPred;
}
+ // During constant propagation, the types also need to be propagated since a
+ // rewritten expr must have compatible data type with the source expr. So,
+ // when mapping the source to the target expr we add an implicit cast if needed.
+ private static Expr getTargetExpr(SlotRef source, Expr target) {
+ Expr expr = source.getType().equals(target.getType()) ?
+ target : new CastExpr(source.getType(), target);
+ return expr;
+ }
+
}
diff --git a/testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test b/testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test
index 02e4ccf..5e645c9 100644
--- a/testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test
+++ b/testdata/workloads/functional-planner/queries/PlannerTest/constant-propagation.test
@@ -526,3 +526,16 @@ PLAN-ROOT SINK
predicates: timestamp_col > TIMESTAMP '2009-02-01 00:00:00', date_col = CAST(timestamp_col AS DATE)
row-size=65B cardinality=21
====
+# IMPALA-9745
+# Constant propagation in the presence of implicit casts
+select * from
+ (select o_orderdate, to_timestamp(o_orderdate, 'yyyy-MM-dd') ts
+ from tpch.orders) dt where ts = '1996-12-01' and o_orderdate = ts;
+---- PLAN
+PLAN-ROOT SINK
+|
+00:SCAN HDFS [tpch.orders]
+ HDFS partitions=1/1 files=1 size=162.56MB
+ predicates: tpch.orders.o_orderdate = TIMESTAMP '1996-12-01 00:00:00'
+ row-size=22B cardinality=608
+====
diff --git a/testdata/workloads/functional-query/queries/QueryTest/range-constant-propagation.test b/testdata/workloads/functional-query/queries/QueryTest/range-constant-propagation.test
index 10ad644..aa71414 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/range-constant-propagation.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/range-constant-propagation.test
@@ -48,3 +48,17 @@ select count(*) from (select * from alltypes_dp_2_view_2 limit 10) t;
---- TYPES
BIGINT
====
+---- QUERY
+# IMPALA-9745
+# Test correctness of constant propagation in the presence of
+# implicit casts
+select * from
+ (select o_orderdate, to_timestamp(o_orderdate, 'yyyy-MM-dd') ts
+ from tpch.orders) dt where ts = '1996-12-01' and o_orderdate = ts
+limit 2;
+---- RESULTS
+'1996-12-01',1996-12-01 00:00:00
+'1996-12-01',1996-12-01 00:00:00
+---- TYPES
+STRING,TIMESTAMP
+====
[impala] 03/03: IMPALA-10389: (supplement) Add impala-profile-tool
to build by default
Posted by bi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bikram pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 9736e4c4777bdac306e8dd3fce58061ecc5d6692
Author: Bikramjeet Vig <bi...@gmail.com>
AuthorDate: Tue Feb 16 15:58:57 2021 -0800
IMPALA-10389: (supplement) Add impala-profile-tool to build by default
This just adds the impala-profile-tool to the list of regular make
targets that will be built by default. It helps streamline automated
build jobs that also verify docker builds.
Change-Id: I58d8392d19b04aceb06874056cb2ffc5d22ca673
Reviewed-on: http://gerrit.cloudera.org:8080/17072
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Bikramjeet Vig <bi...@cloudera.com>
---
CMakeLists.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 023cb9c..77401b6 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -456,7 +456,7 @@ add_custom_target(notests_independent_targets DEPENDS
java cscope tarballs impala_python
)
add_custom_target(notests_regular_targets DEPENDS
- impalad statestored catalogd admissiond fesupport loggingsupport ImpalaUdf udasample udfsample
+ impalad statestored catalogd admissiond fesupport loggingsupport ImpalaUdf udasample udfsample impala-profile-tool
)
add_custom_target(notests_all_targets DEPENDS