You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2023/03/22 03:10:35 UTC
[impala] branch master updated (86f9e881a -> 8edfab000)
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from 86f9e881a IMPALA-11997: Switch encodestring to encodebytes in Python 3
new e8a879c30 IMPALA-12003: Make getPartialCatalogObject logging silent
new 04f7e8c0f IMPALA-8054: Fix BetweenToCompound rewrite rule binary predicate creation
new 8edfab000 IMPALA-12010: [DOCS] Document the support for non-unique primary key
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/topics/impala_kudu.xml | 5 +-
.../java/org/apache/impala/common/JniUtil.java | 18 ++++--
.../impala/rewrite/BetweenToCompoundRule.java | 34 +++++++----
.../java/org/apache/impala/service/JniCatalog.java | 2 +-
.../functional-query/queries/QueryTest/exprs.test | 71 ++++++++++++++++++++++
5 files changed, 109 insertions(+), 21 deletions(-)
[impala] 03/03: IMPALA-12010: [DOCS] Document the support for non-unique primary key
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 8edfab000e9ecd89cb3bed2b24e640489e88b342
Author: Shajini Thayasingh <st...@cloudera.com>
AuthorDate: Mon Mar 20 12:19:50 2023 -0700
IMPALA-12010: [DOCS] Document the support for non-unique primary key
Incorporated new comments.
Made changes to address new comments posted after the patch was merged.
Change-Id: Ifaea9645cf771b76fda3083d73b14524f53eb186
Reviewed-on: http://gerrit.cloudera.org:8080/19636
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Abhishek Chennaka <ac...@cloudera.com>
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
---
docs/topics/impala_kudu.xml | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/docs/topics/impala_kudu.xml b/docs/topics/impala_kudu.xml
index 0bc781b3b..8f9fbf194 100644
--- a/docs/topics/impala_kudu.xml
+++ b/docs/topics/impala_kudu.xml
@@ -210,7 +210,10 @@ under the License.
table. The data engine handles this by appending a system generated auto-incrementing
column to the non-unique primary key columns. This is done to guarantee the uniqueness of
the primary key. This auto-incrementing column is named as 'auto_incrementing_id' with
- bigint type. The assignment to it during insertion is automatic.</p>
+ bigint type and this column is only system generated and cannot be explicitly created by
+ the user. This auto_incrementing_id column is unique across a partition/tablet i.e. every
+ partition/tablet would have this column starting from one and incrementing monotonically.
+ The assignment to this column during insertion is automatic.</p>
</conbody>
</concept>
<concept id="create">
[impala] 02/03: IMPALA-8054: Fix BetweenToCompound rewrite rule binary predicate creation
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 04f7e8c0f9158fdfc1ab9cd85d04f3a72e9a4aea
Author: Peter Rozsa <pr...@cloudera.com>
AuthorDate: Tue Mar 14 10:20:54 2023 +0100
IMPALA-8054: Fix BetweenToCompound rewrite rule binary predicate creation
This patch fixes the BetweenToCompound rewrite rule's binary predicate
creation. When the BETWEEN expression gets separated, the first
operand's reference is assigned to both upper and lower binary
predicates, but in the case of different typed second and third
operands, the first operand must be cloned to make type casting unique
for both binary predicates.
Testing:
- test cases added to exprs.test
Change-Id: Iaff4199f6d0875c38fa7e91033385c9290c57bf5
Reviewed-on: http://gerrit.cloudera.org:8080/19618
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../impala/rewrite/BetweenToCompoundRule.java | 34 +++++++----
.../functional-query/queries/QueryTest/exprs.test | 71 ++++++++++++++++++++++
2 files changed, 92 insertions(+), 13 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java b/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
index bea4525ca..bf2a39ce7 100644
--- a/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
+++ b/fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
@@ -34,29 +34,37 @@ import org.apache.impala.analysis.Predicate;
* A NOT BETWEEN X AND Y ==> A < X OR A > Y
*/
public class BetweenToCompoundRule implements ExprRewriteRule {
- public static ExprRewriteRule INSTANCE = new BetweenToCompoundRule();
+ public static final ExprRewriteRule INSTANCE = new BetweenToCompoundRule();
@Override
public Expr apply(Expr expr, Analyzer analyzer) {
if (!(expr instanceof BetweenPredicate)) return expr;
BetweenPredicate bp = (BetweenPredicate) expr;
- Expr result = null;
+ Expr value = bp.getChild(0);
+ // Using a cloned value for the 'upper' binary predicate to avoid being affected by
+ // changes in the 'lower' binary predicate.
+ Expr clonedValue = value.clone();
+ Expr lowerBound = bp.getChild(1);
+ Expr upperBound = bp.getChild(2);
+
+ BinaryPredicate.Operator lowerOperator;
+ BinaryPredicate.Operator upperOperator;
+ CompoundPredicate.Operator compoundOperator;
+
if (bp.isNotBetween()) {
// Rewrite into disjunction.
- Predicate lower = new BinaryPredicate(BinaryPredicate.Operator.LT,
- bp.getChild(0), bp.getChild(1));
- Predicate upper = new BinaryPredicate(BinaryPredicate.Operator.GT,
- bp.getChild(0), bp.getChild(2));
- result = new CompoundPredicate(CompoundPredicate.Operator.OR, lower, upper);
+ lowerOperator = BinaryPredicate.Operator.LT;
+ upperOperator = BinaryPredicate.Operator.GT;
+ compoundOperator = CompoundPredicate.Operator.OR;
} else {
// Rewrite into conjunction.
- Predicate lower = new BinaryPredicate(BinaryPredicate.Operator.GE,
- bp.getChild(0), bp.getChild(1));
- Predicate upper = new BinaryPredicate(BinaryPredicate.Operator.LE,
- bp.getChild(0), bp.getChild(2));
- result = new CompoundPredicate(CompoundPredicate.Operator.AND, lower, upper);
+ lowerOperator = BinaryPredicate.Operator.GE;
+ upperOperator = BinaryPredicate.Operator.LE;
+ compoundOperator = CompoundPredicate.Operator.AND;
}
- return result;
+ Predicate lower = new BinaryPredicate(lowerOperator, value, lowerBound);
+ Predicate upper = new BinaryPredicate(upperOperator, clonedValue, upperBound);
+ return new CompoundPredicate(compoundOperator, lower, upper);
}
private BetweenToCompoundRule() {}
diff --git a/testdata/workloads/functional-query/queries/QueryTest/exprs.test b/testdata/workloads/functional-query/queries/QueryTest/exprs.test
index cff4f15cf..30e28b39d 100644
--- a/testdata/workloads/functional-query/queries/QueryTest/exprs.test
+++ b/testdata/workloads/functional-query/queries/QueryTest/exprs.test
@@ -1399,6 +1399,77 @@ and cast('2010-01-01 01:40:00' as timestamp)
bigint
====
---- QUERY
+# Regression tests for IMPALA-8054
+select count(*) from functional.alltypes where 10 between float_col and bigint_col;
+---- RESULTS
+6570
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where !(10 between null and bigint_col);
+---- RESULTS
+730
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where !(5 between float_col and null);
+---- RESULTS
+3650
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where 10 between null and null;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where null between null and null;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where 10 not between float_col and bigint_col;
+---- RESULTS
+730
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where 10 not between null and bigint_col;
+---- RESULTS
+730
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where 5 not between float_col and null;
+---- RESULTS
+3650
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where 10 not between null and null;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
+select count(*) from functional.alltypes where null not between null and null;
+---- RESULTS
+0
+---- TYPES
+bigint
+====
+---- QUERY
# Test pid() function, this should only return one pid. If pid() were not implemented
# correctly via the global state variable, this could return multiple pids.
select pid() p from functional.alltypes
[impala] 01/03: IMPALA-12003: Make getPartialCatalogObject logging silent
Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit e8a879c30db31d64108b0b6d97739558fcaf62f0
Author: prozsa <pr...@cloudera.com>
AuthorDate: Thu Mar 16 11:41:57 2023 +0100
IMPALA-12003: Make getPartialCatalogObject logging silent
This patch changes the getPartialCatalogObject catalog operation
to use the silent execution path. The silent execution path is now
logging in trace level instead of suppressing the logging at all, making
it possible to track start-stop events for silent operations.
Change-Id: I45e331eb299c057c2fdc6a3c2ed7e4ed6d1f40b7
Reviewed-on: http://gerrit.cloudera.org:8080/19627
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
fe/src/main/java/org/apache/impala/common/JniUtil.java | 18 ++++++++++++------
.../java/org/apache/impala/service/JniCatalog.java | 2 +-
2 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/common/JniUtil.java b/fe/src/main/java/org/apache/impala/common/JniUtil.java
index d447156c3..6002e88c6 100644
--- a/fe/src/main/java/org/apache/impala/common/JniUtil.java
+++ b/fe/src/main/java/org/apache/impala/common/JniUtil.java
@@ -159,16 +159,22 @@ public class JniUtil {
}
public void logStart() {
- if (!silentStartAndFinish) {
- LOG.info("{} request: {}", methodName, shortDescription);
+ final String startFormat = "{} request: {}";
+ if (silentStartAndFinish) {
+ LOG.trace(startFormat, methodName, shortDescription);
+ } else {
+ LOG.info(startFormat, methodName, shortDescription);
}
}
public void logFinish() {
- if (!silentStartAndFinish) {
- long duration = getDurationFromStart();
- LOG.info("Finished {} request: {}. Time spent: {}", methodName, shortDescription,
- PrintUtils.printTimeMs(duration));
+ final String finishFormat = "Finished {} request: {}. Time spent: {}";
+ long duration = getDurationFromStart();
+ String durationString = PrintUtils.printTimeMs(duration);
+ if (silentStartAndFinish) {
+ LOG.trace(finishFormat, methodName, shortDescription, durationString);
+ } else {
+ LOG.info(finishFormat, methodName, shortDescription, durationString);
}
}
diff --git a/fe/src/main/java/org/apache/impala/service/JniCatalog.java b/fe/src/main/java/org/apache/impala/service/JniCatalog.java
index 8d3488e5c..d5afb46ab 100644
--- a/fe/src/main/java/org/apache/impala/service/JniCatalog.java
+++ b/fe/src/main/java/org/apache/impala/service/JniCatalog.java
@@ -393,7 +393,7 @@ public class JniCatalog {
String shortDesc = "Getting partial catalog object of "
+ Catalog.toCatalogObjectKey(req.getObject_desc());
- return execAndSerialize("getPartialCatalogObject", shortDesc,
+ return execAndSerializeSilentStartAndFinish("getPartialCatalogObject", shortDesc,
() -> catalog_.getPartialCatalogObject(req));
}