You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by bo...@apache.org on 2019/01/02 11:23:09 UTC
[1/4] impala git commit: IMPALA-7992: Reduce iterations for
test_decimal_fuzz.
Repository: impala
Updated Branches:
refs/heads/master a3eb5fa90 -> 27577dd65
IMPALA-7992: Reduce iterations for test_decimal_fuzz.
We're seeing build failures in exhaustive builds on centos6,
because the timeout (defined as 2 hours in tests/conftest.py)
is hit. Though it's nice to run more iterations, we get
that over time anyway.
Change-Id: If84fcfbec0fe69a46d6e20282756794e3d57e332
Reviewed-on: http://gerrit.cloudera.org:8080/12126
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/378f8c27
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/378f8c27
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/378f8c27
Branch: refs/heads/master
Commit: 378f8c279587608a00a8eb4ad06b0e4f9e547c31
Parents: a3eb5fa
Author: Philip Zeyliger <ph...@cloudera.com>
Authored: Fri Dec 21 13:41:59 2018 -0800
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Fri Dec 28 01:38:16 2018 +0000
----------------------------------------------------------------------
tests/query_test/test_decimal_fuzz.py | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/378f8c27/tests/query_test/test_decimal_fuzz.py
----------------------------------------------------------------------
diff --git a/tests/query_test/test_decimal_fuzz.py b/tests/query_test/test_decimal_fuzz.py
index 1433ec3..b0fb572 100644
--- a/tests/query_test/test_decimal_fuzz.py
+++ b/tests/query_test/test_decimal_fuzz.py
@@ -41,10 +41,7 @@ class TestDecimalFuzz(ImpalaTestSuite):
def add_test_dimensions(cls):
cls.ImpalaTestMatrix = ImpalaTestMatrix()
cls.ImpalaTestMatrix.add_dimension(create_single_exec_option_dimension())
- if cls.exploration_strategy() == 'exhaustive':
- cls.iterations = 50000
- else:
- cls.iterations = 10000
+ cls.iterations = 10000
def weighted_choice(self, options):
total_weight = sum(options.itervalues())
[4/4] impala git commit: IMPALA-7902: NumericLiteral fixes,
refactoring
Posted by bo...@apache.org.
IMPALA-7902: NumericLiteral fixes, refactoring
The work to clean up the rewriter logic must start with a stable AST,
which must start with sprucing up some issues with the leaf nodes. This
CR tackles the NumericLiteral used to hold numbers.
IMPALA-7896: Literals should not need explicit analyze step
Partial fix: removes the need to analyze a numeric literal: analyze() is
a no-op. This eliminates the need to do a "fake" analysis with a null
analyzer: numeric literals are now created analyzed. This is useful
because the catalog module creates numeric literals outside of a query
(and outside of an analyzer.)
A literal is immutable except for type. Modified the constructor to set
the type and cost, then mark the node as analyzed. A later call to
analyze() has nothing to do.
Code that created and dummy-analyzed numeric literals changed to use
static create() methods resulting in simpler literal creation, and
eliminates the special "analyzer == null" checks in analyze().
IMPALA-7886: NumericLiteral constructor fails to round values to
Decimal type
IMPALA-7887: NumericLiteral fails to detect numeric overflow
IMPALA-7888: Incorrect NumericLiteral overflow checks for FLOAT,
DOUBLE
IMPALA-7891: Analyzer does not detect numeric overflow in CAST
IMPALA-7894: Parser does not catch double overflow
These are all caused by the somewhat cluttered state of the numeric
range check code after years of incremental changes. This patch
centralizes all checks into a series of constants and methods for
uniformity. All values are set in the constructor which now checks
that the value is legal for the type. Cast operations verify that the
cast is valid. Multiple semi-parallel versions of the same logic is
replaced by calls to a single implementation.
The numeric checks now follow the SQL standard which says that
implementations should fail if a cast would trucate the most significant
digits, but round when truncating the least significant.
IMPALA-7865: Repeated type widening of arithmetic expressions
Partial fix. Replaces the "is explicit cast" flag in the numeric literal
with the explicit type. This allows reseting an implicit type back to
the explciit type if an arithmetic expression is analyzed multiple
times. A later patch will feed this type information into the type
inference mechanism to complete the fix.
Finally, adds a set of new exceptions that begin to unify error
reporting. These handle casts (SqlCastException), value validation
(InvalidValueException) and unsupported features
(UnsupportedFeatureException.) These all derive from AnalysisException
for backward compatibility. Tests use the new exceptions to check for
expected errors rather than parsing text strings (which tend to
change.)
Testing:
* Added unit tests just for numeric literals. Refactored code to
simplify the tests.
* Added a test case for the obscure case in Decimal V1 of an implicit
cast overflow.
* The depth-check tests needed one extra level of nesting to trigger
failure.
* Ran all FE tests.
Change-Id: I484600747b2871d3a6fe9153751973af9a8534f2
Reviewed-on: http://gerrit.cloudera.org:8080/12001
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/27577dd6
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/27577dd6
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/27577dd6
Branch: refs/heads/master
Commit: 27577dd652554dda5a03016e2d1e3ab66fe6b1f5
Parents: 6f6d1f6
Author: Paul Rogers <pr...@cloudera.com>
Authored: Tue Nov 27 21:29:24 2018 -0800
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Fri Dec 28 23:25:06 2018 +0000
----------------------------------------------------------------------
be/src/exprs/expr-test.cc | 6 +-
.../apache/impala/analysis/AnalyticExpr.java | 19 +-
.../java/org/apache/impala/analysis/Expr.java | 10 +-
.../org/apache/impala/analysis/LiteralExpr.java | 24 +-
.../apache/impala/analysis/NumericLiteral.java | 480 ++++++++++----
.../java/org/apache/impala/analysis/Parser.java | 26 +-
.../org/apache/impala/analysis/QueryStmt.java | 8 +-
.../apache/impala/catalog/ScalarFunction.java | 3 +
.../org/apache/impala/catalog/ScalarType.java | 8 +
.../java/org/apache/impala/catalog/Type.java | 8 +-
.../java/org/apache/impala/catalog/View.java | 11 +-
.../impala/common/InvalidValueException.java | 35 +
.../apache/impala/common/SqlCastException.java | 40 ++
.../common/UnsupportedFeatureException.java | 31 +
.../impala/analysis/AnalyzeExprsTest.java | 7 +-
.../impala/analysis/ExprRewriteRulesTest.java | 53 ++
.../org/apache/impala/analysis/ExprTest.java | 90 ---
.../apache/impala/analysis/LiteralExprTest.java | 176 +++++
.../impala/analysis/NumericLiteralTest.java | 655 +++++++++++++++++++
.../org/apache/impala/analysis/ParserTest.java | 7 +-
.../catalog/CatalogObjectToFromThriftTest.java | 12 +-
.../impala/catalog/HdfsPartitionTest.java | 15 +-
.../apache/impala/common/FrontendTestBase.java | 6 +-
23 files changed, 1434 insertions(+), 296 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/be/src/exprs/expr-test.cc
----------------------------------------------------------------------
diff --git a/be/src/exprs/expr-test.cc b/be/src/exprs/expr-test.cc
index 2bc3145..0e0125c 100644
--- a/be/src/exprs/expr-test.cc
+++ b/be/src/exprs/expr-test.cc
@@ -9054,7 +9054,7 @@ int main(int argc, char** argv) {
executor_->ClearExecOptions();
executor_->PushExecOption("ENABLE_EXPR_REWRITES=0");
executor_->PushExecOption("DISABLE_CODEGEN=1");
- cout << "Running without codegen" << endl;
+ cout << "Running without codegen and without expr rewrites" << endl;
ret = RUN_ALL_TESTS();
if (ret != 0) return ret;
@@ -9065,7 +9065,7 @@ int main(int argc, char** argv) {
executor_->PushExecOption("DISABLE_CODEGEN=0");
executor_->PushExecOption("EXEC_SINGLE_NODE_ROWS_THRESHOLD=0");
executor_->PushExecOption("DISABLE_CODEGEN_ROWS_THRESHOLD=0");
- cout << endl << "Running with codegen" << endl;
+ cout << endl << "Running with codegen and without expr rewrites" << endl;
ret = RUN_ALL_TESTS();
if (ret != 0) return ret;
@@ -9075,6 +9075,6 @@ int main(int argc, char** argv) {
executor_->ClearExecOptions();
executor_->PushExecOption("ENABLE_EXPR_REWRITES=1");
executor_->PushExecOption("DISABLE_CODEGEN=1");
- cout << endl << "Running without codegen and expr rewrites" << endl;
+ cout << endl << "Running without codegen and with expr rewrites" << endl;
return RUN_ALL_TESTS();
}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java b/fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
index c0872d6..7eeb86c 100644
--- a/fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
+++ b/fe/src/main/java/org/apache/impala/analysis/AnalyticExpr.java
@@ -17,7 +17,6 @@
package org.apache.impala.analysis;
-import java.math.BigDecimal;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.List;
@@ -263,8 +262,8 @@ public class AnalyticExpr extends Expr {
Preconditions.checkState(
AnalyticExpr.isPercentRankFn(analyticExpr.getFnCall().getFn()));
- NumericLiteral zero = new NumericLiteral(BigInteger.valueOf(0), ScalarType.BIGINT);
- NumericLiteral one = new NumericLiteral(BigInteger.valueOf(1), ScalarType.BIGINT);
+ NumericLiteral zero = NumericLiteral.create(0, ScalarType.BIGINT);
+ NumericLiteral one = NumericLiteral.create(1, ScalarType.BIGINT);
AnalyticExpr countExpr = create("count", analyticExpr, false, false);
AnalyticExpr rankExpr = create("rank", analyticExpr, true, false);
@@ -297,7 +296,7 @@ public class AnalyticExpr extends Expr {
AnalyticExpr.isCumeDistFn(analyticExpr.getFnCall().getFn()));
AnalyticExpr rankExpr = create("rank", analyticExpr, true, true);
AnalyticExpr countExpr = create("count", analyticExpr, false, false);
- NumericLiteral one = new NumericLiteral(BigInteger.valueOf(1), ScalarType.BIGINT);
+ NumericLiteral one = NumericLiteral.create(1, ScalarType.BIGINT);
ArithmeticExpr arithmeticRewrite =
new ArithmeticExpr(ArithmeticExpr.Operator.DIVIDE,
new ArithmeticExpr(ArithmeticExpr.Operator.ADD,
@@ -329,7 +328,7 @@ public class AnalyticExpr extends Expr {
ifParams.add(bucketExpr);
ifParams.add(countExpr);
- NumericLiteral one = new NumericLiteral(BigInteger.valueOf(1), ScalarType.BIGINT);
+ NumericLiteral one = NumericLiteral.create(1, ScalarType.BIGINT);
ArithmeticExpr minMultiplyRowMinusOne =
new ArithmeticExpr(ArithmeticExpr.Operator.MULTIPLY,
new ArithmeticExpr(ArithmeticExpr.Operator.SUBTRACT, rowNumExpr, one),
@@ -616,7 +615,7 @@ public class AnalyticExpr extends Expr {
newExprParams = Lists.newArrayListWithExpectedSize(3);
newExprParams.addAll(getFnCall().getChildren());
// Default offset is 1.
- newExprParams.add(new NumericLiteral(BigDecimal.valueOf(1)));
+ newExprParams.add(NumericLiteral.create(1));
// Default default value is NULL.
newExprParams.add(new NullLiteral());
} else if (getFnCall().getChildren().size() == 2) {
@@ -666,13 +665,13 @@ public class AnalyticExpr extends Expr {
if (window_.getRightBoundary().getType() == BoundaryType.PRECEDING) {
// The number of rows preceding for the end bound determines the number of
// rows at the beginning of each partition that should have a NULL value.
- paramExprs.add(new NumericLiteral(window_.getRightBoundary().getOffsetValue(),
- Type.BIGINT));
+ paramExprs.add(NumericLiteral.create(
+ window_.getRightBoundary().getOffsetValue(), Type.BIGINT));
} else {
// -1 indicates that no NULL values are inserted even though we set the end
// bound to the start bound (which is PRECEDING) below; this is different from
// the default behavior of windows with an end bound PRECEDING.
- paramExprs.add(new NumericLiteral(BigInteger.valueOf(-1), Type.BIGINT));
+ paramExprs.add(NumericLiteral.create(-1, Type.BIGINT));
}
window_ = new AnalyticWindow(window_.getType(),
@@ -770,7 +769,7 @@ public class AnalyticExpr extends Expr {
Preconditions.checkState(isOffsetFn(getFnCall().getFn()));
if (offsetFnCall.getChild(1) != null) return offsetFnCall.getChild(1);
// The default offset is 1.
- return new NumericLiteral(BigDecimal.valueOf(1));
+ return NumericLiteral.create(1);
}
/**
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/analysis/Expr.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/Expr.java b/fe/src/main/java/org/apache/impala/analysis/Expr.java
index 790bcde..0bf1f8d 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Expr.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Expr.java
@@ -35,6 +35,7 @@ import org.apache.impala.catalog.ScalarType;
import org.apache.impala.catalog.Type;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.InternalException;
+import org.apache.impala.common.SqlCastException;
import org.apache.impala.common.TreeNode;
import org.apache.impala.rewrite.ExprRewriter;
import org.apache.impala.service.FeSupport;
@@ -523,7 +524,7 @@ abstract public class Expr extends TreeNode<Expr> implements ParseNode, Cloneabl
Type resolvedWildcardType = getResolvedWildCardType(strictDecimal);
if (resolvedWildcardType != null) {
if (resolvedWildcardType.isNull()) {
- throw new AnalysisException(String.format(
+ throw new SqlCastException(String.format(
"Cannot resolve DECIMAL precision and scale from NULL type in %s function.",
fn_.getFunctionName().getFunction()));
}
@@ -534,7 +535,7 @@ abstract public class Expr extends TreeNode<Expr> implements ParseNode, Cloneabl
Type childType = children_.get(j).type_;
argTypes.append(childType.toSql());
}
- throw new AnalysisException(String.format(
+ throw new SqlCastException(String.format(
"Cannot resolve DECIMAL types of the %s(%s) function arguments. You need " +
"to wrap the arguments in a CAST.", fn_.getFunctionName().getFunction(),
argTypes.toString()));
@@ -1343,8 +1344,10 @@ abstract public class Expr extends TreeNode<Expr> implements ParseNode, Cloneabl
if (!targetType.isDecimal()) {
// requested cast must be to assignment-compatible type
// (which implies no loss of precision)
- Preconditions.checkArgument(targetType.equals(type),
+ if (!targetType.equals(type)) {
+ throw new SqlCastException(
"targetType=" + targetType + " type=" + type);
+ }
}
return uncheckedCastTo(targetType);
}
@@ -1381,7 +1384,6 @@ abstract public class Expr extends TreeNode<Expr> implements ParseNode, Cloneabl
setChild(childIndex, newChild);
}
-
/**
* Convert child to to targetType, possibly by inserting an implicit cast, or by
* returning an altogether new expression, or by returning 'this' with a modified
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java b/fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java
index cde3db3..9d62678 100644
--- a/fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java
+++ b/fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java
@@ -26,6 +26,8 @@ import org.apache.impala.catalog.Type;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.InternalException;
import org.apache.impala.common.NotImplementedException;
+import org.apache.impala.common.SqlCastException;
+import org.apache.impala.common.UnsupportedFeatureException;
import org.apache.impala.service.FeSupport;
import org.apache.impala.thrift.TColumnValue;
import org.apache.impala.thrift.TExprNode;
@@ -43,8 +45,10 @@ public abstract class LiteralExpr extends Expr implements Comparable<LiteralExpr
private final static Logger LOG = LoggerFactory.getLogger(LiteralExpr.class);
public LiteralExpr() {
+ // Literals start analyzed: there is nothing more to check.
evalCost_ = LITERAL_COST;
numDistinctValues_ = 1;
+ // Subclass is responsible for setting the type
}
/**
@@ -59,7 +63,9 @@ public abstract class LiteralExpr extends Expr implements Comparable<LiteralExpr
* LiteralExpr subclass, e.g. TIMESTAMP.
*/
public static LiteralExpr create(String value, Type type) throws AnalysisException {
- Preconditions.checkArgument(type.isValid());
+ if (!type.isValid()) {
+ throw new UnsupportedFeatureException("Invalid literal type: " + type.toSql());
+ }
LiteralExpr e = null;
switch (type.getPrimitiveType()) {
case NULL_TYPE:
@@ -86,9 +92,9 @@ public abstract class LiteralExpr extends Expr implements Comparable<LiteralExpr
case DATETIME:
case TIMESTAMP:
// TODO: we support TIMESTAMP but no way to specify it in SQL.
- return null;
+ throw new UnsupportedFeatureException("Literal unsupported: " + type.toSql());
default:
- Preconditions.checkState(false,
+ throw new UnsupportedFeatureException(
String.format("Literals of type '%s' not supported.", type.toSql()));
}
e.analyze(null);
@@ -219,9 +225,15 @@ public abstract class LiteralExpr extends Expr implements Comparable<LiteralExpr
case FLOAT:
case DOUBLE:
if (val.isSetDouble_val()) {
+ // Create using double directly, at extreme ranges the BigDecimal
+ // value overflows a double due to conversion issues.
// A NumericLiteral cannot represent NaN, infinity or negative zero.
- if (!NumericLiteral.isValidLiteral(val.double_val)) return null;
- result = new NumericLiteral(new BigDecimal(val.double_val), constExpr.getType());
+ // SqlCastException thrown for these cases.
+ try {
+ result = new NumericLiteral(val.double_val, constExpr.getType());
+ } catch (SqlCastException e) {
+ return null;
+ }
}
break;
case DECIMAL:
@@ -269,7 +281,7 @@ public abstract class LiteralExpr extends Expr implements Comparable<LiteralExpr
if (result == null) result = new NullLiteral();
result.analyzeNoThrow(null);
- return (LiteralExpr)result;
+ return result;
}
// Order NullLiterals based on the SQL ORDER BY default behavior: NULLS LAST.
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java b/fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java
index 52f748f..2469174 100644
--- a/fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java
+++ b/fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java
@@ -23,7 +23,7 @@ import java.math.BigInteger;
import org.apache.impala.catalog.ScalarType;
import org.apache.impala.catalog.Type;
import org.apache.impala.common.AnalysisException;
-import org.apache.impala.common.NotImplementedException;
+import org.apache.impala.common.SqlCastException;
import org.apache.impala.thrift.TDecimalLiteral;
import org.apache.impala.thrift.TExprNode;
import org.apache.impala.thrift.TExprNodeType;
@@ -34,10 +34,61 @@ import com.google.common.base.Objects;
import com.google.common.base.Preconditions;
/**
- * Literal for all numeric values, including integer, floating-point and decimal types.
- * Analysis of this expr determines the smallest type that can hold this value.
+ * Literal for all numeric values, including integer, floating-point and decimal
+ * types. The constructor determines the "natural type" of this literal:
+ * smallest type that can hold this value.
+ *
+ * A numeric literal has an "explicit" type which starts as the natural type.
+ * The explicit type can be widened via a user-supplied cast: CAST(1 AS INT).
+ * Constant folding uses the BE to convert such an expression into a new numeric
+ * literal with the explicit type set to the requested type.
+ *
+ * The literal also has an "implicit type" which starts the same as the explicit
+ * type. The implicit type changes as a result of casts the analyzer uses to
+ * adjust input types for functions and arithmetic operations. Here, the code
+ * attempts to "fold" the cast into the literal directly. TODO: Revisit this
+ * implicit folding route; now that constant folding is available, perhaps both
+ * cases should work the same: by using the constant folding rule and allowing
+ * the BE to do the work. This saves trying to keep the FE and BE conversion
+ * code in sync.
+ *
+ * The analyzer makes two analysis passes over the code: analyze, rewrite,
+ * analyze again. Implicit types can lead to repeated type widening of
+ * expressions. In the first analysis round:
+ *
+ * 1:TINYINT + 2:TINYINT --> SMALLINT result
+ *
+ * Cast inputs to the output type:
+ *
+ * CAST(1:TINYINT AS SMALLINT) + CAST(1:TINYINT AS SMALLINT)
+ *
+ * In the second round, we must reset the implicit type back to the explicit
+ * type, else the type propagation works with the previous implicit type
+ * producing:
+ *
+ * 1:SMALLINT + 1:SMALLINT -> INT result
+ *
+ * With new implicit casts added:
+ *
+ * CAST(1:SMALLINT AS INT) + CAST(1:SMALLINT AS INT)
+ *
+ * TODO: The above problem would disappear if the analyzer is modified to make
+ * only one analysis/type propagation pass over each expression.
*/
public class NumericLiteral extends LiteralExpr {
+ public static final BigDecimal MIN_TINYINT = BigDecimal.valueOf(Byte.MIN_VALUE);
+ public static final BigDecimal MAX_TINYINT = BigDecimal.valueOf(Byte.MAX_VALUE);
+ public static final BigDecimal MIN_SMALLINT = BigDecimal.valueOf(Short.MIN_VALUE);
+ public static final BigDecimal MAX_SMALLINT = BigDecimal.valueOf(Short.MAX_VALUE);
+ public static final BigDecimal MIN_INT = BigDecimal.valueOf(Integer.MIN_VALUE);
+ public static final BigDecimal MAX_INT = BigDecimal.valueOf(Integer.MAX_VALUE);
+ public static final BigDecimal MIN_BIGINT = BigDecimal.valueOf(Long.MIN_VALUE);
+ public static final BigDecimal MAX_BIGINT = BigDecimal.valueOf(Long.MAX_VALUE);
+ public static final BigDecimal MAX_FLOAT = BigDecimal.valueOf(Float.MAX_VALUE);
+ public static final BigDecimal MIN_FLOAT = MAX_FLOAT.negate();
+ public static final BigDecimal MAX_DOUBLE = BigDecimal.valueOf(Double.MAX_VALUE);
+ public static final BigDecimal MIN_DOUBLE = MAX_DOUBLE.negate();
+
// Use the java BigDecimal (arbitrary scale/precision) to represent the value.
// This object has notions of precision and scale but they do *not* match what
// we need. BigDecimal's precision is similar to significant figures and scale
@@ -51,35 +102,31 @@ public class NumericLiteral extends LiteralExpr {
// negative zero.
private BigDecimal value_;
- // If true, this literal has been explicitly cast to a type and should not
- // be analyzed (which infers the type from value_).
- private boolean explicitlyCast_;
+ // The explicit type of the literal, which must be wider than the "natural"
+ // type. Set via the constructor or as the result of pushing an explicit
+ // (user-provided) CAST into this literal.
+ private Type explicitType_;
- public NumericLiteral(BigDecimal value) {
+ public NumericLiteral(BigDecimal value) throws SqlCastException {
value_ = value;
+ type_ = inferType(value);
+ explicitType_ = type_;
+ // No further analysis needed for a (numeric) literal.
+ analysisDone();
}
- public NumericLiteral(String value, Type t) throws AnalysisException {
- BigDecimal val = null;
- try {
- val = new BigDecimal(value);
- } catch (NumberFormatException e) {
- throw new AnalysisException("invalid numeric literal: " + value, e);
- }
- value_ = val;
- this.analyze(null);
- if (type_.isDecimal() && t.isDecimal()) {
- // Verify that the input decimal value is consistent with the specified
- // column type.
- ScalarType scalarType = (ScalarType) t;
- if (!scalarType.isSupertypeOf((ScalarType) type_)) {
- StringBuilder errMsg = new StringBuilder();
- errMsg.append("invalid ").append(t);
- errMsg.append(" value: " + value);
- throw new AnalysisException(errMsg.toString());
- }
- }
- if (t.isFloatingPointType()) explicitlyCastToFloat(t);
+ /**
+ * Constructor, primarily for testing, that takes a string in BigDecimal format.
+ * Note that this format is slightly different than the Impala SQL format. (No
+ * double minus signs, etc.)
+ *
+ * @param value the numeric value as a string
+ * @param type the desired target type
+ * @throws SqlCastException if the value given is not valid for the type
+ * provided
+ */
+ public NumericLiteral(String value, Type type) throws SqlCastException {
+ this(new BigDecimal(value), type);
}
/**
@@ -87,17 +134,37 @@ public class NumericLiteral extends LiteralExpr {
* and the NumericLiteral is created as analyzed with that type. The specified
* type is preserved across substitutions and re-analysis.
*/
- public NumericLiteral(BigInteger value, Type type) {
- value_ = new BigDecimal(value);
+ public NumericLiteral(BigInteger value, Type type) throws SqlCastException {
+ this(new BigDecimal(value), type);
+ }
+
+ public NumericLiteral(BigDecimal value, Type type) throws SqlCastException {
+ value_ = convertValue(value, type);
type_ = type;
- explicitlyCast_ = true;
+ explicitType_ = type_;
analysisDone();
}
- public NumericLiteral(BigDecimal value, Type type) {
- value_ = value;
+ /**
+ * Constructor for double values. Converting to BigDecimal and then
+ * validating the value leads to overflow at the extreme ranges.
+ * That is, conversion to a BigDecimal and back is not stable.
+ * @throws SqlCastException
+ */
+ public NumericLiteral(double value, Type type) throws SqlCastException {
+ Preconditions.checkArgument(type == Type.DOUBLE || type == Type.FLOAT);
+ // Reject INF, NaN
+ if (!isImpalaDouble(value)) {
+ throw new SqlCastException("Value " + Double.toString(value) +
+ " cannot be cast to " + type.toSql());
+ }
+ value_ = new BigDecimal(value);
+ // Ensure value is in range of a FLOAT
+ if (type == Type.FLOAT && !fitsInFloat(value_)) {
+ throw new SqlCastException(value_, type);
+ }
type_ = type;
- explicitlyCast_ = true;
+ explicitType_ = type_;
analysisDone();
}
@@ -107,23 +174,33 @@ public class NumericLiteral extends LiteralExpr {
protected NumericLiteral(NumericLiteral other) {
super(other);
value_ = other.value_;
- explicitlyCast_ = other.explicitlyCast_;
+ explicitType_ = other.explicitType_;
}
- public static NumericLiteral create(int value) {
- return new NumericLiteral(new BigDecimal(value));
+ public static NumericLiteral create(BigDecimal value) {
+ try {
+ return new NumericLiteral(value);
+ } catch (SqlCastException e) {
+ // Should never occur for int values
+ throw new IllegalStateException(e);
+ }
}
- /**
- * Returns true if 'v' can be represented by a NumericLiteral, false otherwise.
- * Special float values like NaN, infinity, and negative zero cannot be represented
- * by a NumericLiteral.
- */
- public static boolean isValidLiteral(double v) {
- if (Double.isNaN(v) || Double.isInfinite(v)) return false;
- // Check for negative zero.
- if (v == 0 && 1.0 / v == Double.NEGATIVE_INFINITY) return false;
- return true;
+ public static NumericLiteral create(BigDecimal value, Type type) {
+ try {
+ return new NumericLiteral(value, type);
+ } catch (AnalysisException e) {
+ // Should never occur for int values
+ throw new IllegalStateException(e);
+ }
+ }
+
+ public static NumericLiteral create(long value) {
+ return create(new BigDecimal(value));
+ }
+
+ public static NumericLiteral create(long value, Type type) {
+ return create(new BigDecimal(value), type);
}
@Override
@@ -135,14 +212,22 @@ public class NumericLiteral extends LiteralExpr {
}
@Override
+ public String toString() {
+ return value_.toString() + ":" + type_.toSql();
+ }
+
+ public Type getExplicitType() { return explicitType_; }
+
+ @Override
public boolean localEquals(Expr that) {
if (!super.localEquals(that)) return false;
-
- NumericLiteral tmp = (NumericLiteral) that;
- if (!tmp.value_.equals(value_)) return false;
// Analyzed Numeric literals of different types are distinct.
- if ((isAnalyzed() && tmp.isAnalyzed()) && (!getType().equals(tmp.getType()))) return false;
- return true;
+ if (!getType().equals(that.getType())) { return false; }
+
+ NumericLiteral other = (NumericLiteral) that;
+ // Must use compareTo() since equals() won't match if the
+ // values have different precisions.
+ return other.value_.compareTo(value_) == 0;
}
@Override
@@ -166,7 +251,7 @@ public class NumericLiteral extends LiteralExpr {
public double getDoubleValue() { return value_.doubleValue(); }
public long getLongValue() { return value_.longValue(); }
- public long getIntValue() { return value_.intValue(); }
+ public int getIntValue() { return value_.intValue(); }
@Override
protected void toThrift(TExprNode msg) {
@@ -196,93 +281,218 @@ public class NumericLiteral extends LiteralExpr {
public BigDecimal getValue() { return value_; }
- @Override
- protected void analyzeImpl(Analyzer analyzer) throws AnalysisException {
- if (!explicitlyCast_) {
- // Compute the precision and scale from the BigDecimal.
- type_ = TypesUtil.computeDecimalType(value_);
- if (type_ == null) {
- Double d = new Double(value_.doubleValue());
- if (d.isInfinite()) {
- throw new AnalysisException("Numeric literal '" + toSql() +
- "' exceeds maximum range of doubles.");
- } else if (d.doubleValue() == 0 && value_ != BigDecimal.ZERO) {
- throw new AnalysisException("Numeric literal '" + toSql() +
- "' underflows minimum resolution of doubles.");
- }
-
- // Literal could not be stored in any of the supported decimal precisions and
- // scale. Store it as a float/double instead.
- float fvalue;
- fvalue = value_.floatValue();
- if (fvalue == value_.doubleValue()) {
- type_ = Type.FLOAT;
- } else {
- type_ = Type.DOUBLE;
- }
- } else {
- // Check for integer types.
- Preconditions.checkState(type_.isScalarType());
- ScalarType scalarType = (ScalarType) type_;
- if (scalarType.decimalScale() == 0) {
- if (value_.compareTo(BigDecimal.valueOf(Byte.MAX_VALUE)) <= 0 &&
- value_.compareTo(BigDecimal.valueOf(Byte.MIN_VALUE)) >= 0) {
- type_ = Type.TINYINT;
- } else if (value_.compareTo(BigDecimal.valueOf(Short.MAX_VALUE)) <= 0 &&
- value_.compareTo(BigDecimal.valueOf(Short.MIN_VALUE)) >= 0) {
- type_ = Type.SMALLINT;
- } else if (value_.compareTo(BigDecimal.valueOf(Integer.MAX_VALUE)) <= 0 &&
- value_.compareTo(BigDecimal.valueOf(Integer.MIN_VALUE)) >= 0) {
- type_ = Type.INT;
- } else if (value_.compareTo(BigDecimal.valueOf(Long.MAX_VALUE)) <= 0 &&
- value_.compareTo(BigDecimal.valueOf(Long.MIN_VALUE)) >= 0) {
- type_ = Type.BIGINT;
- }
- }
+ public static boolean isBetween(BigDecimal value, BigDecimal low, BigDecimal high) {
+ return value.compareTo(low) >= 0 &&
+ value.compareTo(high) <= 0;
+ }
+
+ // Predicates to determine if the given value fits within the range of
+ // the given data type.
+ public static boolean fitsInTinyInt(BigDecimal value) {
+ return isBetween(value, MIN_TINYINT, MAX_TINYINT);
+ }
+
+ public static boolean fitsInSmallInt(BigDecimal value) {
+ return isBetween(value, MIN_SMALLINT, MAX_SMALLINT);
+ }
+
+ public static boolean fitsInInt(BigDecimal value) {
+ return isBetween(value, MIN_INT, MAX_INT);
+ }
+
+ public static boolean fitsInBigInt(BigDecimal value) {
+ return isBetween(value, MIN_BIGINT, MAX_BIGINT);
+ }
+
+ public static boolean fitsInFloat(BigDecimal value) {
+ return isBetween(value, MIN_FLOAT, MAX_FLOAT);
+ }
+
+ public static boolean fitsInDouble(BigDecimal value) {
+ return !Double.isInfinite(value.doubleValue());
+ }
+
+ /**
+ * Returns true if 'v' can be represented by a NumericLiteral, false otherwise.
+ * Special float values like NaN, infinity, and negative zero cannot be represented
+ * by a NumericLiteral.
+ */
+ public static boolean isImpalaDouble(double v) {
+ if (Double.isNaN(v) || Double.isInfinite(v)) return false;
+ // Check for negative zero.
+ if (v == 0 && 1.0 / v == Double.NEGATIVE_INFINITY) return false;
+ return true;
+ }
+
+ /**
+ * Verifies that the given BigDecimal is valid for Impala's
+ * DECIMAL range. BigDecimal can represents a larger range of
+ * values than DECIMAL, which is limited to 10e38.
+ */
+ public static boolean isImpalaDecimal(BigDecimal value) {
+ return TypesUtil.computeDecimalType(value) != null;
+ }
+
+ /**
+ * Infer the natural (smallest) type required to hold the given numeric value.
+ *
+ * @throws SqlCastException if the value is not valid for any type
+ */
+ public static ScalarType inferType(BigDecimal value) throws SqlCastException {
+ // Compute the precision and scale from the BigDecimal.
+ Type type = TypesUtil.computeDecimalType(value);
+ if (type == null) {
+ // Literal could not be stored in any of the supported decimal precisions and
+ // scale. Store it as a float/double instead.
+ double d = value.doubleValue();
+ if (!fitsInDouble(value)) {
+ throw new SqlCastException("Numeric literal '" + value.toString() +
+ "' exceeds maximum range of DOUBLE.");
+ } else if (d == 0 && value != BigDecimal.ZERO) {
+ // Note: according to the SQL standard, section 4.4, this
+ // should simply truncate the lower digits, returning 0.
+ throw new SqlCastException("Numeric literal '" + value.toString() +
+ "' underflows minimum resolution of DOUBLE.");
}
+
+ // Always return a double. FLOAT does not add much value.
+ // FLOAT can store up to 1e38, which is within the range of a DECIMAL.
+ return Type.DOUBLE;
}
+
+ // The value is a valid Decimal. Prefer an integer type.
+ Preconditions.checkState(type.isScalarType());
+ ScalarType scalarType = (ScalarType) type;
+ if (scalarType.decimalScale() != 0) return scalarType;
+ if (fitsInTinyInt(value)) return Type.TINYINT;
+ if (fitsInSmallInt(value)) return Type.SMALLINT;
+ if (fitsInInt(value)) return Type.INT;
+ if (fitsInBigInt(value)) return Type.BIGINT;
+ // Value is too large for BIGINT, keep decimal.
+ return scalarType;
+ }
+
+ @Override
+ protected void analyzeImpl(Analyzer analyzer) throws AnalysisException {
+ // On re-analysis after rewrite or other operation, revert to the
+ // explicit type. Ideally would not be needed, but seems to be
+ // required after a clone() operation.
+ type_ = explicitType_;
+ }
+
+ @Override
+ public Expr reset() {
+ // Literals are always analyzed, don't call super method as it will clear
+ // the analysis flag. Clear any implicit type.
+ type_ = explicitType_;
+ return this;
+ }
+
+ @Override
+ protected void resetAnalysisState() {
+ type_ = explicitType_;
}
/**
- * Explicitly cast this literal to 'targetType'. The targetType must be a
- * float point type.
+ * Explicitly cast to FLOAT or DOUBLE.
+ *
+ * Called from the CastExpr constructor to force a type to float.
+ * Don't do validation here, as it can cause issues.
+ *
+ * TODO: Revisit this: either leave the cast to float in place
+ * (constant folding disabled), or rely on constant folding to
+ * remove the cast. Relying on constant folding ensures all paths
+ * work the same, including in the case of overflow, etc.
*/
protected void explicitlyCastToFloat(Type targetType) {
Preconditions.checkState(targetType.isFloatingPointType());
type_ = targetType;
- explicitlyCast_ = true;
+ explicitType_ = targetType;
}
+ /**
+ * Convert to the given type, performing range checks. If the cast would result
+ * in a change of value (such as rounding for decimal), a new value is returned.
+ * If the existing value is suitable for the new type, then the original value
+ * is returned.
+ *
+ * Truncates trailing fractional digits as needed to fit the target type (as
+ * allowed by the SQL standard, section 4.4)
+ */
+ public static BigDecimal convertValue(BigDecimal value,
+ Type targetType) throws SqlCastException {
+ Preconditions.checkState(targetType.isNumericType());
+ // Don't allow overflow. Checks only extreme range for DECIMAL.
+ if (isOverflow(value, targetType)) {
+ throw new SqlCastException(value, targetType);
+ }
+
+ // If cast to an integer type, round the fractional part.
+ if (targetType.isIntegerType() && value.scale() != 0) {
+ return value.setScale(0, BigDecimal.ROUND_HALF_UP);
+ }
+
+ // If non-decimal (integer or float), use the existing value.
+ if (!targetType.isDecimal()) return value;
+
+ // Check for Decimal overflow.
+ ScalarType decimalType = (ScalarType) targetType;
+ int valLeftDigits = value.precision() - value.scale();
+ int typeLeftDigits = decimalType.decimalPrecision() - decimalType.decimalScale();
+ // Special case 0, it is reported as having 1 left digit, while 0.1
+ // has zero left digits.
+ if (typeLeftDigits < valLeftDigits && value.compareTo(BigDecimal.ZERO) != 0) {
+ throw new SqlCastException(value, targetType);
+ }
+
+ // Truncate (round) extra digits if necessary.
+ if (value.scale() > decimalType.decimalScale()) {
+ return value.setScale(decimalType.decimalScale(), BigDecimal.ROUND_HALF_UP);
+ }
+
+ // Existing value fits, use it.
+ return value;
+ }
+
+ /**
+ * Perform an implicit cast. An implicit cast is added by the analyzer
+ * and is assumed to be valid. Does not change the explicit type to
+ * avoid instability when repeating type propagation.
+ * @throws SqlCastException
+ */
@Override
- protected Expr uncheckedCastTo(Type targetType) throws AnalysisException {
+ protected Expr uncheckedCastTo(Type targetType) throws SqlCastException {
Preconditions.checkState(targetType.isNumericType());
- // Implicit casting to decimals allows truncating digits from the left of the
- // decimal point (see TypesUtil). A literal that is implicitly cast to a decimal
- // with truncation is wrapped into a CastExpr so the BE can evaluate it and report
- // a warning. This behavior is consistent with casting/overflow of non-constant
- // exprs that return decimal.
- // IMPALA-1837: Without the CastExpr wrapping, such literals can exceed the max
- // expected byte size sent to the BE in toThrift().
- if (targetType.isDecimal()) {
- ScalarType decimalType = (ScalarType) targetType;
- // analyze() ensures that value_ never exceeds the maximum scale and precision.
- Preconditions.checkState(isAnalyzed());
- // Sanity check that our implicit casting does not allow a reduced precision or
- // truncating values from the right of the decimal point.
- Preconditions.checkState(value_.precision() <= decimalType.decimalPrecision());
- Preconditions.checkState(value_.scale() <= decimalType.decimalScale());
- int valLeftDigits = value_.precision() - value_.scale();
- int typeLeftDigits = decimalType.decimalPrecision() - decimalType.decimalScale();
- if (typeLeftDigits < valLeftDigits) return new CastExpr(targetType, this);
+ if (type_ == targetType) return this;
+ try {
+ BigDecimal converted = convertValue(value_, targetType);
+ if (converted == value_) {
+ // Use existing value, cast in place.
+ type_ = targetType;
+ return this;
+ } else {
+ // Value changed, create a new literal.
+ return new NumericLiteral(converted, targetType);
+ }
+ } catch (SqlCastException e) {
+ return new CastExpr(targetType, this);
}
- type_ = targetType;
- return this;
}
@Override
- public void swapSign() throws NotImplementedException {
- // swapping sign does not change the type
+ public void swapSign() {
+ // Swapping the sign may change the type:
+ // 128 is a SMALLINT, -128 is a TINYINT
+ // Also -<max bigint> starts as DECIMAL(19.0), but
+ // will flip to BIGINT in this call.
value_ = value_.negate();
+ try {
+ type_ = inferType(value_);
+ explicitType_ = type_;
+ } catch (SqlCastException e) {
+ // Should never occur
+ throw new IllegalStateException(e);
+ }
}
@Override
@@ -312,31 +522,25 @@ public class NumericLiteral extends LiteralExpr {
/**
* Check overflow.
*/
- public static boolean isOverflow(BigDecimal value, Type type)
- throws AnalysisException {
+ public static boolean isOverflow(BigDecimal value, Type type) {
switch (type.getPrimitiveType()) {
case TINYINT:
- return (value.compareTo(BigDecimal.valueOf(Byte.MAX_VALUE)) > 0 ||
- value.compareTo(BigDecimal.valueOf(Byte.MIN_VALUE)) < 0);
+ return !fitsInTinyInt(value);
case SMALLINT:
- return (value.compareTo(BigDecimal.valueOf(Short.MAX_VALUE)) > 0 ||
- value.compareTo(BigDecimal.valueOf(Short.MIN_VALUE)) < 0);
+ return !fitsInSmallInt(value);
case INT:
- return (value.compareTo(BigDecimal.valueOf(Integer.MAX_VALUE)) > 0 ||
- value.compareTo(BigDecimal.valueOf(Integer.MIN_VALUE)) < 0);
+ return !fitsInInt(value);
case BIGINT:
- return (value.compareTo(BigDecimal.valueOf(Long.MAX_VALUE)) > 0 ||
- value.compareTo(BigDecimal.valueOf(Long.MIN_VALUE)) < 0);
+ return !fitsInBigInt(value);
case FLOAT:
- return (value.compareTo(BigDecimal.valueOf(Float.MAX_VALUE)) > 0 ||
- value.compareTo(BigDecimal.valueOf(Float.MIN_VALUE)) < 0);
+ return !fitsInFloat(value);
case DOUBLE:
- return (value.compareTo(BigDecimal.valueOf(Double.MAX_VALUE)) > 0 ||
- value.compareTo(BigDecimal.valueOf(Double.MIN_VALUE)) < 0);
+ return !fitsInDouble(value);
case DECIMAL:
- return (TypesUtil.computeDecimalType(value) == null);
+ return !isImpalaDecimal(value);
default:
- throw new AnalysisException("Overflow check on " + type + " isn't supported.");
+ throw new IllegalArgumentException("Overflow check on " + type.toSql() +
+ " isn't supported.");
}
}
}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/analysis/Parser.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/Parser.java b/fe/src/main/java/org/apache/impala/analysis/Parser.java
index 776bc79..184d975 100644
--- a/fe/src/main/java/org/apache/impala/analysis/Parser.java
+++ b/fe/src/main/java/org/apache/impala/analysis/Parser.java
@@ -34,20 +34,40 @@ public class Parser {
}
}
- public static StatementBase parse(String query) throws ParseException {
+ /**
+ * Parse the statement using default options. Used for testing and for
+ * parsing internally-generated statements. See the full version for details.
+ */
+ public static StatementBase parse(String query) throws AnalysisException {
return parse(query, new TQueryOptions());
}
+ /**
+ * Run the parser with the given input and options, returning an
+ * abstract syntax tree (AST) for the statement, ready for semantic analysis.
+ *
+ * @param query the query: a valid SQL statement
+ * @param options query options
+ * @return the AST for the statement
+ * @throws AnalysisException for errors which are of two kinds:
+ * {@link ParseException} for syntactic errors, or the generic
+ * AnalysisException for semantic errors (such as overflow of a
+ * numeric literal)
+ */
public static StatementBase parse(String query, TQueryOptions options)
- throws ParseException {
+ throws AnalysisException {
SqlScanner input = new SqlScanner(new StringReader(query));
SqlParser parser = new SqlParser(input);
parser.setQueryOptions(options);
try {
return (StatementBase) parser.parse().value;
+ } catch (AnalysisException e) {
+ // Pass along any analysis exceptions as they are. Since AnalysisException
+ // is a kind of Exception, we have to special-case it here
+ throw e;
} catch (Exception e) {
// Annoyingly, the parser uses a generic Exception to report
- // errors. Translate it to something more helpful.
+ // syntax errors. Translate it to something more helpful.
throw new ParseException(parser.getErrorMsg(query), e);
}
}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java b/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
index 1c03a9e..4426026 100644
--- a/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
+++ b/fe/src/main/java/org/apache/impala/analysis/QueryStmt.java
@@ -26,6 +26,7 @@ import org.apache.impala.catalog.FeView;
import org.apache.impala.catalog.Type;
import org.apache.impala.catalog.View;
import org.apache.impala.common.AnalysisException;
+import org.apache.impala.common.SqlCastException;
import org.apache.impala.planner.DataSink;
import org.apache.impala.planner.PlanRootSink;
@@ -425,11 +426,12 @@ public abstract class QueryStmt extends StatementBase {
public SortInfo getSortInfo() { return sortInfo_; }
public boolean evaluateOrderBy() { return evaluateOrderBy_; }
public List<Expr> getBaseTblResultExprs() { return baseTblResultExprs_; }
- public void setLimit(long limit) throws AnalysisException {
+
+ public void setLimit(long limit) throws SqlCastException {
Preconditions.checkState(limit >= 0);
long newLimit = hasLimit() ? Math.min(limit, getLimit()) : limit;
- limitElement_ = new LimitElement(new NumericLiteral(Long.toString(newLimit),
- Type.BIGINT), limitElement_.getOffsetExpr());
+ limitElement_ = new LimitElement(NumericLiteral.create(newLimit, Type.BIGINT),
+ limitElement_.getOffsetExpr());
}
/**
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java b/fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
index 6f29dc0..653633a 100644
--- a/fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
+++ b/fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java
@@ -290,6 +290,9 @@ public class ScalarFunction extends Function {
}
@Override
+ public String toString() { return toSql(false); }
+
+ @Override
public TFunction toThrift() {
TFunction fn = super.toThrift();
fn.setScalar_fn(new TScalarFunction());
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/catalog/ScalarType.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/catalog/ScalarType.java b/fe/src/main/java/org/apache/impala/catalog/ScalarType.java
index 43df496..12f9897 100644
--- a/fe/src/main/java/org/apache/impala/catalog/ScalarType.java
+++ b/fe/src/main/java/org/apache/impala/catalog/ScalarType.java
@@ -526,4 +526,12 @@ public class ScalarType extends Type {
boolean strict, boolean strictDecimal) {
return getAssignmentCompatibleType(t1, t2, strict, strictDecimal).matchesType(t2);
}
+
+ /**
+ * @return true if dest = source is valid (if source is at least as
+ * wide as dest.)
+ */
+ public static boolean isAssignable(ScalarType dest, ScalarType source) {
+ return isImplicitlyCastable(source, dest, false, false);
+ }
}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/catalog/Type.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/catalog/Type.java b/fe/src/main/java/org/apache/impala/catalog/Type.java
index d685c0a..73f15e0 100644
--- a/fe/src/main/java/org/apache/impala/catalog/Type.java
+++ b/fe/src/main/java/org/apache/impala/catalog/Type.java
@@ -17,21 +17,17 @@
package org.apache.impala.catalog;
-import java.io.StringReader;
import java.util.ArrayList;
import java.util.List;
import org.apache.impala.analysis.CreateTableStmt;
import org.apache.impala.analysis.Parser;
-import org.apache.impala.analysis.Parser.ParseException;
-import org.apache.impala.analysis.SqlParser;
-import org.apache.impala.analysis.SqlScanner;
import org.apache.impala.analysis.StatementBase;
import org.apache.impala.analysis.TypeDef;
+import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.Pair;
import org.apache.impala.thrift.TColumnType;
import org.apache.impala.thrift.TPrimitiveType;
-import org.apache.impala.thrift.TQueryOptions;
import org.apache.impala.thrift.TScalarType;
import org.apache.impala.thrift.TStructField;
import org.apache.impala.thrift.TTypeNode;
@@ -276,7 +272,7 @@ public abstract class Type {
// Should never get here.
throw new IllegalStateException("Invalid create table stmt.");
}
- } catch (ParseException e) {
+ } catch (AnalysisException e) {
return null;
}
TypeDef typeDef = createTableStmt.getColumnDefs().get(0).getTypeDef();
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/catalog/View.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/catalog/View.java b/fe/src/main/java/org/apache/impala/catalog/View.java
index 5d8be36..abc6be8 100644
--- a/fe/src/main/java/org/apache/impala/catalog/View.java
+++ b/fe/src/main/java/org/apache/impala/catalog/View.java
@@ -17,26 +17,21 @@
package org.apache.impala.catalog;
-import java.io.StringReader;
import java.util.List;
import java.util.Set;
import org.apache.hadoop.hive.metastore.IMetaStoreClient;
import org.apache.hadoop.hive.metastore.api.FieldSchema;
-
-import org.apache.impala.analysis.ParseNode;
import org.apache.impala.analysis.Parser;
-import org.apache.impala.analysis.Parser.ParseException;
import org.apache.impala.analysis.QueryStmt;
-import org.apache.impala.analysis.SqlParser;
-import org.apache.impala.analysis.SqlScanner;
import org.apache.impala.analysis.StatementBase;
+import org.apache.impala.common.AnalysisException;
import org.apache.impala.thrift.TCatalogObjectType;
-import org.apache.impala.thrift.TQueryOptions;
import org.apache.impala.thrift.TTable;
import org.apache.impala.thrift.TTableDescriptor;
import org.apache.impala.thrift.TTableStats;
import org.apache.impala.thrift.TTableType;
+
import com.google.common.collect.Lists;
/**
@@ -143,7 +138,7 @@ public class View extends Table implements FeView {
StatementBase node;
try {
node = Parser.parse(inlineViewDef);
- } catch (ParseException e) {
+ } catch (AnalysisException e) {
// Do not pass e as the exception cause because it might reveal the existence
// of tables that the user triggering this load may not have privileges on.
throw new TableLoadingException(
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/common/InvalidValueException.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/common/InvalidValueException.java b/fe/src/main/java/org/apache/impala/common/InvalidValueException.java
new file mode 100644
index 0000000..9be30e1
--- /dev/null
+++ b/fe/src/main/java/org/apache/impala/common/InvalidValueException.java
@@ -0,0 +1,35 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.common;
+
+/**
+ * Indicates an invalid value: constant, name, etc.
+ * Fine-grained exception for better testing.
+ */
+@SuppressWarnings("serial")
+public class InvalidValueException extends AnalysisException {
+
+ public InvalidValueException(String msg) {
+ super(msg);
+ }
+
+ public InvalidValueException(String msg, NumberFormatException e) {
+ super(msg, e);
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/common/SqlCastException.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/common/SqlCastException.java b/fe/src/main/java/org/apache/impala/common/SqlCastException.java
new file mode 100644
index 0000000..370bed0
--- /dev/null
+++ b/fe/src/main/java/org/apache/impala/common/SqlCastException.java
@@ -0,0 +1,40 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.common;
+
+import java.math.BigDecimal;
+
+import org.apache.impala.catalog.Type;
+
+/**
+ * Indicates a plan-time cast exception which is caught and handled by some
+ * parts of the planner.
+ */
+@SuppressWarnings("serial")
+public class SqlCastException extends AnalysisException {
+
+ public SqlCastException(String msg) {
+ super(msg);
+ }
+
+ public SqlCastException(BigDecimal value, Type type) {
+ this("Value " + value.toString() +
+ " cannot be cast to type " + type.toSql());
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/main/java/org/apache/impala/common/UnsupportedFeatureException.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/common/UnsupportedFeatureException.java b/fe/src/main/java/org/apache/impala/common/UnsupportedFeatureException.java
new file mode 100644
index 0000000..e04664c
--- /dev/null
+++ b/fe/src/main/java/org/apache/impala/common/UnsupportedFeatureException.java
@@ -0,0 +1,31 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.common;
+
+/**
+ * Identifies that a feature or operation is not supported in
+ * the analyzer. Allows fine-grain error checking in tests.
+ */
+@SuppressWarnings("serial")
+public class UnsupportedFeatureException extends AnalysisException {
+
+ public UnsupportedFeatureException(String msg) {
+ super(msg);
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java b/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
index b547526..668b4de 100644
--- a/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
@@ -103,9 +103,9 @@ public class AnalyzeExprsTest extends AnalyzerTest {
testNumericLiteral("-" + Double.toString(Double.MAX_VALUE), Type.DOUBLE);
AnalysisError(String.format("select %s1", Double.toString(Double.MAX_VALUE)),
- "Numeric literal '1.7976931348623157E+3081' exceeds maximum range of doubles.");
+ "Numeric literal '1.7976931348623157E+3081' exceeds maximum range of DOUBLE.");
AnalysisError(String.format("select %s1", Double.toString(Double.MIN_VALUE)),
- "Numeric literal '4.9E-3241' underflows minimum resolution of doubles.");
+ "Numeric literal '4.9E-3241' underflows minimum resolution of DOUBLE.");
// Test edge cases near the upper limits of decimal precision - 38 digits.
// Integer literal.
@@ -2700,6 +2700,7 @@ public class AnalyzeExprsTest extends AnalyzerTest {
}
AnalyzesOk(exprStr.toString());
exprStr.append(repeatSuffix);
+ exprStr.append(repeatSuffix);
AnalysisError(exprStr.toString(),
String.format("Exceeded the maximum depth of an expression tree (%s).",
Expr.EXPR_DEPTH_LIMIT));
@@ -2722,7 +2723,7 @@ public class AnalyzeExprsTest extends AnalyzerTest {
AnalyzesOk("select " + getNestedFuncExpr(openFunc, baseArg, closeFunc,
Expr.EXPR_DEPTH_LIMIT - 1));
AnalysisError("select " + getNestedFuncExpr(openFunc, baseArg, closeFunc,
- Expr.EXPR_DEPTH_LIMIT),
+ Expr.EXPR_DEPTH_LIMIT + 1),
String.format("Exceeded the maximum depth of an expression tree (%s).",
Expr.EXPR_DEPTH_LIMIT));
// Test 10x the safe depth.
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java b/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java
index 4dc5b02..88421a0 100644
--- a/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java
@@ -19,13 +19,16 @@ package org.apache.impala.analysis;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
import java.util.ArrayList;
import java.util.List;
+import org.apache.impala.catalog.ScalarType;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.FrontendTestBase;
import org.apache.impala.common.ImpalaException;
+import org.apache.impala.common.SqlCastException;
import org.apache.impala.rewrite.BetweenToCompoundRule;
import org.apache.impala.rewrite.EqualityDisjunctsToInRule;
import org.apache.impala.rewrite.ExprRewriteRule;
@@ -532,6 +535,56 @@ public class ExprRewriteRulesTest extends FrontendTestBase {
"coalesce(min(tinyint_col), 42)");
}
+ /**
+ * Special case in which a numeric literal is explicitly cast to an
+ * incompatible type. In this case, no constant folding should be done.
+ * Runtime relies on the fact that a DECIMAL numeric overflow in V1
+ * resulted in a NULL.
+ */
+ @Test
+ public void TestCoalesceDecimal() throws ImpalaException {
+ String query =
+ "SELECT coalesce(1.8, CAST(0 AS DECIMAL(38,38))) AS c " +
+ " FROM functional.alltypestiny";
+ // Try both with and without rewrites
+ for (int i = 0; i < 2; i++) {
+ boolean rewrite = i == 1;
+
+ // Analyze the expression with the rewrite option and
+ // with Decimal V2 disabled.
+ AnalysisContext ctx = createAnalysisCtx();
+ ctx.getQueryOptions().setEnable_expr_rewrites(rewrite);
+ ctx.getQueryOptions().setDecimal_v2(false);
+ SelectStmt stmt = (SelectStmt) AnalyzesOk(query, ctx);
+
+ // Select list expr takes widest type
+ Expr expr = stmt.getSelectList().getItems().get(0).getExpr();
+ assertTrue(expr instanceof FunctionCallExpr);
+ assertEquals(ScalarType.createDecimalType(38,38), expr.getType());
+
+ // First arg to coalesce should be an implicit cast
+ Expr arg = expr.getChild(0);
+ assertTrue(arg instanceof CastExpr);
+ assertEquals(ScalarType.createDecimalType(38,38), arg.getType());
+
+ // Input to the cast is the numeric literal with its original type
+ Expr num = arg.getChild(0);
+ assertTrue(num instanceof NumericLiteral);
+ assertEquals(ScalarType.createDecimalType(2,1), num.getType());
+ }
+
+ // In V2, the query fails with a cast exception
+ try {
+ AnalysisContext ctx = createAnalysisCtx();
+ ctx.getQueryOptions().setEnable_expr_rewrites(true);
+ ctx.getQueryOptions().setDecimal_v2(true);
+ parseAndAnalyze(query, ctx);
+ fail();
+ } catch (SqlCastException e) {
+ // Expected
+ }
+ }
+
@Test
public void testNormalizeExprsRule() throws ImpalaException {
ExprRewriteRule rule = NormalizeExprsRule.INSTANCE;
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/analysis/ExprTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/ExprTest.java b/fe/src/test/java/org/apache/impala/analysis/ExprTest.java
deleted file mode 100644
index e0eca0a..0000000
--- a/fe/src/test/java/org/apache/impala/analysis/ExprTest.java
+++ /dev/null
@@ -1,90 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.impala.analysis;
-
-import static org.junit.Assert.fail;
-
-import org.apache.impala.catalog.ScalarType;
-import org.apache.impala.catalog.Type;
-import org.junit.Test;
-
-
-public class ExprTest {
- // Test creation of LiteralExprs from Strings, e.g., for partitioning keys.
- @Test
- public void TestLiteralExpr() {
- testLiteralExprPositive("false", Type.BOOLEAN);
- testLiteralExprPositive("1", Type.TINYINT);
- testLiteralExprPositive("1", Type.SMALLINT);
- testLiteralExprPositive("1", Type.INT);
- testLiteralExprPositive("1", Type.BIGINT);
- testLiteralExprPositive("1.0", Type.FLOAT);
- testLiteralExprPositive("1.0", Type.DOUBLE);
- testLiteralExprPositive("ABC", Type.STRING);
- testLiteralExprPositive("1.1", ScalarType.createDecimalType(2, 1));
-
- // INVALID_TYPE should always fail
- testLiteralExprNegative("ABC", Type.INVALID);
-
- // Invalid casts
- testLiteralExprNegative("ABC", Type.BOOLEAN);
- testLiteralExprNegative("ABC", Type.TINYINT);
- testLiteralExprNegative("ABC", Type.SMALLINT);
- testLiteralExprNegative("ABC", Type.INT);
- testLiteralExprNegative("ABC", Type.BIGINT);
- testLiteralExprNegative("ABC", Type.FLOAT);
- testLiteralExprNegative("ABC", Type.DOUBLE);
- testLiteralExprNegative("ABC", Type.TIMESTAMP);
- testLiteralExprNegative("ABC", ScalarType.createDecimalType());
-
- // Date types not implemented
- testLiteralExprNegative("2010-01-01", Type.DATE);
- testLiteralExprNegative("2010-01-01", Type.DATETIME);
- testLiteralExprNegative("2010-01-01", Type.TIMESTAMP);
- }
-
- private void testLiteralExprPositive(String value, Type type) {
- LiteralExpr expr = null;
- try {
- expr = LiteralExpr.create(value, type);
- } catch (Exception e) {
- fail("\nFailed to create LiteralExpr of type: " + type.toString() + " from: " + value
- + " due to " + e.getMessage() + "\n");
- }
- if (expr == null) {
- fail("\nFailed to create LiteralExpr\n");
- }
- }
-
- private void testLiteralExprNegative(String value, Type type) {
- boolean failure = false;
- LiteralExpr expr = null;
- try {
- expr = LiteralExpr.create(value, type);
- } catch (Exception e) {
- failure = true;
- }
- if (expr == null) {
- failure = true;
- }
- if (!failure) {
- fail("\nUnexpectedly succeeded to create LiteralExpr of type: "
- + type.toString() + " from: " + value + "\n");
- }
- }
-}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/analysis/LiteralExprTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/LiteralExprTest.java b/fe/src/test/java/org/apache/impala/analysis/LiteralExprTest.java
new file mode 100644
index 0000000..bc2ae0c
--- /dev/null
+++ b/fe/src/test/java/org/apache/impala/analysis/LiteralExprTest.java
@@ -0,0 +1,176 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.analysis;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import org.apache.impala.catalog.ScalarType;
+import org.apache.impala.catalog.Type;
+import org.apache.impala.common.FrontendTestBase;
+import org.apache.impala.common.ImpalaException;
+import org.apache.impala.common.SqlCastException;
+import org.apache.impala.rewrite.ExprRewriteRule;
+import org.apache.impala.rewrite.FoldConstantsRule;
+import org.junit.Test;
+
+/**
+ * Test literal expressions from the Java side. Note that a comprehensive
+ * set of tests exist on the C++ site in expr-test.cc. Those tests call
+ * into Java, then verify the results. Very complete, but hard to debug
+ * from the Java side.
+ */
+public class LiteralExprTest extends FrontendTestBase {
+ // Test creation of LiteralExprs from Strings, e.g., for partitioning keys.
+ @Test
+ public void TestLiteralExpr() {
+ testLiteralExprPositive("false", Type.BOOLEAN);
+ testLiteralExprPositive("1", Type.TINYINT);
+ testLiteralExprPositive("1", Type.SMALLINT);
+ testLiteralExprPositive("1", Type.INT);
+ testLiteralExprPositive("1", Type.BIGINT);
+ testLiteralExprPositive("1.0", Type.FLOAT);
+ testLiteralExprPositive("1.0", Type.DOUBLE);
+ testLiteralExprPositive("ABC", Type.STRING);
+ testLiteralExprPositive("1.1", ScalarType.createDecimalType(2, 1));
+
+ // INVALID_TYPE should always fail
+ testLiteralExprNegative("ABC", Type.INVALID);
+
+ // Invalid casts
+ testLiteralExprNegative("ABC", Type.BOOLEAN);
+ testLiteralExprNegative("ABC", Type.TINYINT);
+ testLiteralExprNegative("ABC", Type.SMALLINT);
+ testLiteralExprNegative("ABC", Type.INT);
+ testLiteralExprNegative("ABC", Type.BIGINT);
+ testLiteralExprNegative("ABC", Type.FLOAT);
+ testLiteralExprNegative("ABC", Type.DOUBLE);
+ testLiteralExprNegative("ABC", Type.TIMESTAMP);
+ testLiteralExprNegative("ABC", ScalarType.createDecimalType());
+
+ // Date types not implemented
+ testLiteralExprNegative("2010-01-01", Type.DATE);
+ testLiteralExprNegative("2010-01-01", Type.DATETIME);
+ testLiteralExprNegative("2010-01-01", Type.TIMESTAMP);
+ }
+
+ private void testLiteralExprPositive(String value, Type type) {
+ LiteralExpr expr = null;
+ try {
+ expr = LiteralExpr.create(value, type);
+ } catch (Exception e) {
+ fail("\nFailed to create LiteralExpr of type: " + type.toString() +
+ " from: " + value + " due to " + e.getMessage() + "\n");
+ }
+ if (expr == null) {
+ fail("\nFailed to create LiteralExpr\n");
+ }
+ }
+
+ private void testLiteralExprNegative(String value, Type type) {
+ boolean failure = false;
+ LiteralExpr expr = null;
+ try {
+ expr = LiteralExpr.create(value, type);
+ } catch (Exception e) {
+ failure = true;
+ }
+ if (expr == null) {
+ failure = true;
+ }
+ if (!failure) {
+ fail("\nUnexpectedly succeeded to create LiteralExpr of type: "
+ + type.toString() + " from: " + value + "\n");
+ }
+ }
+
+ private Expr analyze(String query, boolean useDecimalV2, boolean enableRewrite) {
+ AnalysisContext ctx = createAnalysisCtx();
+ ctx.getQueryOptions().setDecimal_v2(useDecimalV2);
+ ctx.getQueryOptions().setEnable_expr_rewrites(enableRewrite);
+ return ((SelectStmt) AnalyzesOk(query, ctx)).getSelectList()
+ .getItems().get(0).getExpr();
+ }
+
+ /**
+ * Test extreme literal cases to ensure the value passes
+ * through the analyzer correctly.
+ */
+ @Test
+ public void testLiteralCast() {
+ for (int i = 0; i < 3; i++) {
+ boolean useDecimalV2 = i > 1;
+ boolean enableRewrite = (i % 2) == 1;
+ {
+ // Boundary case in which the positive value is a DECIMAL,
+ // becomes BIGINT when negated by the parser.
+ String query = "select getBit(" + Long.MIN_VALUE + ", 63)";
+ Expr expr = analyze(query, useDecimalV2, enableRewrite);
+ assertEquals(Type.TINYINT, expr.getType());
+ }
+ {
+ // Would eval to NaN, so keep original expr.
+ String query = "select cast(10 as double) / cast(0 as double)";
+ Expr expr = analyze(query, useDecimalV2, enableRewrite);
+ assertEquals(Type.DOUBLE, expr.getType());
+ assertTrue(expr instanceof ArithmeticExpr);
+ }
+ {
+ // Extreme double value. Ensure double-->BigDecimal noise
+ // does not cause overflows
+ String query = "select cast(" + Double.toString(Double.MAX_VALUE) +
+ " as double)";
+ Expr expr = analyze(query, useDecimalV2, enableRewrite);
+ assertEquals(Type.DOUBLE, expr.getType());
+ if (enableRewrite) {
+ assertTrue(expr instanceof NumericLiteral);
+ } else {
+ assertTrue(expr instanceof CastExpr);
+ }
+ }
+ {
+ // As above, but for extreme minimum (smallest) values
+ String query = "select cast(" + Double.toString(Double.MIN_VALUE) +
+ " as double)";
+ Expr expr = analyze(query, useDecimalV2, enableRewrite);
+ assertEquals(Type.DOUBLE, expr.getType());
+ if (enableRewrite) {
+ assertTrue(expr instanceof NumericLiteral);
+ } else {
+ assertTrue(expr instanceof CastExpr);
+ }
+ }
+ {
+ // Math would cause overflow, don't rewrite
+ String query = "select cast(1.7976931348623157e+308 as double)" +
+ " / cast(2.2250738585072014e-308 as double)";
+ Expr expr = analyze(query, useDecimalV2, enableRewrite);
+ assertEquals(Type.DOUBLE, expr.getType());
+ assertTrue(expr instanceof ArithmeticExpr);
+ }
+ {
+ // Math would cause overflow, don't rewrite
+ String query = "select cast(cast(1.7976931348623157e+308 as double) as float)";
+ Expr expr = analyze(query, useDecimalV2, enableRewrite);
+ assertEquals(Type.FLOAT, expr.getType());
+ assertTrue(expr instanceof CastExpr);
+ }
+ }
+ }
+}
[2/4] impala git commit: IMPALA-6544: S3 tests fail because S3 is
eventually consistent.
Posted by bo...@apache.org.
IMPALA-6544: S3 tests fail because S3 is eventually consistent.
This skips test_insert_parquet on S3, as S3's eventually consistent
behavior is exposing us to "file not found" errors.
To remove this band-aid, we'll need to either handle s3 consistency
natively in some way, or to retry certain tests.
Note that this may be only a drop in a big bucket at the moment;
I think we see these issues in a variety of places.
Change-Id: I22e7620e97366bece69a20885f5c3b75de05fab6
Reviewed-on: http://gerrit.cloudera.org:8080/12127
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/6f6d1f64
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/6f6d1f64
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/6f6d1f64
Branch: refs/heads/master
Commit: 6f6d1f64a10f9a958dfa6162ec8bdab4cdcc6d92
Parents: 378f8c2
Author: Philip Zeyliger <ph...@cloudera.com>
Authored: Fri Dec 21 13:36:54 2018 -0800
Committer: Impala Public Jenkins <im...@cloudera.com>
Committed: Fri Dec 28 01:50:39 2018 +0000
----------------------------------------------------------------------
tests/common/skip.py | 2 ++
tests/query_test/test_insert_parquet.py | 2 +-
2 files changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/6f6d1f64/tests/common/skip.py
----------------------------------------------------------------------
diff --git a/tests/common/skip.py b/tests/common/skip.py
index 8ba7a06..b58f057 100644
--- a/tests/common/skip.py
+++ b/tests/common/skip.py
@@ -56,6 +56,8 @@ class SkipIfS3:
hbase = pytest.mark.skipif(IS_S3, reason="HBase not started with S3")
qualified_path = pytest.mark.skipif(IS_S3,
reason="Tests rely on HDFS qualified paths, IMPALA-1872")
+ eventually_consistent = pytest.mark.skipif(IS_S3,
+ reason="Flakiness on account of S3 eventual consistency.")
class SkipIfABFS:
http://git-wip-us.apache.org/repos/asf/impala/blob/6f6d1f64/tests/query_test/test_insert_parquet.py
----------------------------------------------------------------------
diff --git a/tests/query_test/test_insert_parquet.py b/tests/query_test/test_insert_parquet.py
index 0bf28fe..6924d26 100644
--- a/tests/query_test/test_insert_parquet.py
+++ b/tests/query_test/test_insert_parquet.py
@@ -22,7 +22,6 @@ import os
from collections import namedtuple
from datetime import datetime
from decimal import Decimal
-from shutil import rmtree
from subprocess import check_call
from parquet.ttypes import ColumnOrder, SortingColumn, TypeDefinedOrder
@@ -104,6 +103,7 @@ class TestInsertParquetQueries(ImpalaTestSuite):
@SkipIfEC.oom
@SkipIfLocal.multiple_impalad
+ @SkipIfS3.eventually_consistent
@UniqueDatabase.parametrize(sync_ddl=True)
def test_insert_parquet(self, vector, unique_database):
vector.get_value('exec_option')['PARQUET_FILE_SIZE'] = \
[3/4] impala git commit: IMPALA-7902: NumericLiteral fixes,
refactoring
Posted by bo...@apache.org.
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/analysis/NumericLiteralTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/NumericLiteralTest.java b/fe/src/test/java/org/apache/impala/analysis/NumericLiteralTest.java
new file mode 100644
index 0000000..23f8d88
--- /dev/null
+++ b/fe/src/test/java/org/apache/impala/analysis/NumericLiteralTest.java
@@ -0,0 +1,655 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.impala.analysis;
+
+import static org.junit.Assert.*;
+
+import java.math.BigDecimal;
+
+import org.apache.impala.catalog.ScalarType;
+import org.apache.impala.catalog.Type;
+import org.apache.impala.common.AnalysisException;
+import org.apache.impala.common.InvalidValueException;
+import org.apache.impala.common.SqlCastException;
+import org.junit.Test;
+
+/**
+ * Tests the numeric literal which is complex because of its ability to hold
+ * values across many types. The value and type must be compatible at all times.
+ *
+ * Note that a comprehensive set of tests exist on the C++ site in expr-test.cc.
+ * Those tests call into Java, then verify the results. Very complete, but hard
+ * to debug from the Java side.
+ */
+public class NumericLiteralTest {
+
+ // Approximate maximum DECIMAL scale for a BIGINT
+ // 9,223,372,036,854,775,807
+ private static final int MAX_BIGINT_PRECISION = 19;
+
+ // "One above" and "one below" values for testing type ranges
+ private static final BigDecimal ABOVE_TINYINT =
+ NumericLiteral.MAX_TINYINT.add(BigDecimal.ONE);
+ private static final BigDecimal BELOW_TINYINT =
+ NumericLiteral.MIN_TINYINT.subtract(BigDecimal.ONE);
+ private static final BigDecimal ABOVE_SMALLINT =
+ NumericLiteral.MAX_SMALLINT.add(BigDecimal.ONE);
+ private static final BigDecimal BELOW_SMALLINT =
+ NumericLiteral.MIN_SMALLINT.subtract(BigDecimal.ONE);
+ private static final BigDecimal ABOVE_INT =
+ NumericLiteral.MAX_INT.add(BigDecimal.ONE);
+ private static final BigDecimal BELOW_INT =
+ NumericLiteral.MIN_INT.subtract(BigDecimal.ONE);
+ private static final BigDecimal ABOVE_BIGINT =
+ NumericLiteral.MAX_BIGINT.add(BigDecimal.ONE);
+ private static final BigDecimal BELOW_BIGINT =
+ NumericLiteral.MIN_BIGINT.subtract(BigDecimal.ONE);
+
+ private static String repeat(String str, int n) {
+ StringBuilder buf = new StringBuilder();
+ for (int i = 0; i < n; i++) buf.append(str);
+ return buf.toString();
+ }
+
+ private static String genDecimal(int precision, int scale) {
+ if (scale == 0) {
+ return repeat("9", precision);
+ } else {
+ return repeat("9", precision - scale) + "." +
+ repeat("4", scale);
+ }
+ }
+
+ @Test
+ public void testBasics() throws AnalysisException {
+ NumericLiteral n = NumericLiteral.create(0);
+ // Starts with the smallest possible type
+ assertEquals(Type.TINYINT, n.getType());
+ // Literals start analyzed
+ assertTrue(n.isAnalyzed());
+ // With their costs set
+ assertEquals(LiteralExpr.LITERAL_COST, n.getCost(), 0.01);
+ assertEquals(-1, n.getSelectivity(), 0.01);
+ // Sanity check of the string representation
+ assertEquals("0", n.getStringValue());
+ assertEquals("0:TINYINT", n.toString());
+ // Sanity check of casting
+ n.castTo(Type.SMALLINT);
+ assertEquals(Type.SMALLINT, n.getType());
+ assertEquals("0:SMALLINT", n.toString());
+ }
+
+ /**
+ * Sanity test: we rely on the constants to be accurate the other tests.
+ * This will, hopefully, catch any accidental changes to them.
+ */
+ @Test
+ public void testConstants() throws InvalidValueException {
+ assertEquals(BigDecimal.valueOf(Byte.MIN_VALUE), NumericLiteral.MIN_TINYINT);
+ assertEquals(BigDecimal.valueOf(Byte.MAX_VALUE), NumericLiteral.MAX_TINYINT);
+ assertEquals(BigDecimal.valueOf(Short.MIN_VALUE), NumericLiteral.MIN_SMALLINT);
+ assertEquals(BigDecimal.valueOf(Short.MAX_VALUE), NumericLiteral.MAX_SMALLINT);
+ assertEquals(BigDecimal.valueOf(Integer.MIN_VALUE), NumericLiteral.MIN_INT);
+ assertEquals(BigDecimal.valueOf(Integer.MAX_VALUE), NumericLiteral.MAX_INT);
+ assertEquals(BigDecimal.valueOf(Long.MIN_VALUE), NumericLiteral.MIN_BIGINT);
+ assertEquals(BigDecimal.valueOf(Long.MAX_VALUE), NumericLiteral.MAX_BIGINT);
+ assertEquals(BigDecimal.valueOf(-Float.MAX_VALUE), NumericLiteral.MIN_FLOAT);
+ assertEquals(BigDecimal.valueOf(Float.MAX_VALUE), NumericLiteral.MAX_FLOAT);
+ assertEquals(BigDecimal.valueOf(-Double.MAX_VALUE), NumericLiteral.MIN_DOUBLE);
+ assertEquals(BigDecimal.valueOf(Double.MAX_VALUE), NumericLiteral.MAX_DOUBLE);
+ }
+
+ /**
+ * Detailed test of the mechanism to infer the "natural type" (smallest
+ * type) for a value.
+ */
+ @Test
+ public void testInferType() throws SqlCastException {
+ assertEquals(Type.TINYINT, NumericLiteral.inferType(BigDecimal.ZERO));
+ assertEquals(Type.TINYINT, NumericLiteral.inferType(NumericLiteral.MIN_TINYINT));
+ assertEquals(Type.TINYINT, NumericLiteral.inferType(NumericLiteral.MAX_TINYINT));
+
+ assertEquals(Type.SMALLINT, NumericLiteral.inferType(ABOVE_TINYINT));
+ assertEquals(Type.SMALLINT, NumericLiteral.inferType(BELOW_TINYINT));
+ assertEquals(Type.SMALLINT, NumericLiteral.inferType(NumericLiteral.MIN_SMALLINT));
+ assertEquals(Type.SMALLINT, NumericLiteral.inferType(NumericLiteral.MAX_SMALLINT));
+
+ assertEquals(Type.INT, NumericLiteral.inferType(ABOVE_SMALLINT));
+ assertEquals(Type.INT, NumericLiteral.inferType(BELOW_SMALLINT));
+ assertEquals(Type.INT, NumericLiteral.inferType(NumericLiteral.MIN_INT));
+ assertEquals(Type.INT, NumericLiteral.inferType(NumericLiteral.MAX_INT));
+
+ assertEquals(Type.BIGINT, NumericLiteral.inferType(ABOVE_INT));
+ assertEquals(Type.BIGINT, NumericLiteral.inferType(BELOW_INT));
+ assertEquals(Type.BIGINT, NumericLiteral.inferType(NumericLiteral.MIN_BIGINT));
+ assertEquals(Type.BIGINT, NumericLiteral.inferType(NumericLiteral.MAX_BIGINT));
+
+ // 9.9. Is DECIMAL
+ assertEquals(ScalarType.createDecimalType(2, 1),
+ NumericLiteral.inferType(
+ new BigDecimal(genDecimal(2, 1))));
+ assertEquals(ScalarType.createDecimalType(2, 1),
+ NumericLiteral.inferType(
+ new BigDecimal(genDecimal(2, 1)).negate()));
+
+ // One bigger or smaller than BIGINT
+ assertEquals(ScalarType.createDecimalType(MAX_BIGINT_PRECISION, 0),
+ NumericLiteral.inferType(ABOVE_BIGINT));
+ assertEquals(ScalarType.createDecimalType(MAX_BIGINT_PRECISION, 0),
+ NumericLiteral.inferType(BELOW_BIGINT));
+
+ // All 9s, just bigger than BIGINT
+ assertEquals(ScalarType.createDecimalType(MAX_BIGINT_PRECISION, 0),
+ NumericLiteral.inferType(
+ new BigDecimal(genDecimal(MAX_BIGINT_PRECISION, 0))));
+ assertEquals(ScalarType.createDecimalType(MAX_BIGINT_PRECISION, 0),
+ NumericLiteral.inferType(
+ new BigDecimal(genDecimal(MAX_BIGINT_PRECISION, 0)).negate()));
+
+ // All 9s, at limits of DECIMAL precision
+ assertEquals(ScalarType.createDecimalType(ScalarType.MAX_SCALE, 0),
+ NumericLiteral.inferType(
+ new BigDecimal(genDecimal(ScalarType.MAX_SCALE, 0))));
+ assertEquals(ScalarType.createDecimalType(ScalarType.MAX_SCALE, 0),
+ NumericLiteral.inferType(
+ new BigDecimal(genDecimal(ScalarType.MAX_SCALE, 0)).negate()));
+
+ // Too large for DECIMAL, flips to DOUBLE
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(
+ new BigDecimal(genDecimal(ScalarType.MAX_SCALE + 1, 0))));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(
+ new BigDecimal(genDecimal(ScalarType.MAX_SCALE + 1, 0)).negate()));
+
+ // Too large for Decimal, small enough for FLOAT
+ // DECIMAL range is e38 as is FLOAT. So, there is a small range
+ // in which we could flip from DECIMAL to FLOAT, but we choose
+ // to use DOUBLE even in this range.
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(NumericLiteral.MIN_FLOAT));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(NumericLiteral.MAX_FLOAT));
+ // Float.MIN_VALUE means smallest positive value, confusingly
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(Float.MIN_VALUE)));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(-Float.MIN_VALUE)));
+
+ // Too large for Decimal, exponent too large for FLOAT
+ String value = "12345" + repeat("0", 40);
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(value)));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(value).negate()));
+
+ value = repeat("9", 10) + repeat("0", 40);
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(value)));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(value).negate()));
+
+ // Too many digits for DOUBLE, but exponent fits
+ value = repeat("9", 30) + repeat("0", 50);
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(value)));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(value).negate()));
+ value = genDecimal(100, 10);
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(value)));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(value).negate()));
+
+ // Limit of DOUBLE range
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(NumericLiteral.MIN_DOUBLE));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(NumericLiteral.MAX_DOUBLE));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(Double.MIN_VALUE)));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(new BigDecimal(-Double.MIN_VALUE)));
+
+ // Too big for DOUBLE or DECIMAL
+ try {
+ NumericLiteral.inferType(new BigDecimal(genDecimal(309, 0)));
+ fail();
+ } catch (SqlCastException e) {
+ // Expected
+ }
+
+ // Overflows DOUBLE, but low digits are truncated, so is DOUBLE
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(
+ NumericLiteral.MAX_DOUBLE.add(BigDecimal.ONE)));
+ assertEquals(Type.DOUBLE,
+ NumericLiteral.inferType(
+ NumericLiteral.MAX_DOUBLE.add(BigDecimal.ONE).negate()));
+
+ // Another power of 10, actual overflow
+ try {
+ NumericLiteral.inferType(
+ NumericLiteral.MAX_DOUBLE.multiply(BigDecimal.TEN));
+ fail();
+ } catch (SqlCastException e) {
+ // Expected
+ }
+ try {
+ NumericLiteral.inferType(
+ NumericLiteral.MAX_DOUBLE.multiply(BigDecimal.TEN).negate());
+ fail();
+ } catch (SqlCastException e) {
+ // Expected
+ }
+
+ // Too small for DOUBLE, too many digits for DECIMAL
+ // BUG: Should round to zero per SQL standard.
+ value = "." + repeat("0", 325) + "1";
+ try {
+ NumericLiteral.inferType(new BigDecimal(value));
+ fail();
+ } catch (SqlCastException e) {
+ // Expected
+ }
+ }
+
+ @Test
+ public void testIsOverflow() throws InvalidValueException {
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.ZERO, Type.TINYINT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MIN_TINYINT, Type.TINYINT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MAX_TINYINT, Type.TINYINT));
+ assertTrue(NumericLiteral.isOverflow(ABOVE_TINYINT, Type.TINYINT));
+ assertTrue(NumericLiteral.isOverflow(BELOW_TINYINT, Type.TINYINT));
+
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.ZERO, Type.SMALLINT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MIN_SMALLINT, Type.SMALLINT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MAX_SMALLINT, Type.SMALLINT));
+ assertTrue(NumericLiteral.isOverflow(ABOVE_SMALLINT, Type.SMALLINT));
+ assertTrue(NumericLiteral.isOverflow(BELOW_SMALLINT, Type.SMALLINT));
+
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.ZERO, Type.INT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MIN_INT, Type.INT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MAX_INT, Type.INT));
+ assertTrue(NumericLiteral.isOverflow(ABOVE_INT, Type.INT));
+ assertTrue(NumericLiteral.isOverflow(BELOW_INT, Type.INT));
+
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.ZERO, Type.BIGINT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MIN_BIGINT, Type.BIGINT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MAX_BIGINT, Type.BIGINT));
+ assertTrue(NumericLiteral.isOverflow(ABOVE_BIGINT, Type.BIGINT));
+ assertTrue(NumericLiteral.isOverflow(BELOW_BIGINT, Type.BIGINT));
+
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.ZERO, Type.FLOAT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MIN_FLOAT, Type.FLOAT));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MAX_FLOAT, Type.FLOAT));
+ assertFalse(NumericLiteral.isOverflow(
+ BigDecimal.valueOf(Float.MIN_VALUE), Type.FLOAT));
+ assertTrue(NumericLiteral.isOverflow(
+ NumericLiteral.MAX_FLOAT.add(BigDecimal.ONE), Type.FLOAT));
+ assertTrue(NumericLiteral.isOverflow(
+ NumericLiteral.MAX_FLOAT.multiply(BigDecimal.TEN), Type.FLOAT));
+ // Underflow is not overflow
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.valueOf(
+ Float.MIN_VALUE).divide(BigDecimal.TEN), Type.FLOAT));
+
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.ZERO, Type.DOUBLE));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MIN_DOUBLE, Type.DOUBLE));
+ assertFalse(NumericLiteral.isOverflow(NumericLiteral.MAX_DOUBLE, Type.DOUBLE));
+ assertFalse(NumericLiteral.isOverflow(
+ BigDecimal.valueOf(Double.MIN_VALUE), Type.DOUBLE));
+ // Have to add quite a bit (enough to hit a significant digit in the
+ // double), else BigDecimal truncates the bits when converting to double.
+ assertTrue(NumericLiteral.isOverflow(
+ NumericLiteral.MAX_DOUBLE.add(new BigDecimal("1e300")), Type.DOUBLE));
+ assertTrue(NumericLiteral.isOverflow(
+ NumericLiteral.MAX_DOUBLE.multiply(BigDecimal.TEN), Type.DOUBLE));
+ // Underflow is not overflow
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.valueOf(
+ Double.MIN_VALUE).divide(BigDecimal.TEN), Type.DOUBLE));
+
+ assertFalse(NumericLiteral.isOverflow(BigDecimal.ZERO, Type.DECIMAL));
+ assertFalse(NumericLiteral.isOverflow(
+ new BigDecimal(genDecimal(10,5)), Type.DECIMAL));
+ assertFalse(NumericLiteral.isOverflow(
+ new BigDecimal(genDecimal(10,5)), Type.DECIMAL));
+ assertFalse(NumericLiteral.isOverflow(
+ new BigDecimal(genDecimal(ScalarType.MAX_PRECISION, 0)), Type.DECIMAL));
+ assertFalse(NumericLiteral.isOverflow(
+ new BigDecimal(genDecimal(0, ScalarType.MAX_PRECISION)), Type.DECIMAL));
+ assertTrue(NumericLiteral.isOverflow(
+ new BigDecimal(genDecimal(ScalarType.MAX_PRECISION + 1, 0)), Type.DECIMAL));
+ assertTrue(NumericLiteral.isOverflow(
+ new BigDecimal(genDecimal(ScalarType.MAX_PRECISION + 1, 1)), Type.DECIMAL));
+ assertTrue(NumericLiteral.isOverflow(
+ new BigDecimal(genDecimal(0, ScalarType.MAX_PRECISION + 1)), Type.DECIMAL));
+ }
+
+ /**
+ * Test the constructor that takes a BigDecimal argument and sets the
+ * literal type to the "natural" type (the smallest type that can hold
+ * the value.)
+ */
+ @Test
+ public void testSimpleCtor() throws SqlCastException {
+ // Spot check. Assumes uses inferType() tested above.
+ NumericLiteral n = new NumericLiteral(BigDecimal.ZERO);
+ assertEquals(0, n.getLongValue());
+ assertEquals(Type.TINYINT, n.getType());
+
+ n = new NumericLiteral(NumericLiteral.MAX_TINYINT);
+ assertEquals(Byte.MAX_VALUE, n.getLongValue());
+ assertEquals(Type.TINYINT, n.getType());
+
+ n = new NumericLiteral(NumericLiteral.MAX_BIGINT);
+ assertEquals(Type.BIGINT, n.getType());
+ assertEquals(Long.MAX_VALUE, n.getLongValue());
+
+ n = new NumericLiteral(NumericLiteral.MAX_DOUBLE);
+ assertEquals(Double.MAX_VALUE, n.getDoubleValue(), 1.0);
+ assertEquals(Type.DOUBLE, n.getType());
+
+ n = new NumericLiteral(new BigDecimal(genDecimal(35, 0)));
+ assertEquals(ScalarType.createDecimalType(35, 0), n.getType());
+
+ try {
+ new NumericLiteral(NumericLiteral.MAX_DOUBLE.multiply(BigDecimal.TEN));
+ fail();
+ } catch(SqlCastException e) {
+ // Expected
+ }
+ }
+
+ @Test
+ public void testTypeCtor() throws InvalidValueException, SqlCastException {
+ NumericLiteral n = new NumericLiteral(BigDecimal.ZERO);
+ assertEquals(0, n.getLongValue());
+ assertEquals(Type.TINYINT, n.getType());
+
+ n = new NumericLiteral(NumericLiteral.MAX_TINYINT);
+ assertEquals(Byte.MAX_VALUE, n.getLongValue());
+ assertEquals(Type.TINYINT, n.getType());
+
+ n = new NumericLiteral(NumericLiteral.MAX_BIGINT);
+ assertEquals(Type.BIGINT, n.getType());
+ assertEquals(Long.MAX_VALUE, n.getLongValue());
+
+ n = new NumericLiteral(NumericLiteral.MAX_DOUBLE);
+ assertEquals(Double.MAX_VALUE, n.getDoubleValue(), 1.0);
+ assertEquals(Type.DOUBLE, n.getType());
+
+ n = new NumericLiteral(new BigDecimal(genDecimal(35, 0)));
+ assertEquals(ScalarType.createDecimalType(35, 0), n.getType());
+
+ try {
+ new NumericLiteral(NumericLiteral.MAX_DOUBLE.multiply(BigDecimal.TEN));
+ fail();
+ } catch(SqlCastException e) {
+ // Expected
+ }
+ try {
+ new NumericLiteral(new BigDecimal("123.45"),
+ ScalarType.createDecimalType(3, 1));
+ fail();
+ } catch(SqlCastException e) {
+ // Expected
+ }
+ try {
+ new NumericLiteral(new BigDecimal(Integer.MAX_VALUE),
+ Type.TINYINT);
+ fail();
+ } catch(SqlCastException e) {
+ // Expected
+ }
+
+ n = new NumericLiteral(new BigDecimal("1.567"), ScalarType.createDecimalType(2, 1));
+ assertEquals(ScalarType.createDecimalType(2, 1), n.getType());
+ assertEquals("1.6", n.getValue().toString());
+ }
+
+ @Test
+ public void testExtremes() throws InvalidValueException, SqlCastException {
+ NumericLiteral n = new NumericLiteral(NumericLiteral.MAX_DOUBLE);
+ assertEquals(Double.MAX_VALUE, n.getDoubleValue(), 1);
+ n = new NumericLiteral(NumericLiteral.MIN_DOUBLE);
+ assertEquals(-Double.MAX_VALUE, n.getDoubleValue(), 1);
+ }
+
+ @Test
+ public void testCastTo() throws AnalysisException {
+ {
+ // Integral types
+ NumericLiteral n = new NumericLiteral(BigDecimal.ZERO);
+ Expr result = n.uncheckedCastTo(Type.BIGINT);
+ assertSame(n, result);
+ assertEquals(Type.BIGINT, n.getType());
+ result = n.uncheckedCastTo(Type.TINYINT);
+ assertSame(n, result);
+ assertEquals(Type.TINYINT, n.getType());
+ result = n.uncheckedCastTo(ScalarType.createDecimalType(5, 0));
+ assertSame(n, result);
+ assertEquals(ScalarType.createDecimalType(5, 0), n.getType());
+ }
+ {
+ // Integral types, with overflow
+ NumericLiteral n = new NumericLiteral(ABOVE_SMALLINT);
+ Expr result = n.uncheckedCastTo(Type.BIGINT);
+ assertSame(n, result);
+ assertEquals(Type.BIGINT, n.getType());
+ Expr result2 = n.uncheckedCastTo(Type.SMALLINT);
+ assertTrue(result2 instanceof CastExpr);
+ assertEquals(Type.SMALLINT, result2.getType());
+ }
+ {
+ // Decimal types, with overflow
+ // Note: not safe to reuse above value after exception
+ NumericLiteral n = new NumericLiteral(ABOVE_SMALLINT);
+ Expr result = n.uncheckedCastTo(Type.BIGINT);
+ assertSame(n, result);
+ assertEquals(Type.BIGINT, n.getType());
+ Expr result2 = n.uncheckedCastTo(ScalarType.createDecimalType(2, 0));
+ assertTrue(result2 instanceof CastExpr);
+ assertEquals(ScalarType.createDecimalType(2, 0), result2.getType());
+ }
+ {
+ // Decimal types
+ NumericLiteral n = new NumericLiteral(new BigDecimal("123.45"));
+ assertEquals(ScalarType.createDecimalType(5, 2), n.getType());
+ Expr result = n.uncheckedCastTo(ScalarType.createDecimalType(6, 3));
+ assertSame(n, result);
+ assertEquals(ScalarType.createDecimalType(6, 3), n.getType());
+ result = n.uncheckedCastTo(ScalarType.createDecimalType(5, 2));
+ assertSame(n, result);
+ assertEquals(ScalarType.createDecimalType(5, 2), n.getType());
+ result = n.uncheckedCastTo(ScalarType.createDecimalType(4, 1));
+ assertNotSame(n, result);
+ assertEquals(ScalarType.createDecimalType(4, 1), result.getType());
+ assertEquals("123.5", ((NumericLiteral) result).toSql());
+ }
+ }
+
+ /**
+ * Test the swap() sign method used by the parser which recognizes
+ * numbers as:
+ *
+ * 12345 --> number
+ * - number --> swap sign
+ *
+ * Note that swap sign can be applied multiple times:
+ *
+ * - -1234 --> number, swap sign, swap sign
+ *
+ * Swapping sign is not as simple as it might seem. For integers,
+ * the positive range is smaller than the negative range, so:
+ *
+ * 256 --> SMALLINT
+ * -256 --> TINYINT
+ *
+ * This means that 256 starts as a SMALLINT, but drops one
+ * size to TINYINT when it becomes negative. And, if negated again, it jumps
+ * up one size to again become SMALLINT.
+ */
+ @Test
+ public void testSwapSign() {
+ {
+ // TINYINT size promotion
+ int absValue = -(int) Byte.MIN_VALUE;
+ NumericLiteral n = NumericLiteral.create(absValue);
+ assertEquals(Type.SMALLINT, n.getType());
+ assertEquals(Type.SMALLINT, n.getExplicitType());
+ assertEquals(absValue, n.getIntValue());
+ n.swapSign();
+ assertEquals(Type.TINYINT, n.getType());
+ assertEquals(Type.TINYINT, n.getExplicitType());
+ assertEquals(-absValue, n.getIntValue());
+ n.swapSign();
+ assertEquals(Type.SMALLINT, n.getType());
+ assertEquals(Type.SMALLINT, n.getExplicitType());
+ assertEquals(absValue, n.getIntValue());
+ }
+ {
+ // Max BIGINT promotion: can't become another, larger
+ // integer, so must become a DECIMAL.
+ BigDecimal absValue = NumericLiteral.MIN_BIGINT.negate();
+ NumericLiteral n = NumericLiteral.create(absValue);
+ Type posType = ScalarType.createDecimalType(19);
+ assertEquals(posType, n.getType());
+ assertEquals(posType, n.getExplicitType());
+ assertTrue(absValue.compareTo(n.getValue()) == 0);
+ n.swapSign();
+ assertEquals(Type.BIGINT, n.getType());
+ assertEquals(Type.BIGINT, n.getExplicitType());
+ assertEquals(Long.MIN_VALUE, n.getLongValue());
+ n.swapSign();
+ assertEquals(posType, n.getType());
+ assertEquals(posType, n.getExplicitType());
+ assertTrue(absValue.compareTo(n.getValue()) == 0);
+ }
+ }
+
+ /**
+ * Test of the major cases for convertValue(). Details of overflow
+ * detection are tested above.
+ */
+ @Test
+ public void testConvertValue() throws SqlCastException {
+ BigDecimal result = NumericLiteral.convertValue(BigDecimal.ZERO, Type.TINYINT);
+ assertSame(result, BigDecimal.ZERO);
+ result = NumericLiteral.convertValue(BigDecimal.ZERO, Type.DOUBLE);
+ assertSame(result, BigDecimal.ZERO);
+ result = NumericLiteral.convertValue(BigDecimal.ZERO,
+ ScalarType.createDecimalType(2, 2));
+ assertSame(result, BigDecimal.ZERO);
+
+ // Overflow case
+ try {
+ NumericLiteral.convertValue(ABOVE_TINYINT, Type.TINYINT);
+ fail();
+ } catch(SqlCastException e) {
+ // Expected
+ }
+
+ // Round to integer
+ result = NumericLiteral.convertValue(
+ new BigDecimal("1234.56"), Type.INT);
+ assertEquals("1235", result.toString());
+
+ // Round to decimal precision
+ BigDecimal input = new BigDecimal("1234.56789");
+ result = NumericLiteral.convertValue(
+ input, ScalarType.createDecimalType(7, 3));
+ assertEquals("1234.568", result.toString());
+ result = NumericLiteral.convertValue(
+ input, ScalarType.createDecimalType(4, 0));
+ assertEquals("1235", result.toString());
+
+ // Decimal overflow
+ try {
+ NumericLiteral.convertValue(
+ new BigDecimal("1234.56789"), ScalarType.createDecimalType(3, 2));
+ fail();
+ } catch(SqlCastException e) {
+ // Expected
+ }
+
+ // Reuse value as decimal
+ input = new BigDecimal("1235.56");
+ result = NumericLiteral.convertValue(input,
+ ScalarType.createDecimalType(6, 2));
+ assertSame(input, result);
+ input = new BigDecimal("0.01");
+ result = NumericLiteral.convertValue(input,
+ ScalarType.createDecimalType(2, 2));
+ assertSame(input, result);
+ }
+
+ @Test
+ public void testCast() throws SqlCastException {
+ NumericLiteral n = NumericLiteral.create(1000);
+ assertEquals(Type.SMALLINT, n.getType());
+ Expr result = n.uncheckedCastTo(Type.TINYINT);
+ assertTrue(result instanceof CastExpr);
+ assertEquals(Type.TINYINT, result.getType());
+
+ result = n.uncheckedCastTo(Type.INT);
+ assertSame(n, result);
+ assertEquals(Type.INT, n.getType());
+
+ n = new NumericLiteral(new BigDecimal("123.45"));
+ assertEquals(ScalarType.createDecimalType(5, 2), n.getType());
+ assertSame(n, n.uncheckedCastTo(ScalarType.createDecimalType(6, 3)));
+ assertEquals(ScalarType.createDecimalType(6, 3), n.getType());
+
+ n = new NumericLiteral(new BigDecimal("123.45"));
+ Type newType = ScalarType.createDecimalType(4, 1);
+ result = n.uncheckedCastTo(newType);
+ assertNotSame(result, n);
+ assertTrue(result instanceof NumericLiteral);
+ assertEquals(newType, result.getType());
+ NumericLiteral n2 = (NumericLiteral) result;
+ assertEquals("123.5", n2.getValue().toString());
+
+ Expr result2 = n2.uncheckedCastTo(Type.SMALLINT);
+ assertNotSame(result2, result);
+ assertEquals(Type.SMALLINT, result2.getType());
+ assertEquals("124", ((NumericLiteral)result2).getValue().toString());
+ }
+
+ @Test
+ public void testEquality() throws SqlCastException {
+ NumericLiteral n1 = NumericLiteral.create(10);
+ NumericLiteral n2 = NumericLiteral.create(10);
+ NumericLiteral n3 = NumericLiteral.create(10, Type.INT);
+ NumericLiteral n4 = NumericLiteral.create(11);
+ NumericLiteral n5 = NumericLiteral.create(
+ new BigDecimal("10.000"), Type.TINYINT);
+
+ // Same object
+ assertTrue(n1.localEquals(n1));
+ // Types and values match
+ assertTrue(n1.localEquals(n2));
+ // Types differ, values same
+ assertFalse(n1.localEquals(n3));
+ // Types same, values differ
+ assertFalse(n1.localEquals(n4));
+ // Types same, values are considered the same
+ // (Though BigDecimal.equals() considers the values different
+ // due to different precisions.)
+ assertTrue(n1.localEquals(n5));
+ }
+}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/analysis/ParserTest.java b/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
index ad5d111..62c74c5 100644
--- a/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
+++ b/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
@@ -25,7 +25,6 @@ import java.math.BigInteger;
import java.util.ArrayList;
import java.util.List;
-import org.apache.impala.analysis.Parser.ParseException;
import org.apache.impala.analysis.TimestampArithmeticExpr.TimeUnit;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.FrontendTestBase;
@@ -69,7 +68,7 @@ public class ParserTest extends FrontendTestBase {
StatementBase result = null; // Save this object to make debugging easier
try {
result = Parser.parse(stmt);
- } catch (ParseException e) {
+ } catch (AnalysisException e) {
if (expectedErrorString != null) {
String errorString = e.getMessage();
StringBuilder message = new StringBuilder();
@@ -1062,8 +1061,8 @@ public class ParserTest extends FrontendTestBase {
ParsesOk(String.format("select -%s", Double.toString(Double.MAX_VALUE)));
// Test overflow and underflow
- ParsesOk(String.format("select %s1", Double.toString(Double.MIN_VALUE)));
- ParsesOk(String.format("select %s1", Double.toString(Double.MAX_VALUE)));
+ ParserError(String.format("select %s1", Double.toString(Double.MIN_VALUE)));
+ ParserError(String.format("select %s1", Double.toString(Double.MAX_VALUE)));
}
@Test
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java b/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
index bf182d6..114e289 100644
--- a/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java
@@ -25,6 +25,7 @@ import java.util.Map;
import org.apache.impala.analysis.LiteralExpr;
import org.apache.impala.common.AnalysisException;
import org.apache.impala.common.ImpalaException;
+import org.apache.impala.common.SqlCastException;
import org.apache.impala.testutil.CatalogServiceTestCatalog;
import org.apache.impala.thrift.CatalogObjectsConstants;
import org.apache.impala.thrift.TAccessLevel;
@@ -224,14 +225,15 @@ public class CatalogObjectToFromThriftTest {
Assert.assertNotNull(part);;
// Create a dummy partition with an invalid decimal type.
try {
- HdfsPartition dummyPart = new HdfsPartition(hdfsTable, part.toHmsPartition(),
- Lists.newArrayList(LiteralExpr.create("1.1", ScalarType.createDecimalType(1, 0)),
- LiteralExpr.create("1.1", ScalarType.createDecimalType(1, 0))),
+ new HdfsPartition(hdfsTable, part.toHmsPartition(),
+ Lists.newArrayList(LiteralExpr.create("11.1", ScalarType.createDecimalType(1, 0)),
+ LiteralExpr.create("11.1", ScalarType.createDecimalType(1, 0))),
null, Lists.<HdfsPartition.FileDescriptor>newArrayList(),
TAccessLevel.READ_WRITE);
fail("Expected metadata to be malformed.");
- } catch (AnalysisException e) {
- Assert.assertTrue(e.getMessage().contains("invalid DECIMAL(1,0) value: 1.1"));
+ } catch (SqlCastException e) {
+ Assert.assertTrue(e.getMessage().contains(
+ "Value 11.1 cannot be cast to type DECIMAL(1,0)"));
}
}
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java b/fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java
index 2e093c1..f40897d 100644
--- a/fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java
@@ -20,7 +20,6 @@ package org.apache.impala.catalog;
import static org.apache.impala.catalog.HdfsPartition.comparePartitionKeyValues;
import static org.junit.Assert.*;
-import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;
@@ -60,18 +59,18 @@ public class HdfsPartitionTest {
public HdfsPartitionTest() {
valuesNull_.add(NullLiteral.create(Type.BIGINT));
- valuesDecimal_.add(new NumericLiteral(BigDecimal.valueOf(1)));
- valuesDecimal1_.add(new NumericLiteral(BigDecimal.valueOf(3)));
- valuesDecimal2_.add(new NumericLiteral(BigDecimal.valueOf(5)));
+ valuesDecimal_.add(NumericLiteral.create(1));
+ valuesDecimal1_.add(NumericLiteral.create(3));
+ valuesDecimal2_.add(NumericLiteral.create(5));
- valuesMixed_.add(new NumericLiteral(BigDecimal.valueOf(3)));
+ valuesMixed_.add(NumericLiteral.create(3));
valuesMixed_.add(NullLiteral.create(Type.BIGINT));
- valuesMixed1_.add(new NumericLiteral(BigDecimal.valueOf(1)));
+ valuesMixed1_.add(NumericLiteral.create(1));
valuesMixed1_.add(NullLiteral.create(Type.STRING));
valuesMixed1_.add(new BoolLiteral(true));
- valuesMixed2_.add(new NumericLiteral(BigDecimal.valueOf(1)));
+ valuesMixed2_.add(NumericLiteral.create(1));
valuesMixed2_.add(new StringLiteral("Large"));
valuesMixed2_.add(new BoolLiteral(false));
}
@@ -98,7 +97,7 @@ public class HdfsPartitionTest {
}
List<LiteralExpr> valuesTest = Lists.newArrayList();
- valuesTest.add(new NumericLiteral(BigDecimal.valueOf(3)));
+ valuesTest.add(NumericLiteral.create(3));
verifyAntiSymmetric(valuesDecimal1_, valuesTest, valuesNull_);
valuesTest.add(NullLiteral.create(Type.BIGINT));
verifyAntiSymmetric(valuesMixed_, valuesTest, valuesDecimal_);
http://git-wip-us.apache.org/repos/asf/impala/blob/27577dd6/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
----------------------------------------------------------------------
diff --git a/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java b/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
index c3d5350..de4266b 100644
--- a/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
+++ b/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
@@ -20,7 +20,6 @@ package org.apache.impala.common;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.fail;
-import java.io.StringReader;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
@@ -35,10 +34,7 @@ import org.apache.impala.analysis.FunctionName;
import org.apache.impala.analysis.InsertStmt;
import org.apache.impala.analysis.ParseNode;
import org.apache.impala.analysis.Parser;
-import org.apache.impala.analysis.Parser.ParseException;
import org.apache.impala.analysis.QueryStmt;
-import org.apache.impala.analysis.SqlParser;
-import org.apache.impala.analysis.SqlScanner;
import org.apache.impala.analysis.StatementBase;
import org.apache.impala.analysis.StmtMetadataLoader;
import org.apache.impala.analysis.StmtMetadataLoader.StmtTableCache;
@@ -280,7 +276,7 @@ public class FrontendTestBase {
StatementBase node = Parser.parse(stmt);
assertNotNull(node);
return node;
- } catch (ParseException e) {
+ } catch (AnalysisException e) {
fail("\nParser error:\n" + e.getMessage());
throw new IllegalStateException(); // Keep compiler happy
}