You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@calcite.apache.org by "mihaibudiu (via GitHub)" <gi...@apache.org> on 2023/12/21 23:20:54 UTC
[PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
mihaibudiu opened a new pull request, #3589:
URL: https://github.com/apache/calcite/pull/3589
I initially started with the goal of fixing just [CALCITE-3522], https://issues.apache.org/jira/browse/CALCITE-3522, which is limiting the size of decimal literals to 64 bits. The fix is to actually check the precision of the type system when rejecting a decimal literal. That's easy. I am not checking the scale, under the assumption that rounding is OK for literals but losing precision is not.
However, fixing this problem unmasked a bunch of other bugs in handling large values, so in order to make the tests pass I had to also fix [CALCITE-6169], which is about incorrect casts between numeric values. There is still one case which is not handled, which is casts between decimal values that change the scale of the result, but I plan to do that in a separate PR (because I also have to think about it more.)
You will notice that this fix re-enables some tests that were disabled a long time ago. I think that disabling failing tests is not a good practice, and should be avoided. There are still some disabled tests in these categories, but strictly fewer than before.
I couldn't make this into 2 separate PRs that both keep all tests running.
I am not 100% happy with the code, in particular with the implementation of the new Expression `ConvertChecked`. The implementation (which is in the unparse method...) in fact just calls the existing code in `Primitive` which handles numeric casts. I think that's better than duplicating the code. There is already a very large number of places where these conversions are handled, which is shown by the many files that I had to patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-1969568419
@libenchao I have added some review comments to this PR, if you can help with a review we can hopefully merge this. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-1869784859
Thank you @tanclary for the review, I pushed a new commit which hopefully addresses your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] SqlValidator.validateLiteral rejects literals with a DECIMAL type that require more than 64 bits [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu merged PR #3589:
URL: https://github.com/apache/calcite/pull/3589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2243770596
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "tanclary (via GitHub)" <gi...@apache.org>.
tanclary commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1435829039
##########
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -3334,20 +3333,11 @@ private void registerOperandSubQueries(
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
- // Decimal and long have the same precision (as 64-bit integers), so
- // the unscaled value of a decimal must fit into a long.
-
- // REVIEW jvs 4-Aug-2004: This should probably be calling over to
- // the available calculator implementations to see what they
- // support. For now use ESP instead.
- //
- // jhyde 2006/12/21: I think the limits should be baked into the
- // type system, not dependent on the calculator implementation.
+ final RelDataTypeSystem typeSystem = getTypeFactory().getTypeSystem();
+ final int precision = typeSystem.getMaxNumericPrecision();
BigDecimal bd = literal.getValueAs(BigDecimal.class);
- BigInteger unscaled = bd.unscaledValue();
- long longValue = unscaled.longValue();
- if (!BigInteger.valueOf(longValue).equals(unscaled)) {
- // overflow
+ BigDecimal simpler = bd.stripTrailingZeros();
Review Comment:
Some of the naming here confuses me, what does `simpler` mean? (in this context, haha).
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/EnumUtils.java:
##########
@@ -406,11 +407,28 @@ public static Expression convert(Expression operand, Type fromType,
if (toPrimitive != null) {
if (fromPrimitive != null) {
// E.g. from "float" to "double"
- return Expressions.convert_(
+ if (toPrimitive == Primitive.BOOLEAN) {
+ return Expressions.convert_(
+ operand, toPrimitive.getPrimitiveClass());
+ }
+ return Expressions.convertChecked(
operand, toPrimitive.getPrimitiveClass());
}
- if (fromNumber || fromBox == Primitive.CHAR) {
- // Generate "x.shortValue()".
+ if (fromType == BigDecimal.class
+ && (toPrimitive == Primitive.INT
Review Comment:
could add a method to the `Primitive` class similar to `isNumeric()` in case another check like this is needed in the future, also reduces `if` complexity.
##########
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -3334,20 +3333,11 @@ private void registerOperandSubQueries(
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
- // Decimal and long have the same precision (as 64-bit integers), so
- // the unscaled value of a decimal must fit into a long.
-
Review Comment:
is there a better comment to leave in its place or no?>
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/EnumUtils.java:
##########
@@ -406,11 +407,28 @@ public static Expression convert(Expression operand, Type fromType,
if (toPrimitive != null) {
if (fromPrimitive != null) {
// E.g. from "float" to "double"
- return Expressions.convert_(
+ if (toPrimitive == Primitive.BOOLEAN) {
Review Comment:
nit: third level of nested `if` might be a bit much, no worries if refactoring isn't simple/easy.
##########
linq4j/src/main/java/org/apache/calcite/linq4j/tree/UnaryExpression.java:
##########
@@ -51,6 +51,24 @@ public class UnaryExpression extends Expression {
expression.accept(writer, nodeType.rprec, rprec);
}
return;
+ case ConvertChecked:
+ // This is ugly, but Java does not seem to have any facilities
+ // to perform checked cast between scalar types!
+ // So we use the existing linq4j Primitive.numberValue method
+ // which does overflow checking.
+ if (!writer.requireParentheses(this, lprec, rprec)) {
+ // Generate e.g.,
+ // ((Number)org.apache.calcite.linq4j.tree.Primitive.of(int.class)
+ // .numberValue(literal_value)).intValue();
+ writer.append("((Number)")
Review Comment:
Unfortunate you had to write this yourself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2243732568
I have reworked this PR since it was split into two separate PRs, one of which has just been merged #3764. Now it's much simpler.
This will hopefully allow us to close a 5-year old issue in Calcite.
Incidentally it also seems to fix [CALCITE-2539] which I plan to resolve if this is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "jduo (via GitHub)" <gi...@apache.org>.
jduo commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1528985794
##########
core/src/main/java/org/apache/calcite/rex/RexBuilder.java:
##########
@@ -699,22 +699,23 @@ boolean canRemoveCastFromLiteral(RelDataType toType, @Nullable Comparable value,
if (SqlTypeName.INT_TYPES.contains(sqlType)) {
final BigDecimal decimalValue = (BigDecimal) value;
- final int s = decimalValue.scale();
- if (s != 0) {
+ try {
+ // Will throw if the value does not fit in a long
+ long l = decimalValue.longValueExact();
Review Comment:
Should these checks factor in unsigned types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2253243570
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1696653862
##########
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -3334,20 +3333,14 @@ private void registerOperandSubQueries(
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
- // Decimal and long have the same precision (as 64-bit integers), so
- // the unscaled value of a decimal must fit into a long.
-
- // REVIEW jvs 4-Aug-2004: This should probably be calling over to
- // the available calculator implementations to see what they
- // support. For now use ESP instead.
- //
- // jhyde 2006/12/21: I think the limits should be baked into the
- // type system, not dependent on the calculator implementation.
- BigDecimal bd = literal.getValueAs(BigDecimal.class);
- BigInteger unscaled = bd.unscaledValue();
- long longValue = unscaled.longValue();
- if (!BigInteger.valueOf(longValue).equals(unscaled)) {
- // overflow
+ // Accept any decimal value that does not exceed the max
+ // precision of the type system.
+ final RelDataTypeSystem typeSystem = getTypeFactory().getTypeSystem();
+ final int maxPrecision = typeSystem.getMaxNumericPrecision();
+ final BigDecimal bd = literal.getValueAs(BigDecimal.class);
+ final BigDecimal noTrailingZeros = bd.stripTrailingZeros();
+ // If we don't strip trailing zeros we may reject values such as 1.000....0.
+ if (noTrailingZeros.precision() > maxPrecision) {
Review Comment:
Thanks @mihaibudiu, can we add a negative test for it so we make sure it works as intended and it will prevent regressions? After that for me it's good to be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1688024263
##########
core/src/main/java/org/apache/calcite/sql/SqlNumericLiteral.java:
##########
@@ -103,7 +103,7 @@ public boolean isExact() {
BigDecimal bd = getValueNonNull();
SqlTypeName result;
// Will throw if the number cannot be represented as a long.
- long l = bd.longValue();
+ long l = bd.longValueExact();
Review Comment:
As per the [documentation](https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html#longValueExact--), this will throw an `ArithmeticException` in case of overflow, should we catch it and fallback do decimal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1693416372
##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -626,16 +644,15 @@ void testCastExactNumericLimits(CastType castType, SqlOperatorFixture f) {
// Overflow test
if (numeric == Numeric.BIGINT) {
- // Calcite cannot even represent a literal so large, so
- // for this query even the safe casts fail at compile-time
- // (runtime == false).
- f.checkFails("cast(^" + numeric.maxOverflowNumericString + "^ as BIGINT)",
- LITERAL_OUT_OF_RANGE_MESSAGE, false);
- f.checkFails("cast(^" + numeric.minOverflowNumericString + "^ as BIGINT)",
- LITERAL_OUT_OF_RANGE_MESSAGE, false);
+ // Overflow for casting decimals produces a different error
+ f.checkCastFails(numeric.maxOverflowNumericString,
+ type, "Overflow", true, castType);
+ f.checkCastFails(numeric.minOverflowNumericString,
+ type, "Overflow", true, castType);
} else {
if (numeric != Numeric.DECIMAL5_2) {
// This condition is for bug [CALCITE-6078], not yet fixed
+ // FIXME: decimal casts are still incorrect.
Review Comment:
this is actually fixed as part of another PR which I have had in the queue for a long time. I will remove this comment.
##########
core/src/main/java/org/apache/calcite/sql/SqlNumericLiteral.java:
##########
@@ -103,7 +103,7 @@ public boolean isExact() {
BigDecimal bd = getValueNonNull();
SqlTypeName result;
// Will throw if the number cannot be represented as a long.
- long l = bd.longValue();
+ long l = bd.longValueExact();
Review Comment:
yes, that is exactly what is going on here. This is within a try-catch block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2253232305
@asolimando I hope I have answered your questions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2261823926
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [96.2% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-1911208876
If anyone wants to do a pair review we can arrange a short meeting so I can explain what's going on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1699265587
##########
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -3334,20 +3333,14 @@ private void registerOperandSubQueries(
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
- // Decimal and long have the same precision (as 64-bit integers), so
- // the unscaled value of a decimal must fit into a long.
-
- // REVIEW jvs 4-Aug-2004: This should probably be calling over to
- // the available calculator implementations to see what they
- // support. For now use ESP instead.
- //
- // jhyde 2006/12/21: I think the limits should be baked into the
- // type system, not dependent on the calculator implementation.
- BigDecimal bd = literal.getValueAs(BigDecimal.class);
- BigInteger unscaled = bd.unscaledValue();
- long longValue = unscaled.longValue();
- if (!BigInteger.valueOf(longValue).equals(unscaled)) {
- // overflow
+ // Accept any decimal value that does not exceed the max
+ // precision of the type system.
+ final RelDataTypeSystem typeSystem = getTypeFactory().getTypeSystem();
+ final int maxPrecision = typeSystem.getMaxNumericPrecision();
+ final BigDecimal bd = literal.getValueAs(BigDecimal.class);
+ final BigDecimal noTrailingZeros = bd.stripTrailingZeros();
+ // If we don't strip trailing zeros we may reject values such as 1.000....0.
+ if (noTrailingZeros.precision() > maxPrecision) {
Review Comment:
I should have done this from the beginning, thank you for reminding me. I added a tiny commit with negative tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1694762786
##########
core/src/main/java/org/apache/calcite/sql/SqlNumericLiteral.java:
##########
@@ -103,7 +103,7 @@ public boolean isExact() {
BigDecimal bd = getValueNonNull();
SqlTypeName result;
// Will throw if the number cannot be represented as a long.
- long l = bd.longValue();
+ long l = bd.longValueExact();
Review Comment:
Sorry, I didn't notice that part of the code below was "collapsed" in the UI, thanks for the clarification
##########
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -3334,20 +3333,14 @@ private void registerOperandSubQueries(
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
- // Decimal and long have the same precision (as 64-bit integers), so
- // the unscaled value of a decimal must fit into a long.
-
- // REVIEW jvs 4-Aug-2004: This should probably be calling over to
- // the available calculator implementations to see what they
- // support. For now use ESP instead.
- //
- // jhyde 2006/12/21: I think the limits should be baked into the
- // type system, not dependent on the calculator implementation.
- BigDecimal bd = literal.getValueAs(BigDecimal.class);
- BigInteger unscaled = bd.unscaledValue();
- long longValue = unscaled.longValue();
- if (!BigInteger.valueOf(longValue).equals(unscaled)) {
- // overflow
+ // Accept any decimal value that does not exceed the max
+ // precision of the type system.
+ final RelDataTypeSystem typeSystem = getTypeFactory().getTypeSystem();
+ final int maxPrecision = typeSystem.getMaxNumericPrecision();
+ final BigDecimal bd = literal.getValueAs(BigDecimal.class);
+ final BigDecimal noTrailingZeros = bd.stripTrailingZeros();
+ // If we don't strip trailing zeros we may reject values such as 1.000....0.
+ if (noTrailingZeros.precision() > maxPrecision) {
Review Comment:
I think we should, I can't think of any reason why we should not, do you have anything in mind?
To this extend, we could use
[BigDecimal::scale()](https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html#scale--) to check compatibility with what the type system supports.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2257050139
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [97.6% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1695828675
##########
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -3334,20 +3333,14 @@ private void registerOperandSubQueries(
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
- // Decimal and long have the same precision (as 64-bit integers), so
- // the unscaled value of a decimal must fit into a long.
-
- // REVIEW jvs 4-Aug-2004: This should probably be calling over to
- // the available calculator implementations to see what they
- // support. For now use ESP instead.
- //
- // jhyde 2006/12/21: I think the limits should be baked into the
- // type system, not dependent on the calculator implementation.
- BigDecimal bd = literal.getValueAs(BigDecimal.class);
- BigInteger unscaled = bd.unscaledValue();
- long longValue = unscaled.longValue();
- if (!BigInteger.valueOf(longValue).equals(unscaled)) {
- // overflow
+ // Accept any decimal value that does not exceed the max
+ // precision of the type system.
+ final RelDataTypeSystem typeSystem = getTypeFactory().getTypeSystem();
+ final int maxPrecision = typeSystem.getMaxNumericPrecision();
+ final BigDecimal bd = literal.getValueAs(BigDecimal.class);
+ final BigDecimal noTrailingZeros = bd.stripTrailingZeros();
+ // If we don't strip trailing zeros we may reject values such as 1.000....0.
+ if (noTrailingZeros.precision() > maxPrecision) {
Review Comment:
I added this check, no tests required changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] SqlValidator.validateLiteral rejects literals with a DECIMAL type that require more than 64 bits [calcite]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2263687251
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [96.2% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1506378183
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/EnumUtils.java:
##########
@@ -406,11 +407,24 @@ public static Expression convert(Expression operand, Type fromType,
if (toPrimitive != null) {
if (fromPrimitive != null) {
// E.g. from "float" to "double"
- return Expressions.convert_(
+ if (toPrimitive == Primitive.BOOLEAN) {
+ return Expressions.convert_(
+ operand, toPrimitive.getPrimitiveClass());
+ }
+ return Expressions.convertChecked(
operand, toPrimitive.getPrimitiveClass());
}
- if (fromNumber || fromBox == Primitive.CHAR) {
- // Generate "x.shortValue()".
+ if (fromType == BigDecimal.class && toPrimitive.isFixedNumeric()) {
+ // Conversion from decimal to an exact type
+ ConstantExpression zero = Expressions.constant(0);
+ // Elsewhere Calcite uses this rounding mode implicitly, so we have to be consistent.
+ // E.g., this is the rounding mode used by BigDecimal.longValue().
+ Expression rounding = Expressions.constant(RoundingMode.DOWN);
+ // rounded = operand.setScale(0, RoundingMode.DOWN);
+ Expression rounded = Expressions.call(operand, "setScale", zero, rounding);
+ // return rounded.to*ValueExact()
+ return Expressions.unboxExact(rounded, toPrimitive);
Review Comment:
conversion from decimal to some integer type needs rounding, here we use the DOWN rounding, which is implicitly used in Calcite wherever BigDecimal.longValue is used. If a different rounding mode is desired we should file an issue.
##########
core/src/main/java/org/apache/calcite/rex/RexBuilder.java:
##########
@@ -699,22 +699,23 @@ boolean canRemoveCastFromLiteral(RelDataType toType, @Nullable Comparable value,
if (SqlTypeName.INT_TYPES.contains(sqlType)) {
final BigDecimal decimalValue = (BigDecimal) value;
- final int s = decimalValue.scale();
- if (s != 0) {
+ try {
+ // Will throw if the value does not fit in a long
+ long l = decimalValue.longValueExact();
Review Comment:
check if the decimal value will fit in the target type
##########
core/src/main/java/org/apache/calcite/adapter/enumerable/EnumUtils.java:
##########
@@ -406,11 +407,24 @@ public static Expression convert(Expression operand, Type fromType,
if (toPrimitive != null) {
if (fromPrimitive != null) {
// E.g. from "float" to "double"
- return Expressions.convert_(
+ if (toPrimitive == Primitive.BOOLEAN) {
+ return Expressions.convert_(
+ operand, toPrimitive.getPrimitiveClass());
+ }
+ return Expressions.convertChecked(
Review Comment:
convertChecked is implemented in this PR.
##########
core/src/main/java/org/apache/calcite/sql/SqlNumericLiteral.java:
##########
@@ -99,17 +99,23 @@ public boolean isExact() {
if (isExact) {
int scaleValue = requireNonNull(scale, "scale");
if (0 == scaleValue) {
- BigDecimal bd = getValueNonNull();
- SqlTypeName result;
- long l = bd.longValue();
- if ((l >= Integer.MIN_VALUE) && (l <= Integer.MAX_VALUE)) {
- result = SqlTypeName.INTEGER;
- } else {
- result = SqlTypeName.BIGINT;
+ try {
+ BigDecimal bd = getValueNonNull();
Review Comment:
numeric literals are always represented as Decimal by the parser.
Here we check whether the value can be represented by a narrower type without information loss.
##########
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -3334,20 +3333,14 @@ private void registerOperandSubQueries(
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
- // Decimal and long have the same precision (as 64-bit integers), so
- // the unscaled value of a decimal must fit into a long.
-
- // REVIEW jvs 4-Aug-2004: This should probably be calling over to
- // the available calculator implementations to see what they
- // support. For now use ESP instead.
- //
- // jhyde 2006/12/21: I think the limits should be baked into the
- // type system, not dependent on the calculator implementation.
- BigDecimal bd = literal.getValueAs(BigDecimal.class);
- BigInteger unscaled = bd.unscaledValue();
- long longValue = unscaled.longValue();
- if (!BigInteger.valueOf(longValue).equals(unscaled)) {
- // overflow
+ // Accept any decimal value that does not exceed the max
+ // precision of the type system.
+ final RelDataTypeSystem typeSystem = getTypeFactory().getTypeSystem();
+ final int maxPrecision = typeSystem.getMaxNumericPrecision();
+ final BigDecimal bd = literal.getValueAs(BigDecimal.class);
+ final BigDecimal noTrailingZeros = bd.stripTrailingZeros();
+ // If we don't strip trailing zeros we may reject values such as 1.000....0.
+ if (noTrailingZeros.precision() > maxPrecision) {
Review Comment:
this code will accept any literal which fits in the specified precision.
I am not sure whether we have to check the scale too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1528991914
##########
core/src/main/java/org/apache/calcite/rex/RexBuilder.java:
##########
@@ -699,22 +699,23 @@ boolean canRemoveCastFromLiteral(RelDataType toType, @Nullable Comparable value,
if (SqlTypeName.INT_TYPES.contains(sqlType)) {
final BigDecimal decimalValue = (BigDecimal) value;
- final int s = decimalValue.scale();
- if (s != 0) {
+ try {
+ // Will throw if the value does not fit in a long
+ long l = decimalValue.longValueExact();
Review Comment:
There are no unsigned types in SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2263578819
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [96.2% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2065329884
I have submitted https://github.com/apache/calcite/pull/3764, which provides just a fix for [CALCITE-6169].
Once that is merged, I will resubmit a much smaller PR here to fix separately [CALCITE-3522]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits and [CALCITE-6169] EnumUtils.convert does not implement the correct SQL cast semantics [calcite]
Posted by "mihaibudiu (via GitHub)" <gi...@apache.org>.
mihaibudiu commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1506248012
##########
core/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java:
##########
@@ -3334,20 +3333,11 @@ private void registerOperandSubQueries(
@Override public void validateLiteral(SqlLiteral literal) {
switch (literal.getTypeName()) {
case DECIMAL:
- // Decimal and long have the same precision (as 64-bit integers), so
- // the unscaled value of a decimal must fit into a long.
-
Review Comment:
this comment is now obsolete, we are no longer tied to the precision of long
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3589:
URL: https://github.com/apache/calcite/pull/3589#issuecomment-2243761233
## [](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589) **Quality Gate passed**
Issues
 [7 New issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_calcite&pullRequest=3589&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_calcite&pullRequest=3589&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_calcite&pullRequest=3589&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_calcite&pullRequest=3589)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CALCITE-3522] Sql validator limits decimal literals to 64 bits [calcite]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on code in PR #3589:
URL: https://github.com/apache/calcite/pull/3589#discussion_r1688143896
##########
testkit/src/main/java/org/apache/calcite/test/SqlOperatorTest.java:
##########
@@ -626,16 +644,15 @@ void testCastExactNumericLimits(CastType castType, SqlOperatorFixture f) {
// Overflow test
if (numeric == Numeric.BIGINT) {
- // Calcite cannot even represent a literal so large, so
- // for this query even the safe casts fail at compile-time
- // (runtime == false).
- f.checkFails("cast(^" + numeric.maxOverflowNumericString + "^ as BIGINT)",
- LITERAL_OUT_OF_RANGE_MESSAGE, false);
- f.checkFails("cast(^" + numeric.minOverflowNumericString + "^ as BIGINT)",
- LITERAL_OUT_OF_RANGE_MESSAGE, false);
+ // Overflow for casting decimals produces a different error
+ f.checkCastFails(numeric.maxOverflowNumericString,
+ type, "Overflow", true, castType);
+ f.checkCastFails(numeric.minOverflowNumericString,
+ type, "Overflow", true, castType);
} else {
if (numeric != Numeric.DECIMAL5_2) {
// This condition is for bug [CALCITE-6078], not yet fixed
+ // FIXME: decimal casts are still incorrect.
Review Comment:
Nit: Is this comment adding anything more than the existing one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@calcite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org