You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/11 15:10:57 UTC
[GitHub] [spark] wangyum opened a new pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
wangyum opened a new pull request #29731:
URL: https://github.com/apache/spark/pull/29731
### What changes were proposed in this pull request?
This pr makes make decimal fast set to `null` if precision larger that 309(To support Double).
### Why are the changes needed?
It is very slow is this case:

We can reproduce this issue by:
```
// Download https://issues.apache.org/jira/secure/attachment/13011406/part-00000.parquet first
spark.read.parquet("/path/to/part-00000.parquet").selectExpr("cast(bd as decimal(18, 0)) as x").write.mode("overwrite").save("/tmp/spark/decimal")
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691861839
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692744289
**[Test build #128716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128716/testReport)** for PR 29731 at commit [`a98c38d`](https://github.com/apache/spark/commit/a98c38d3a082823f8685d9ee4ec77ad63041052c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487111962
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693220879
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692763618
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693216725
**[Test build #128744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128744/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154999
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693375038
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691815722
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154999
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692763618
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154146
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691862321
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154146
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487712474
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -658,6 +658,41 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
}
+ private[this] def stringToDecimal(str: UTF8String, decimalType: DecimalType): Decimal = {
+ val bigDecimal = try {
+ // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
+ // Please refer to https://github.com/apache/spark/pull/26640
+ new JavaBigDecimal(str.toString.trim)
+ } catch {
+ case _: NumberFormatException =>
+ if (ansiEnabled) {
+ throw new NumberFormatException(s"invalid input syntax for type numeric: $str")
+ } else {
+ null
+ }
+ }
+
+ if (bigDecimal != null) {
+ val precision = if (bigDecimal.scale < 0) {
+ bigDecimal.precision - bigDecimal.scale
+ } else {
+ bigDecimal.precision
+ }
+
+ if (precision > DecimalType.MAX_PRECISION) {
Review comment:
+1, let's explain which part is slow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691861821
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r488673141
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -658,6 +658,41 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
}
+ private[this] def stringToDecimal(str: UTF8String, decimalType: DecimalType): Decimal = {
+ val bigDecimal = try {
+ // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
Review comment:
Filed a ticket: https://issues.apache.org/jira/browse/SPARK-32891
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691291962
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691291099
**[Test build #128575 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128575/testReport)** for PR 29731 at commit [`a74349b`](https://github.com/apache/spark/commit/a74349b64454933946e3e320419648a7ff28ec37).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487711537
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1192,13 +1218,34 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
} else {
s"$evNull =true;"
}
+ val outOfDecimalCode = if (!ansiEnabled) {
+ s"$evNull = true;"
+ } else {
+ s"""throw new ArithmeticException("out of decimal type range: " + $c);"""
+ }
code"""
+ java.math.BigDecimal $tmpBigDecimal = null;
try {
- Decimal $tmp = Decimal.apply(new java.math.BigDecimal($c.toString().trim()));
- ${changePrecision(tmp, target, evPrim, evNull, canNullSafeCast)}
+ $tmpBigDecimal = new java.math.BigDecimal($c.toString().trim());
} catch (java.lang.NumberFormatException e) {
$handleException
}
+
+ if ($tmpBigDecimal != null) {
Review comment:
it's risky to duplicate the code in codegen. How about we create `Decimal.fromString` and `Decimal.fromStringANSI`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154146
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691804981
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693218201
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693182177
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691804981
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r488708553
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -589,6 +587,50 @@ object Decimal {
}
}
+ private def stringToJavaBigDecimal(str: UTF8String): (JavaBigDecimal, Int) = {
+ // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
+ // Please refer to https://github.com/apache/spark/pull/26640
+ val bigDecimal = new JavaBigDecimal(str.toString.trim)
+ val precision = if (bigDecimal.scale < 0) {
+ bigDecimal.precision - bigDecimal.scale
+ } else {
+ bigDecimal.precision
+ }
+ (bigDecimal, precision)
+ }
+
+ def fromString(str: UTF8String): Decimal = {
+ try {
+ val (bigDecimal, precision) = stringToJavaBigDecimal(str)
+ // We fast fail because constructing a very large JavaBigDecimal to Decimal very slow.
+ // For example: Decimal("6.0790316E+25569151")
+ if (precision > DecimalType.MAX_PRECISION) {
+ null
+ } else {
+ Decimal(bigDecimal)
+ }
+ } catch {
+ case _: NumberFormatException =>
+ null
+ }
+ }
+
+ def fromStringANSI(str: UTF8String): Decimal = {
+ try {
+ val (bigDecimal, precision) = stringToJavaBigDecimal(str)
+ // We fast fail because constructing a very large JavaBigDecimal to Decimal very slow.
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691820277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693220302
**[Test build #128750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128750/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154999
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154171
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487154730
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487111962
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154999
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691819843
**[Test build #128617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128617/testReport)** for PR 29731 at commit [`7f54640`](https://github.com/apache/spark/commit/7f546402235485b1efcbebe124a9c93524e26a3a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693217485
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128744/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487154945
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693220879
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487111962
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692909244
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693373688
**[Test build #128750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128750/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487111962
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
`309` to support double. or it is should be `DecimalType.MAX_PRECISION`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691786956
**[Test build #128614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128614/testReport)** for PR 29731 at commit [`d568e11`](https://github.com/apache/spark/commit/d568e111a24fb111b47c3f91253e5d59f1e9ec73).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692908273
**[Test build #128716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128716/testReport)** for PR 29731 at commit [`a98c38d`](https://github.com/apache/spark/commit/a98c38d3a082823f8685d9ee4ec77ad63041052c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r488676063
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1405,4 +1426,35 @@ class AnsiCastSuite extends CastSuiteBase {
checkEvaluation(cast(negativeTs, LongType), expectedSecs)
}
}
+
+ test("Fast fail for cast string type to decimal type in ansi mode") {
+ checkEvaluation(cast("12345678901234567890123456789012345678", DecimalType(38, 0)),
+ Decimal("12345678901234567890123456789012345678"))
Review comment:
Yes. It should throw exception In ANSI mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r489159909
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -646,7 +646,7 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
* NOTE: this modifies `value` in-place, so don't call it on external data.
*/
private[this] def changePrecision(value: Decimal, decimalType: DecimalType): Decimal = {
- if (value.changePrecision(decimalType.precision, decimalType.scale)) {
+ if (value != null && value.changePrecision(decimalType.precision, decimalType.scale)) {
Review comment:
the null check is only needed for non-ansi mode. Shall we put it there?
```
buildCast[UTF8String](_, s => {
val d = Decimal.fromString(s)
if (d == null) null else changePrecision(d, target)
})
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29731:
URL: https://github.com/apache/spark/pull/29731
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691291962
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487709672
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -658,6 +658,41 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
}
+ private[this] def stringToDecimal(str: UTF8String, decimalType: DecimalType): Decimal = {
+ val bigDecimal = try {
+ // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
Review comment:
it sounds like `UTF8String.trim` is not implemented well. We may need to look at how `java.lang.String.trim` is implemented.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693217476
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154171
cc @cloud-fan @Ngone51 @maropu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r488710003
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -589,6 +587,50 @@ object Decimal {
}
}
+ private def stringToJavaBigDecimal(str: UTF8String): (JavaBigDecimal, Int) = {
+ // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
+ // Please refer to https://github.com/apache/spark/pull/26640
+ val bigDecimal = new JavaBigDecimal(str.toString.trim)
+ val precision = if (bigDecimal.scale < 0) {
+ bigDecimal.precision - bigDecimal.scale
+ } else {
+ bigDecimal.precision
+ }
+ (bigDecimal, precision)
Review comment:
This is a performance-critical code path. Can we avoid repeatedly creating Tuple2 objects? We can add a private method to calculate precision.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693220302
**[Test build #128750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128750/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693375038
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692933053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487630070
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
Yes. I have moved this logical to `Cast`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691820277
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691862333
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128616/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693184148
**[Test build #128744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128744/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691861655
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693217476
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487154730
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
Review comment:
I think the max precision is DecimalType.MAX_PRECISION (38)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -143,8 +143,12 @@ final class Decimal extends Ordered[Decimal] with Serializable {
} else if (decimal.scale < 0 && !SQLConf.get.allowNegativeScaleOfDecimalEnabled) {
this._precision = decimal.precision - decimal.scale
this._scale = 0
- // set scale to 0 to correct unscaled value
- this.decimalVal = decimal.setScale(0)
+ if (this._precision > 309) {
+ this.decimalVal = null
Review comment:
I think there should be exception in ANSI mode
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692932012
**[Test build #128718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128718/testReport)** for PR 29731 at commit [`94555ef`](https://github.com/apache/spark/commit/94555ef48fae5dd8baa4caba565fbbdedb4a6a00).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692762770
**[Test build #128718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128718/testReport)** for PR 29731 at commit [`94555ef`](https://github.com/apache/spark/commit/94555ef48fae5dd8baa4caba565fbbdedb4a6a00).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692745315
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154146
**[Test build #128575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128575/testReport)** for PR 29731 at commit [`a74349b`](https://github.com/apache/spark/commit/a74349b64454933946e3e320419648a7ff28ec37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691815267
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154146
**[Test build #128575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128575/testReport)** for PR 29731 at commit [`a74349b`](https://github.com/apache/spark/commit/a74349b64454933946e3e320419648a7ff28ec37).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691787270
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692909244
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692933053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693182177
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154999
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154171
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154999
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154146
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691786956
**[Test build #128614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128614/testReport)** for PR 29731 at commit [`d568e11`](https://github.com/apache/spark/commit/d568e111a24fb111b47c3f91253e5d59f1e9ec73).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r488676798
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -670,18 +705,7 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
private[this] def castToDecimal(from: DataType, target: DecimalType): Any => Any = from match {
case StringType =>
- buildCast[UTF8String](_, s => try {
- // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
- // Please refer to https://github.com/apache/spark/pull/26640
- changePrecision(Decimal(new JavaBigDecimal(s.toString.trim)), target)
- } catch {
- case _: NumberFormatException =>
- if (ansiEnabled) {
- throw new NumberFormatException(s"invalid input syntax for type numeric: $s")
- } else {
- null
- }
- })
+ buildCast[UTF8String](_, s => stringToDecimal(s, target))
Review comment:

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693430988
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691861821
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692762770
**[Test build #128718 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128718/testReport)** for PR 29731 at commit [`94555ef`](https://github.com/apache/spark/commit/94555ef48fae5dd8baa4caba565fbbdedb4a6a00).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r488707989
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
##########
@@ -589,6 +587,50 @@ object Decimal {
}
}
+ private def stringToJavaBigDecimal(str: UTF8String): (JavaBigDecimal, Int) = {
+ // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
+ // Please refer to https://github.com/apache/spark/pull/26640
+ val bigDecimal = new JavaBigDecimal(str.toString.trim)
+ val precision = if (bigDecimal.scale < 0) {
+ bigDecimal.precision - bigDecimal.scale
+ } else {
+ bigDecimal.precision
+ }
+ (bigDecimal, precision)
+ }
+
+ def fromString(str: UTF8String): Decimal = {
+ try {
+ val (bigDecimal, precision) = stringToJavaBigDecimal(str)
+ // We fast fail because constructing a very large JavaBigDecimal to Decimal very slow.
Review comment:
nit: `very slow` => `is very slow`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-693184148
**[Test build #128744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128744/testReport)** for PR 29731 at commit [`abf2f7e`](https://github.com/apache/spark/commit/abf2f7e45b60f15742bf04c47ad1ff6d0ab130dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691787270
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691815267
**[Test build #128616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128616/testReport)** for PR 29731 at commit [`3f2e2a5`](https://github.com/apache/spark/commit/3f2e2a5a97774734170ca558037962e0bdc07f1b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691154146
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691815722
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29731:
URL: https://github.com/apache/spark/pull/29731#discussion_r487696800
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -658,6 +658,41 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
}
+ private[this] def stringToDecimal(str: UTF8String, decimalType: DecimalType): Decimal = {
+ val bigDecimal = try {
+ // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
+ // Please refer to https://github.com/apache/spark/pull/26640
+ new JavaBigDecimal(str.toString.trim)
+ } catch {
+ case _: NumberFormatException =>
+ if (ansiEnabled) {
+ throw new NumberFormatException(s"invalid input syntax for type numeric: $str")
+ } else {
+ null
+ }
+ }
+
+ if (bigDecimal != null) {
+ val precision = if (bigDecimal.scale < 0) {
Review comment:
The scale can be negative even when converting it from a string? Even if yes, this behaivour depends on `LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED`.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -658,6 +658,41 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
}
+ private[this] def stringToDecimal(str: UTF8String, decimalType: DecimalType): Decimal = {
+ val bigDecimal = try {
+ // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
+ // Please refer to https://github.com/apache/spark/pull/26640
+ new JavaBigDecimal(str.toString.trim)
+ } catch {
+ case _: NumberFormatException =>
+ if (ansiEnabled) {
+ throw new NumberFormatException(s"invalid input syntax for type numeric: $str")
+ } else {
+ null
+ }
+ }
+
+ if (bigDecimal != null) {
+ val precision = if (bigDecimal.scale < 0) {
+ bigDecimal.precision - bigDecimal.scale
+ } else {
+ bigDecimal.precision
+ }
+
+ if (precision > DecimalType.MAX_PRECISION) {
Review comment:
Please leave some comments here about why we need this redundant check?
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1405,4 +1426,35 @@ class AnsiCastSuite extends CastSuiteBase {
checkEvaluation(cast(negativeTs, LongType), expectedSecs)
}
}
+
+ test("Fast fail for cast string type to decimal type in ansi mode") {
+ checkEvaluation(cast("12345678901234567890123456789012345678", DecimalType(38, 0)),
+ Decimal("12345678901234567890123456789012345678"))
Review comment:
Is this behaivour different between ANSI enabled/disabled? I think this test should only include specific behaivours in the ANSI mode.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -670,18 +705,7 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
private[this] def castToDecimal(from: DataType, target: DecimalType): Any => Any = from match {
case StringType =>
- buildCast[UTF8String](_, s => try {
- // According the benchmark test, `s.toString.trim` is much faster than `s.trim.toString`.
- // Please refer to https://github.com/apache/spark/pull/26640
- changePrecision(Decimal(new JavaBigDecimal(s.toString.trim)), target)
- } catch {
- case _: NumberFormatException =>
- if (ansiEnabled) {
- throw new NumberFormatException(s"invalid input syntax for type numeric: $s")
- } else {
- null
- }
- })
+ buildCast[UTF8String](_, s => stringToDecimal(s, target))
Review comment:
nit: How about `buildCast[UTF8String](_, stringToDecimal(_, target))`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691804986
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128614/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29731: [WIP][SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-691804754
**[Test build #128614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128614/testReport)** for PR 29731 at commit [`d568e11`](https://github.com/apache/spark/commit/d568e111a24fb111b47c3f91253e5d59f1e9ec73).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692745315
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29731: [SPARK-32706][SQL] Improve cast string to decimal type
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29731:
URL: https://github.com/apache/spark/pull/29731#issuecomment-692744289
**[Test build #128716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128716/testReport)** for PR 29731 at commit [`a98c38d`](https://github.com/apache/spark/commit/a98c38d3a082823f8685d9ee4ec77ad63041052c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org