You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/05 07:56:54 UTC
[GitHub] [spark] gengliangwang opened a new pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
gengliangwang opened a new pull request #30260:
URL: https://github.com/apache/spark/pull/30260
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
In section 6.13 of the ANSI SQL standard, there are syntax rules for valid combinations of the source and target data types.
To make Spark's ANSI mode more ANSI SQL Compatible，I propose to disallow the following casting in ANSI mode:
```
TimeStamp <=> Boolean
Date <=> Boolean
Numeric <=> Timestamp
Numeric <=> Date
Numeric <=> Binary
String <=> Array
String <=> Map
String <=> Struct
```
The following castings are considered invalid in ANSI SQL standard, but they are quite straight forward. Let's Allow them for now
```
Numeric <=> Boolean
String <=> Boolean
String <=> Binary
```
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Better ANSI SQL compliance
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes, the following castings will not be allowed in ANSI mode:
```
TimeStamp <=> Boolean
Date <=> Boolean
Numeric <=> Timestamp
Numeric <=> Date
Numeric <=> Binary
String <=> Array
String <=> Map
String <=> Struct
```
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Unit test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729913045
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35902/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725464020
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726087187
**[Test build #131003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131003/testReport)** for PR 30260 at commit [`ce0e775`](https://github.com/apache/spark/commit/ce0e7757927dc8082ce09d33a9f08ac0f2410169).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724144166
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35405/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726276564
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726087187
**[Test build #131003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131003/testReport)** for PR 30260 at commit [`ce0e775`](https://github.com/apache/spark/commit/ce0e7757927dc8082ce09d33a9f08ac0f2410169).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725547759
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725463997
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35543/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729866690
**[Test build #131283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131283/testReport)** for PR 30260 at commit [`6003bef`](https://github.com/apache/spark/commit/6003bef62c59d623ca59e51532b4fa5006c56fa2).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class AnsiCastSuiteWithAnsiModeOn extends AnsiCastSuiteBase `
* `class AnsiCastSuiteWithAnsiModeOff extends AnsiCastSuiteBase `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729724925
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r525914139
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -930,150 +733,254 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
}
}
- test("Throw exception on casting out-of-range value to byte type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(ByteType)
- Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
- checkExceptionInExpression[ArithmeticException](cast(value, ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
- }
+ test("ANSI mode: Throw exception on casting out-of-range value to byte type") {
+ testIntMaxAndMin(ByteType)
+ Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, ByteType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
+ }
- Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
- checkEvaluation(cast(value, ByteType), value)
- checkEvaluation(cast(value.toString, ByteType), value)
- checkEvaluation(cast(Decimal(value.toString), ByteType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType), value)
- checkEvaluation(cast(Literal(value.toInt, DateType), ByteType), null)
- checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType), value)
- checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType), value)
- }
+ Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
+ checkEvaluation(cast(value, ByteType), value)
+ checkEvaluation(cast(value.toString, ByteType), value)
+ checkEvaluation(cast(Decimal(value.toString), ByteType), value)
+ checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType), value)
+ checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType), value)
}
}
- test("Throw exception on casting out-of-range value to short type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(ShortType)
- Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
- checkExceptionInExpression[ArithmeticException](cast(value, ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
- }
+ test("ANSI mode: Throw exception on casting out-of-range value to short type") {
+ testIntMaxAndMin(ShortType)
+ Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, ShortType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
+ }
- Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
- checkEvaluation(cast(value, ShortType), value)
- checkEvaluation(cast(value.toString, ShortType), value)
- checkEvaluation(cast(Decimal(value.toString), ShortType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType), value)
- checkEvaluation(cast(Literal(value.toInt, DateType), ShortType), null)
- checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType), value)
- checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType), value)
- }
+ Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
+ checkEvaluation(cast(value, ShortType), value)
+ checkEvaluation(cast(value.toString, ShortType), value)
+ checkEvaluation(cast(Decimal(value.toString), ShortType), value)
+ checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType), value)
+ checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType), value)
}
}
- test("Throw exception on casting out-of-range value to int type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(IntegerType)
- testLongMaxAndMin(IntegerType)
+ test("ANSI mode: Throw exception on casting out-of-range value to int type") {
+ testIntMaxAndMin(IntegerType)
+ testLongMaxAndMin(IntegerType)
- Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
- checkEvaluation(cast(value, IntegerType), value)
- checkEvaluation(cast(value.toString, IntegerType), value)
- checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), IntegerType), value)
- checkEvaluation(cast(Literal(value * 1.0, DoubleType), IntegerType), value)
- }
- checkEvaluation(cast(Int.MaxValue + 0.9D, IntegerType), Int.MaxValue)
- checkEvaluation(cast(Int.MinValue - 0.9D, IntegerType), Int.MinValue)
+ Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
+ checkEvaluation(cast(value, IntegerType), value)
+ checkEvaluation(cast(value.toString, IntegerType), value)
+ checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
+ checkEvaluation(cast(Literal(value * 1.0, DoubleType), IntegerType), value)
}
+ checkEvaluation(cast(Int.MaxValue + 0.9D, IntegerType), Int.MaxValue)
+ checkEvaluation(cast(Int.MinValue - 0.9D, IntegerType), Int.MinValue)
}
- test("Throw exception on casting out-of-range value to long type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testLongMaxAndMin(LongType)
+ test("ANSI mode: Throw exception on casting out-of-range value to long type") {
+ testLongMaxAndMin(LongType)
- Seq(Long.MaxValue, 0, Long.MinValue).foreach { value =>
- checkEvaluation(cast(value, LongType), value)
- checkEvaluation(cast(value.toString, LongType), value)
- checkEvaluation(cast(Decimal(value.toString), LongType), value)
- checkEvaluation(cast(Literal(value, TimestampType), LongType),
- Math.floorDiv(value, MICROS_PER_SECOND))
- }
- checkEvaluation(cast(Long.MaxValue + 0.9F, LongType), Long.MaxValue)
- checkEvaluation(cast(Long.MinValue - 0.9F, LongType), Long.MinValue)
- checkEvaluation(cast(Long.MaxValue + 0.9D, LongType), Long.MaxValue)
- checkEvaluation(cast(Long.MinValue - 0.9D, LongType), Long.MinValue)
+ Seq(Long.MaxValue, 0, Long.MinValue).foreach { value =>
+ checkEvaluation(cast(value, LongType), value)
+ checkEvaluation(cast(value.toString, LongType), value)
+ checkEvaluation(cast(Decimal(value.toString), LongType), value)
}
+ checkEvaluation(cast(Long.MaxValue + 0.9F, LongType), Long.MaxValue)
+ checkEvaluation(cast(Long.MinValue - 0.9F, LongType), Long.MinValue)
+ checkEvaluation(cast(Long.MaxValue + 0.9D, LongType), Long.MaxValue)
+ checkEvaluation(cast(Long.MinValue - 0.9D, LongType), Long.MinValue)
}
-}
-/**
- * Test suite for data type casting expression [[Cast]].
- */
-class CastSuite extends CastSuiteBase {
- // It is required to set SQLConf.ANSI_ENABLED as true for testing numeric overflow.
- override protected def requiredAnsiEnabledForOverflowTestCases: Boolean = true
+ test("ANSI mode: Throw exception on casting out-of-range value to decimal type") {
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal("134.12"), DecimalType(3, 2)), "cannot be represented")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(BigDecimal(134.12)), DecimalType(3, 2)), "cannot be represented")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(134.12), DecimalType(3, 2)), "cannot be represented")
+ }
- override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
- v match {
- case lit: Expression => Cast(lit, targetType, timeZoneId)
- case _ => Cast(Literal(v), targetType, timeZoneId)
+ test("ANSI mode: disallow type conversions between Numeric types and Timestamp type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(TimestampType)
+ val timestampLiteral = Literal(1L, TimestampType)
+ numericTypes.foreach { numericType =>
+ assert(cast(timestampLiteral, numericType).checkInputDataTypes().isFailure)
}
}
- test("cast from int") {
- checkCast(0, false)
- checkCast(1, true)
- checkCast(-5, true)
- checkCast(1, 1.toByte)
- checkCast(1, 1.toShort)
- checkCast(1, 1)
- checkCast(1, 1.toLong)
- checkCast(1, 1.0f)
- checkCast(1, 1.0)
- checkCast(123, "123")
-
- checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
- checkEvaluation(cast(123, DecimalType(3, 0)), Decimal(123))
- checkEvaluation(cast(123, DecimalType(3, 1)), null)
- checkEvaluation(cast(123, DecimalType(2, 0)), null)
+ test("ANSI mode: disallow type conversions between Numeric types and Date type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(DateType)
+ val dateLiteral = Literal(1, DateType)
+ numericTypes.foreach { numericType =>
+ assert(cast(dateLiteral, numericType).checkInputDataTypes().isFailure)
+ }
}
- test("cast from long") {
- checkCast(0L, false)
- checkCast(1L, true)
- checkCast(-5L, true)
- checkCast(1L, 1.toByte)
- checkCast(1L, 1.toShort)
- checkCast(1L, 1)
- checkCast(1L, 1.toLong)
- checkCast(1L, 1.0f)
- checkCast(1L, 1.0)
- checkCast(123L, "123")
+ test("ANSI mode: disallow type conversions between Numeric types and Binary type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(BinaryType)
+ val binaryLiteral = Literal(new Array[Byte](1.toByte), BinaryType)
+ numericTypes.foreach { numericType =>
+ assert(cast(binaryLiteral, numericType).checkInputDataTypes().isFailure)
+ }
+ }
- checkEvaluation(cast(123L, DecimalType.USER_DEFAULT), Decimal(123))
- checkEvaluation(cast(123L, DecimalType(3, 0)), Decimal(123))
- checkEvaluation(cast(123L, DecimalType(3, 1)), null)
+ test("ANSI mode: disallow type conversions between Datatime types and Boolean types") {
+ val timestampLiteral = Literal(1L, TimestampType)
+ assert(cast(timestampLiteral, BooleanType).checkInputDataTypes().isFailure)
+ val dateLiteral = Literal(1, DateType)
+ assert(cast(dateLiteral, BooleanType).checkInputDataTypes().isFailure)
- checkEvaluation(cast(123L, DecimalType(2, 0)), null)
+ val booleanLiteral = Literal(true, BooleanType)
+ assert(cast(booleanLiteral, TimestampType).checkInputDataTypes().isFailure)
+ assert(cast(booleanLiteral, DateType).checkInputDataTypes().isFailure)
}
- test("cast from int 2") {
- checkEvaluation(cast(1, LongType), 1.toLong)
-
- checkEvaluation(cast(cast(1000, TimestampType), LongType), 1000.toLong)
- checkEvaluation(cast(cast(-1200, TimestampType), LongType), -1200.toLong)
+ test("ANSI mode: disallow casting complex types as String type") {
+ assert(cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType).checkInputDataTypes().isFailure)
+ assert(cast(Literal.create(Map(1 -> "a")), StringType).checkInputDataTypes().isFailure)
+ assert(cast(Literal.create((1, "a", 0.1)), StringType).checkInputDataTypes().isFailure)
+ }
- checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
+ test("cast from invalid string to numeric should throw NumberFormatException") {
+ // cast to IntegerType
+ Seq(IntegerType, ShortType, ByteType, LongType).foreach { dataType =>
+ val array = Literal.create(Seq("123", "true", "f", null),
+ ArrayType(StringType, containsNull = true))
+ checkExceptionInExpression[NumberFormatException](
+ cast(array, ArrayType(dataType, containsNull = true)),
+ "invalid input syntax for type numeric: true")
+ checkExceptionInExpression[NumberFormatException](
+ cast("string", dataType), "invalid input syntax for type numeric: string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("123-string", dataType), "invalid input syntax for type numeric: 123-string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("2020-07-19", dataType), "invalid input syntax for type numeric: 2020-07-19")
+ checkExceptionInExpression[NumberFormatException](
+ cast("1.23", dataType), "invalid input syntax for type numeric: 1.23")
+ }
+
+ Seq(DoubleType, FloatType, DecimalType.USER_DEFAULT).foreach { dataType =>
+ checkExceptionInExpression[NumberFormatException](
+ cast("string", dataType), "invalid input syntax for type numeric: string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("123.000.00", dataType), "invalid input syntax for type numeric: 123.000.00")
+ checkExceptionInExpression[NumberFormatException](
+ cast("abc.com", dataType), "invalid input syntax for type numeric: abc.com")
+ }
+ }
+
+ test("Fast fail for cast string type to decimal type in ansi mode") {
+ checkEvaluation(cast("12345678901234567890123456789012345678", DecimalType(38, 0)),
+ Decimal("12345678901234567890123456789012345678"))
+ checkExceptionInExpression[ArithmeticException](
+ cast("123456789012345678901234567890123456789", DecimalType(38, 0)),
+ "out of decimal type range")
+ checkExceptionInExpression[ArithmeticException](
+ cast("12345678901234567890123456789012345678", DecimalType(38, 1)),
+ "cannot be represented as Decimal(38, 1)")
+
+ checkEvaluation(cast("0.00000000000000000000000000000000000001", DecimalType(38, 0)),
+ Decimal("0"))
+ checkEvaluation(cast("0.00000000000000000000000000000000000000000001", DecimalType(38, 0)),
+ Decimal("0"))
+ checkEvaluation(cast("0.00000000000000000000000000000000000001", DecimalType(38, 18)),
+ Decimal("0E-18"))
+ checkEvaluation(cast("6E-120", DecimalType(38, 0)),
+ Decimal("0"))
+
+ checkEvaluation(cast("6E+37", DecimalType(38, 0)),
+ Decimal("60000000000000000000000000000000000000"))
+ checkExceptionInExpression[ArithmeticException](
+ cast("6E+38", DecimalType(38, 0)),
+ "out of decimal type range")
+ checkExceptionInExpression[ArithmeticException](
+ cast("6E+37", DecimalType(38, 1)),
+ "cannot be represented as Decimal(38, 1)")
+
+ checkExceptionInExpression[NumberFormatException](
+ cast("abcd", DecimalType(38, 1)),
+ "invalid input syntax for type numeric")
+ }
+}
+
+/**
+ * Test suite for data type casting expression [[Cast]].
+ */
+class CastSuite extends CastSuiteBase {
+
+ override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
+ v match {
+ case lit: Expression => Cast(lit, targetType, timeZoneId)
+ case _ => Cast(Literal(v), targetType, timeZoneId)
+ }
+ }
+
+ test("null cast II") {
+ import DataTypeTestUtils._
+
+ checkNullCast(DateType, BooleanType)
+ checkNullCast(TimestampType, BooleanType)
+ checkNullCast(BooleanType, TimestampType)
+ numericTypes.foreach(dt => checkNullCast(dt, TimestampType))
+ numericTypes.foreach(dt => checkNullCast(TimestampType, dt))
+ numericTypes.foreach(dt => checkNullCast(DateType, dt))
+ }
+
+ test("cast from int") {
+ checkCast(0, false)
+ checkCast(1, true)
+ checkCast(-5, true)
+ checkCast(1, 1.toByte)
+ checkCast(1, 1.toShort)
+ checkCast(1, 1)
+ checkCast(1, 1.toLong)
+ checkCast(1, 1.0f)
+ checkCast(1, 1.0)
+ checkCast(123, "123")
+
+ checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
+ checkEvaluation(cast(123, DecimalType(3, 0)), Decimal(123))
+ checkEvaluation(cast(123, DecimalType(3, 1)), null)
+ checkEvaluation(cast(123, DecimalType(2, 0)), null)
+ }
+
+ test("cast from long") {
+ checkCast(0L, false)
+ checkCast(1L, true)
+ checkCast(-5L, true)
+ checkCast(1L, 1.toByte)
+ checkCast(1L, 1.toShort)
+ checkCast(1L, 1)
+ checkCast(1L, 1.toLong)
+ checkCast(1L, 1.0f)
+ checkCast(1L, 1.0)
+ checkCast(123L, "123")
+
+ checkEvaluation(cast(123L, DecimalType.USER_DEFAULT), Decimal(123))
+ checkEvaluation(cast(123L, DecimalType(3, 0)), Decimal(123))
+ checkEvaluation(cast(123L, DecimalType(3, 1)), null)
+
+ checkEvaluation(cast(123L, DecimalType(2, 0)), null)
+ }
+
+ test("cast from int 2") {
Review comment:
`cast from int #2`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725494779
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r525913920
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -930,150 +733,254 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
}
}
- test("Throw exception on casting out-of-range value to byte type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(ByteType)
- Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
- checkExceptionInExpression[ArithmeticException](cast(value, ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
- }
+ test("ANSI mode: Throw exception on casting out-of-range value to byte type") {
+ testIntMaxAndMin(ByteType)
+ Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, ByteType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
+ }
- Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
- checkEvaluation(cast(value, ByteType), value)
- checkEvaluation(cast(value.toString, ByteType), value)
- checkEvaluation(cast(Decimal(value.toString), ByteType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType), value)
- checkEvaluation(cast(Literal(value.toInt, DateType), ByteType), null)
- checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType), value)
- checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType), value)
- }
+ Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
+ checkEvaluation(cast(value, ByteType), value)
+ checkEvaluation(cast(value.toString, ByteType), value)
+ checkEvaluation(cast(Decimal(value.toString), ByteType), value)
+ checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType), value)
+ checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType), value)
}
}
- test("Throw exception on casting out-of-range value to short type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(ShortType)
- Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
- checkExceptionInExpression[ArithmeticException](cast(value, ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
- }
+ test("ANSI mode: Throw exception on casting out-of-range value to short type") {
+ testIntMaxAndMin(ShortType)
+ Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, ShortType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
+ }
- Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
- checkEvaluation(cast(value, ShortType), value)
- checkEvaluation(cast(value.toString, ShortType), value)
- checkEvaluation(cast(Decimal(value.toString), ShortType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType), value)
- checkEvaluation(cast(Literal(value.toInt, DateType), ShortType), null)
- checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType), value)
- checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType), value)
- }
+ Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
+ checkEvaluation(cast(value, ShortType), value)
+ checkEvaluation(cast(value.toString, ShortType), value)
+ checkEvaluation(cast(Decimal(value.toString), ShortType), value)
+ checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType), value)
+ checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType), value)
}
}
- test("Throw exception on casting out-of-range value to int type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(IntegerType)
- testLongMaxAndMin(IntegerType)
+ test("ANSI mode: Throw exception on casting out-of-range value to int type") {
+ testIntMaxAndMin(IntegerType)
+ testLongMaxAndMin(IntegerType)
- Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
- checkEvaluation(cast(value, IntegerType), value)
- checkEvaluation(cast(value.toString, IntegerType), value)
- checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), IntegerType), value)
- checkEvaluation(cast(Literal(value * 1.0, DoubleType), IntegerType), value)
- }
- checkEvaluation(cast(Int.MaxValue + 0.9D, IntegerType), Int.MaxValue)
- checkEvaluation(cast(Int.MinValue - 0.9D, IntegerType), Int.MinValue)
+ Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
+ checkEvaluation(cast(value, IntegerType), value)
+ checkEvaluation(cast(value.toString, IntegerType), value)
+ checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
+ checkEvaluation(cast(Literal(value * 1.0, DoubleType), IntegerType), value)
}
+ checkEvaluation(cast(Int.MaxValue + 0.9D, IntegerType), Int.MaxValue)
+ checkEvaluation(cast(Int.MinValue - 0.9D, IntegerType), Int.MinValue)
}
- test("Throw exception on casting out-of-range value to long type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testLongMaxAndMin(LongType)
+ test("ANSI mode: Throw exception on casting out-of-range value to long type") {
+ testLongMaxAndMin(LongType)
- Seq(Long.MaxValue, 0, Long.MinValue).foreach { value =>
- checkEvaluation(cast(value, LongType), value)
- checkEvaluation(cast(value.toString, LongType), value)
- checkEvaluation(cast(Decimal(value.toString), LongType), value)
- checkEvaluation(cast(Literal(value, TimestampType), LongType),
- Math.floorDiv(value, MICROS_PER_SECOND))
- }
- checkEvaluation(cast(Long.MaxValue + 0.9F, LongType), Long.MaxValue)
- checkEvaluation(cast(Long.MinValue - 0.9F, LongType), Long.MinValue)
- checkEvaluation(cast(Long.MaxValue + 0.9D, LongType), Long.MaxValue)
- checkEvaluation(cast(Long.MinValue - 0.9D, LongType), Long.MinValue)
+ Seq(Long.MaxValue, 0, Long.MinValue).foreach { value =>
+ checkEvaluation(cast(value, LongType), value)
+ checkEvaluation(cast(value.toString, LongType), value)
+ checkEvaluation(cast(Decimal(value.toString), LongType), value)
}
+ checkEvaluation(cast(Long.MaxValue + 0.9F, LongType), Long.MaxValue)
+ checkEvaluation(cast(Long.MinValue - 0.9F, LongType), Long.MinValue)
+ checkEvaluation(cast(Long.MaxValue + 0.9D, LongType), Long.MaxValue)
+ checkEvaluation(cast(Long.MinValue - 0.9D, LongType), Long.MinValue)
}
-}
-/**
- * Test suite for data type casting expression [[Cast]].
- */
-class CastSuite extends CastSuiteBase {
- // It is required to set SQLConf.ANSI_ENABLED as true for testing numeric overflow.
- override protected def requiredAnsiEnabledForOverflowTestCases: Boolean = true
+ test("ANSI mode: Throw exception on casting out-of-range value to decimal type") {
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal("134.12"), DecimalType(3, 2)), "cannot be represented")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(BigDecimal(134.12)), DecimalType(3, 2)), "cannot be represented")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(134.12), DecimalType(3, 2)), "cannot be represented")
+ }
- override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
- v match {
- case lit: Expression => Cast(lit, targetType, timeZoneId)
- case _ => Cast(Literal(v), targetType, timeZoneId)
+ test("ANSI mode: disallow type conversions between Numeric types and Timestamp type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(TimestampType)
+ val timestampLiteral = Literal(1L, TimestampType)
+ numericTypes.foreach { numericType =>
+ assert(cast(timestampLiteral, numericType).checkInputDataTypes().isFailure)
}
}
- test("cast from int") {
- checkCast(0, false)
- checkCast(1, true)
- checkCast(-5, true)
- checkCast(1, 1.toByte)
- checkCast(1, 1.toShort)
- checkCast(1, 1)
- checkCast(1, 1.toLong)
- checkCast(1, 1.0f)
- checkCast(1, 1.0)
- checkCast(123, "123")
-
- checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
- checkEvaluation(cast(123, DecimalType(3, 0)), Decimal(123))
- checkEvaluation(cast(123, DecimalType(3, 1)), null)
- checkEvaluation(cast(123, DecimalType(2, 0)), null)
+ test("ANSI mode: disallow type conversions between Numeric types and Date type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(DateType)
+ val dateLiteral = Literal(1, DateType)
+ numericTypes.foreach { numericType =>
+ assert(cast(dateLiteral, numericType).checkInputDataTypes().isFailure)
+ }
}
- test("cast from long") {
- checkCast(0L, false)
- checkCast(1L, true)
- checkCast(-5L, true)
- checkCast(1L, 1.toByte)
- checkCast(1L, 1.toShort)
- checkCast(1L, 1)
- checkCast(1L, 1.toLong)
- checkCast(1L, 1.0f)
- checkCast(1L, 1.0)
- checkCast(123L, "123")
+ test("ANSI mode: disallow type conversions between Numeric types and Binary type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(BinaryType)
+ val binaryLiteral = Literal(new Array[Byte](1.toByte), BinaryType)
+ numericTypes.foreach { numericType =>
+ assert(cast(binaryLiteral, numericType).checkInputDataTypes().isFailure)
+ }
+ }
- checkEvaluation(cast(123L, DecimalType.USER_DEFAULT), Decimal(123))
- checkEvaluation(cast(123L, DecimalType(3, 0)), Decimal(123))
- checkEvaluation(cast(123L, DecimalType(3, 1)), null)
+ test("ANSI mode: disallow type conversions between Datatime types and Boolean types") {
+ val timestampLiteral = Literal(1L, TimestampType)
+ assert(cast(timestampLiteral, BooleanType).checkInputDataTypes().isFailure)
+ val dateLiteral = Literal(1, DateType)
+ assert(cast(dateLiteral, BooleanType).checkInputDataTypes().isFailure)
- checkEvaluation(cast(123L, DecimalType(2, 0)), null)
+ val booleanLiteral = Literal(true, BooleanType)
+ assert(cast(booleanLiteral, TimestampType).checkInputDataTypes().isFailure)
+ assert(cast(booleanLiteral, DateType).checkInputDataTypes().isFailure)
}
- test("cast from int 2") {
- checkEvaluation(cast(1, LongType), 1.toLong)
-
- checkEvaluation(cast(cast(1000, TimestampType), LongType), 1000.toLong)
- checkEvaluation(cast(cast(-1200, TimestampType), LongType), -1200.toLong)
+ test("ANSI mode: disallow casting complex types as String type") {
+ assert(cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType).checkInputDataTypes().isFailure)
+ assert(cast(Literal.create(Map(1 -> "a")), StringType).checkInputDataTypes().isFailure)
+ assert(cast(Literal.create((1, "a", 0.1)), StringType).checkInputDataTypes().isFailure)
+ }
- checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
+ test("cast from invalid string to numeric should throw NumberFormatException") {
+ // cast to IntegerType
+ Seq(IntegerType, ShortType, ByteType, LongType).foreach { dataType =>
+ val array = Literal.create(Seq("123", "true", "f", null),
+ ArrayType(StringType, containsNull = true))
+ checkExceptionInExpression[NumberFormatException](
+ cast(array, ArrayType(dataType, containsNull = true)),
+ "invalid input syntax for type numeric: true")
+ checkExceptionInExpression[NumberFormatException](
+ cast("string", dataType), "invalid input syntax for type numeric: string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("123-string", dataType), "invalid input syntax for type numeric: 123-string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("2020-07-19", dataType), "invalid input syntax for type numeric: 2020-07-19")
+ checkExceptionInExpression[NumberFormatException](
+ cast("1.23", dataType), "invalid input syntax for type numeric: 1.23")
+ }
+
+ Seq(DoubleType, FloatType, DecimalType.USER_DEFAULT).foreach { dataType =>
+ checkExceptionInExpression[NumberFormatException](
+ cast("string", dataType), "invalid input syntax for type numeric: string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("123.000.00", dataType), "invalid input syntax for type numeric: 123.000.00")
+ checkExceptionInExpression[NumberFormatException](
+ cast("abc.com", dataType), "invalid input syntax for type numeric: abc.com")
+ }
+ }
+
+ test("Fast fail for cast string type to decimal type in ansi mode") {
+ checkEvaluation(cast("12345678901234567890123456789012345678", DecimalType(38, 0)),
+ Decimal("12345678901234567890123456789012345678"))
+ checkExceptionInExpression[ArithmeticException](
+ cast("123456789012345678901234567890123456789", DecimalType(38, 0)),
+ "out of decimal type range")
+ checkExceptionInExpression[ArithmeticException](
+ cast("12345678901234567890123456789012345678", DecimalType(38, 1)),
+ "cannot be represented as Decimal(38, 1)")
+
+ checkEvaluation(cast("0.00000000000000000000000000000000000001", DecimalType(38, 0)),
+ Decimal("0"))
+ checkEvaluation(cast("0.00000000000000000000000000000000000000000001", DecimalType(38, 0)),
+ Decimal("0"))
+ checkEvaluation(cast("0.00000000000000000000000000000000000001", DecimalType(38, 18)),
+ Decimal("0E-18"))
+ checkEvaluation(cast("6E-120", DecimalType(38, 0)),
+ Decimal("0"))
+
+ checkEvaluation(cast("6E+37", DecimalType(38, 0)),
+ Decimal("60000000000000000000000000000000000000"))
+ checkExceptionInExpression[ArithmeticException](
+ cast("6E+38", DecimalType(38, 0)),
+ "out of decimal type range")
+ checkExceptionInExpression[ArithmeticException](
+ cast("6E+37", DecimalType(38, 1)),
+ "cannot be represented as Decimal(38, 1)")
+
+ checkExceptionInExpression[NumberFormatException](
+ cast("abcd", DecimalType(38, 1)),
+ "invalid input syntax for type numeric")
+ }
+}
+
+/**
+ * Test suite for data type casting expression [[Cast]].
+ */
+class CastSuite extends CastSuiteBase {
+
+ override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
+ v match {
+ case lit: Expression => Cast(lit, targetType, timeZoneId)
+ case _ => Cast(Literal(v), targetType, timeZoneId)
+ }
+ }
+
+ test("null cast II") {
+ import DataTypeTestUtils._
+
+ checkNullCast(DateType, BooleanType)
+ checkNullCast(TimestampType, BooleanType)
+ checkNullCast(BooleanType, TimestampType)
+ numericTypes.foreach(dt => checkNullCast(dt, TimestampType))
+ numericTypes.foreach(dt => checkNullCast(TimestampType, dt))
+ numericTypes.foreach(dt => checkNullCast(DateType, dt))
+ }
+
+ test("cast from int") {
+ checkCast(0, false)
+ checkCast(1, true)
+ checkCast(-5, true)
+ checkCast(1, 1.toByte)
+ checkCast(1, 1.toShort)
Review comment:
are these tests non-ansi specific?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725619864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722246653
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726080293
@maropu Thanks again for the thoughtful suggestions! I have updated the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722212874
**[Test build #130642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130642/testReport)** for PR 30260 at commit [`fe0fe48`](https://github.com/apache/spark/commit/fe0fe4872c54550b44ab9458edb63d91d78b2ae0).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r520241004
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1770,6 +1782,64 @@ case class AnsiCast(child: Expression, dataType: DataType, timeZoneId: Option[St
copy(timeZoneId = Option(timeZoneId))
override protected val ansiEnabled: Boolean = true
+
+ override def canCast(from: DataType, to: DataType): Boolean = AnsiCast.canCast(from, to)
+}
+
+object AnsiCast {
+ def canCast(from: DataType, to: DataType): Boolean = (from, to) match {
+ case (fromType, toType) if fromType == toType => true
+
+ case (NullType, _) => true
+
+ case (_: NumericType, _: NumericType) => true
+ case (StringType, _: NumericType) => true
+ case (BooleanType, _: NumericType) => true
Review comment:
(Just a suggestion) For readability, could you reorder these entries according to `Cast.canCast` where possible? For example, the numeric entries for `Cast` seems to be placed just before complex types:
https://github.com/apache/spark/blob/35ac314181129374b02f8f8c07341b43a734e1c7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L70-L74
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1396,14 +1337,125 @@ class CastSuite extends CastSuiteBase {
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
}
+
+ test("SPARK-22825 Cast array to string") {
Review comment:
(nit and this is not related to this PR though...) `SPARK-22825` -> `SPARK-22825:`
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala
##########
@@ -756,6 +756,47 @@ class InsertSuite extends DataSourceTest with SharedSparkSession {
}
}
+ test("SPARK-33354: Throw exceptions on inserting invalid cast with ANSI casting policy") {
+ withSQLConf(
+ SQLConf.STORE_ASSIGNMENT_POLICY.key -> SQLConf.StoreAssignmentPolicy.ANSI.toString) {
+ withTable("t") {
+ sql("create table t(i int, t timestamp) using parquet")
Review comment:
nit: please use uppercases for SQL keywords, e.g., `create` -> `CREATE`.
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1396,14 +1337,125 @@ class CastSuite extends CastSuiteBase {
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
}
+
+ test("SPARK-22825 Cast array to string") {
+ val ret1 = cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType)
+ checkEvaluation(ret1, "[1, 2, 3, 4, 5]")
+ val ret2 = cast(Literal.create(Array("ab", "cde", "f")), StringType)
+ checkEvaluation(ret2, "[ab, cde, f]")
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret3 = cast(Literal.create(Array("ab", null, "c")), StringType)
+ checkEvaluation(ret3, s"[ab,${if (omitNull) "" else " null"}, c]")
+ }
+ }
+ val ret4 =
+ cast(Literal.create(Array("ab".getBytes, "cde".getBytes, "f".getBytes)), StringType)
+ checkEvaluation(ret4, "[ab, cde, f]")
+ val ret5 = cast(
+ Literal.create(Array("2014-12-03", "2014-12-04", "2014-12-06").map(Date.valueOf)),
+ StringType)
+ checkEvaluation(ret5, "[2014-12-03, 2014-12-04, 2014-12-06]")
+ val ret6 = cast(
+ Literal.create(Array("2014-12-03 13:01:00", "2014-12-04 15:05:00")
+ .map(Timestamp.valueOf)),
+ StringType)
+ checkEvaluation(ret6, "[2014-12-03 13:01:00, 2014-12-04 15:05:00]")
+ val ret7 = cast(Literal.create(Array(Array(1, 2, 3), Array(4, 5))), StringType)
+ checkEvaluation(ret7, "[[1, 2, 3], [4, 5]]")
+ val ret8 = cast(
+ Literal.create(Array(Array(Array("a"), Array("b", "c")), Array(Array("d")))),
+ StringType)
+ checkEvaluation(ret8, "[[[a], [b, c]], [[d]]]")
+ }
+
+ test("SPARK-33291: Cast array with null elements to string") {
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret1 = cast(Literal.create(Array(null, null)), StringType)
+ checkEvaluation(
+ ret1,
+ s"[${if (omitNull) "" else "null"},${if (omitNull) "" else " null"}]")
+ }
+ }
+ }
+
+ test("SPARK-22973 Cast map to string") {
Review comment:
ditto
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1396,14 +1337,125 @@ class CastSuite extends CastSuiteBase {
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
}
+
+ test("SPARK-22825 Cast array to string") {
+ val ret1 = cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType)
+ checkEvaluation(ret1, "[1, 2, 3, 4, 5]")
+ val ret2 = cast(Literal.create(Array("ab", "cde", "f")), StringType)
+ checkEvaluation(ret2, "[ab, cde, f]")
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret3 = cast(Literal.create(Array("ab", null, "c")), StringType)
+ checkEvaluation(ret3, s"[ab,${if (omitNull) "" else " null"}, c]")
+ }
+ }
+ val ret4 =
+ cast(Literal.create(Array("ab".getBytes, "cde".getBytes, "f".getBytes)), StringType)
+ checkEvaluation(ret4, "[ab, cde, f]")
+ val ret5 = cast(
+ Literal.create(Array("2014-12-03", "2014-12-04", "2014-12-06").map(Date.valueOf)),
+ StringType)
+ checkEvaluation(ret5, "[2014-12-03, 2014-12-04, 2014-12-06]")
+ val ret6 = cast(
+ Literal.create(Array("2014-12-03 13:01:00", "2014-12-04 15:05:00")
+ .map(Timestamp.valueOf)),
+ StringType)
+ checkEvaluation(ret6, "[2014-12-03 13:01:00, 2014-12-04 15:05:00]")
+ val ret7 = cast(Literal.create(Array(Array(1, 2, 3), Array(4, 5))), StringType)
+ checkEvaluation(ret7, "[[1, 2, 3], [4, 5]]")
+ val ret8 = cast(
+ Literal.create(Array(Array(Array("a"), Array("b", "c")), Array(Array("d")))),
+ StringType)
+ checkEvaluation(ret8, "[[[a], [b, c]], [[d]]]")
+ }
+
+ test("SPARK-33291: Cast array with null elements to string") {
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret1 = cast(Literal.create(Array(null, null)), StringType)
+ checkEvaluation(
+ ret1,
+ s"[${if (omitNull) "" else "null"},${if (omitNull) "" else " null"}]")
+ }
+ }
+ }
+
+ test("SPARK-22973 Cast map to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create(Map(1 -> "a", 2 -> "b", 3 -> "c")), StringType)
+ checkEvaluation(ret1, s"${lb}1 -> a, 2 -> b, 3 -> c$rb")
+ val ret2 = cast(
+ Literal.create(Map("1" -> "a".getBytes, "2" -> null, "3" -> "c".getBytes)),
+ StringType)
+ checkEvaluation(ret2, s"${lb}1 -> a, 2 ->${if (legacyCast) "" else " null"}, 3 -> c$rb")
+ val ret3 = cast(
+ Literal.create(Map(
+ 1 -> Date.valueOf("2014-12-03"),
+ 2 -> Date.valueOf("2014-12-04"),
+ 3 -> Date.valueOf("2014-12-05"))),
+ StringType)
+ checkEvaluation(ret3, s"${lb}1 -> 2014-12-03, 2 -> 2014-12-04, 3 -> 2014-12-05$rb")
+ val ret4 = cast(
+ Literal.create(Map(
+ 1 -> Timestamp.valueOf("2014-12-03 13:01:00"),
+ 2 -> Timestamp.valueOf("2014-12-04 15:05:00"))),
+ StringType)
+ checkEvaluation(ret4, s"${lb}1 -> 2014-12-03 13:01:00, 2 -> 2014-12-04 15:05:00$rb")
+ val ret5 = cast(
+ Literal.create(Map(
+ 1 -> Array(1, 2, 3),
+ 2 -> Array(4, 5, 6))),
+ StringType)
+ checkEvaluation(ret5, s"${lb}1 -> [1, 2, 3], 2 -> [4, 5, 6]$rb")
+ }
+ }
+ }
+
+ test("SPARK-22981 Cast struct to string") {
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729876872
**[Test build #131300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131300/testReport)** for PR 30260 at commit [`6003bef`](https://github.com/apache/spark/commit/6003bef62c59d623ca59e51532b4fa5006c56fa2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722546948
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r526112368
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1396,93 +1355,199 @@ class CastSuite extends CastSuiteBase {
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
}
+
+ test("SPARK-22825 Cast array to string") {
+ val ret1 = cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType)
+ checkEvaluation(ret1, "[1, 2, 3, 4, 5]")
+ val ret2 = cast(Literal.create(Array("ab", "cde", "f")), StringType)
+ checkEvaluation(ret2, "[ab, cde, f]")
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret3 = cast(Literal.create(Array("ab", null, "c")), StringType)
+ checkEvaluation(ret3, s"[ab,${if (omitNull) "" else " null"}, c]")
+ }
+ }
+ val ret4 =
+ cast(Literal.create(Array("ab".getBytes, "cde".getBytes, "f".getBytes)), StringType)
+ checkEvaluation(ret4, "[ab, cde, f]")
+ val ret5 = cast(
+ Literal.create(Array("2014-12-03", "2014-12-04", "2014-12-06").map(Date.valueOf)),
+ StringType)
+ checkEvaluation(ret5, "[2014-12-03, 2014-12-04, 2014-12-06]")
+ val ret6 = cast(
+ Literal.create(Array("2014-12-03 13:01:00", "2014-12-04 15:05:00")
+ .map(Timestamp.valueOf)),
+ StringType)
+ checkEvaluation(ret6, "[2014-12-03 13:01:00, 2014-12-04 15:05:00]")
+ val ret7 = cast(Literal.create(Array(Array(1, 2, 3), Array(4, 5))), StringType)
+ checkEvaluation(ret7, "[[1, 2, 3], [4, 5]]")
+ val ret8 = cast(
+ Literal.create(Array(Array(Array("a"), Array("b", "c")), Array(Array("d")))),
+ StringType)
+ checkEvaluation(ret8, "[[[a], [b, c]], [[d]]]")
+ }
+
+ test("SPARK-33291: Cast array with null elements to string") {
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret1 = cast(Literal.create(Array(null, null)), StringType)
+ checkEvaluation(
+ ret1,
+ s"[${if (omitNull) "" else "null"},${if (omitNull) "" else " null"}]")
+ }
+ }
+ }
+
+ test("SPARK-22973 Cast map to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create(Map(1 -> "a", 2 -> "b", 3 -> "c")), StringType)
+ checkEvaluation(ret1, s"${lb}1 -> a, 2 -> b, 3 -> c$rb")
+ val ret2 = cast(
+ Literal.create(Map("1" -> "a".getBytes, "2" -> null, "3" -> "c".getBytes)),
+ StringType)
+ checkEvaluation(ret2, s"${lb}1 -> a, 2 ->${if (legacyCast) "" else " null"}, 3 -> c$rb")
+ val ret3 = cast(
+ Literal.create(Map(
+ 1 -> Date.valueOf("2014-12-03"),
+ 2 -> Date.valueOf("2014-12-04"),
+ 3 -> Date.valueOf("2014-12-05"))),
+ StringType)
+ checkEvaluation(ret3, s"${lb}1 -> 2014-12-03, 2 -> 2014-12-04, 3 -> 2014-12-05$rb")
+ val ret4 = cast(
+ Literal.create(Map(
+ 1 -> Timestamp.valueOf("2014-12-03 13:01:00"),
+ 2 -> Timestamp.valueOf("2014-12-04 15:05:00"))),
+ StringType)
+ checkEvaluation(ret4, s"${lb}1 -> 2014-12-03 13:01:00, 2 -> 2014-12-04 15:05:00$rb")
+ val ret5 = cast(
+ Literal.create(Map(
+ 1 -> Array(1, 2, 3),
+ 2 -> Array(4, 5, 6))),
+ StringType)
+ checkEvaluation(ret5, s"${lb}1 -> [1, 2, 3], 2 -> [4, 5, 6]$rb")
+ }
+ }
+ }
+
+ test("SPARK-22981 Cast struct to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create((1, "a", 0.1)), StringType)
+ checkEvaluation(ret1, s"${lb}1, a, 0.1$rb")
+ val ret2 = cast(Literal.create(Tuple3[Int, String, String](1, null, "a")), StringType)
+ checkEvaluation(ret2, s"${lb}1,${if (legacyCast) "" else " null"}, a$rb")
+ val ret3 = cast(Literal.create(
+ (Date.valueOf("2014-12-03"), Timestamp.valueOf("2014-12-03 15:05:00"))), StringType)
+ checkEvaluation(ret3, s"${lb}2014-12-03, 2014-12-03 15:05:00$rb")
+ val ret4 = cast(Literal.create(((1, "a"), 5, 0.1)), StringType)
+ checkEvaluation(ret4, s"$lb${lb}1, a$rb, 5, 0.1$rb")
+ val ret5 = cast(Literal.create((Seq(1, 2, 3), "a", 0.1)), StringType)
+ checkEvaluation(ret5, s"$lb[1, 2, 3], a, 0.1$rb")
+ val ret6 = cast(Literal.create((1, Map(1 -> "a", 2 -> "b", 3 -> "c"))), StringType)
+ checkEvaluation(ret6, s"${lb}1, ${lb}1 -> a, 2 -> b, 3 -> c$rb$rb")
+ }
+ }
+ }
+
+ test("SPARK-33291: Cast struct with null elements to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create(Tuple2[String, String](null, null)), StringType)
+ checkEvaluation(
+ ret1,
+ s"$lb${if (legacyCast) "" else "null"},${if (legacyCast) "" else " null"}$rb")
+ }
+ }
+ }
+
+ test("data type casting II") {
+ checkEvaluation(
+ cast(cast(cast(cast(cast(cast("5", ByteType), TimestampType),
+ DecimalType.SYSTEM_DEFAULT), LongType), StringType), ShortType),
+ 5.toShort)
+ checkEvaluation(
+ cast(cast(cast(cast(cast(cast("5", TimestampType, UTC_OPT), ByteType),
+ DecimalType.SYSTEM_DEFAULT), LongType), StringType), ShortType),
+ null)
+ checkEvaluation(cast(cast(cast(cast(cast(cast("5", DecimalType.SYSTEM_DEFAULT),
+ ByteType), TimestampType), LongType), StringType), ShortType),
+ 5.toShort)
+ }
+
+ test("Cast from double II") {
+ checkEvaluation(cast(cast(1.toDouble, TimestampType), DoubleType), 1.toDouble)
+ }
}
/**
- * Test suite for data type casting expression [[AnsiCast]].
+ * Test suite for data type casting expression [[Cast]] with ANSI mode disabled.
*/
-class AnsiCastSuite extends CastSuiteBase {
- // It is not required to set SQLConf.ANSI_ENABLED as true for testing numeric overflow.
- override protected def requiredAnsiEnabledForOverflowTestCases: Boolean = false
+class CastSuiteWithAnsiModeOn extends AnsiCastSuiteBase {
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ SQLConf.get.setConf(SQLConf.ANSI_ENABLED, true)
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ SQLConf.get.unsetConf(SQLConf.ANSI_ENABLED)
+ }
override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
v match {
- case lit: Expression => AnsiCast(lit, targetType, timeZoneId)
- case _ => AnsiCast(Literal(v), targetType, timeZoneId)
+ case lit: Expression => Cast(lit, targetType, timeZoneId)
+ case _ => Cast(Literal(v), targetType, timeZoneId)
}
}
+}
- test("cast from invalid string to numeric should throw NumberFormatException") {
- // cast to IntegerType
- Seq(IntegerType, ShortType, ByteType, LongType).foreach { dataType =>
- val array = Literal.create(Seq("123", "true", "f", null),
- ArrayType(StringType, containsNull = true))
- checkExceptionInExpression[NumberFormatException](
- cast(array, ArrayType(dataType, containsNull = true)),
- "invalid input syntax for type numeric: true")
- checkExceptionInExpression[NumberFormatException](
- cast("string", dataType), "invalid input syntax for type numeric: string")
- checkExceptionInExpression[NumberFormatException](
- cast("123-string", dataType), "invalid input syntax for type numeric: 123-string")
- checkExceptionInExpression[NumberFormatException](
- cast("2020-07-19", dataType), "invalid input syntax for type numeric: 2020-07-19")
- checkExceptionInExpression[NumberFormatException](
- cast("1.23", dataType), "invalid input syntax for type numeric: 1.23")
- }
+/**
+ * Test suite for data type casting expression [[AnsiCast]] with ANSI mode enabled.
+ */
+class AnsiCastSuiteWithAnsiModeOn extends CastSuiteBase {
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ SQLConf.get.setConf(SQLConf.ANSI_ENABLED, true)
Review comment:
For `SQL.get`, when there is no active SparkSession, it actually creates a new SQLConf instance per thread:
```
/**
* Default config. Only used when there is no active SparkSession for the thread.
* See [[get]] for more information.
*/
private lazy val fallbackConf = new ThreadLocal[SQLConf] {
override def initialValue: SQLConf = new SQLConf
}
```
Thus we don't need to worry about modifying the SQLConf will break the other concurrent tests here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722545692
**[Test build #130650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130650/testReport)** for PR 30260 at commit [`d514f86`](https://github.com/apache/spark/commit/d514f86c268df53bd473009ef387564ddfef0bb6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729724901
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35888/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729913075
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722410969
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35261/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722546948
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722906769
Thanks for the request, @gengliangwang ! I'll review this later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729419446
**[Test build #131250 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131250/testReport)** for PR 30260 at commit [`1d57a24`](https://github.com/apache/spark/commit/1d57a24dfd457266a453f499f036456e0211ae09).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729454825
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35853/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726139244
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35609/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724144195
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35405/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724144184
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724129460
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35405/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729508843
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726275302
**[Test build #131003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131003/testReport)** for PR 30260 at commit [`ce0e775`](https://github.com/apache/spark/commit/ce0e7757927dc8082ce09d33a9f08ac0f2410169).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725619864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729876872
**[Test build #131300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131300/testReport)** for PR 30260 at commit [`6003bef`](https://github.com/apache/spark/commit/6003bef62c59d623ca59e51532b4fa5006c56fa2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725396795
**[Test build #130937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130937/testReport)** for PR 30260 at commit [`e6faf4b`](https://github.com/apache/spark/commit/e6faf4b64a32159116852d446ea9675c6325298a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725494779
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729508854
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131250/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu closed pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
maropu closed pull request #30260:
URL: https://github.com/apache/spark/pull/30260
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724270228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722423235
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35261/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729867859
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722385921
**[Test build #130650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130650/testReport)** for PR 30260 at commit [`d514f86`](https://github.com/apache/spark/commit/d514f86c268df53bd473009ef387564ddfef0bb6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-730041716
Thanks! Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722212961
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722210109
**[Test build #130642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130642/testReport)** for PR 30260 at commit [`fe0fe48`](https://github.com/apache/spark/commit/fe0fe4872c54550b44ab9458edb63d91d78b2ae0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729724935
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35888/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725396795
**[Test build #130937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130937/testReport)** for PR 30260 at commit [`e6faf4b`](https://github.com/apache/spark/commit/e6faf4b64a32159116852d446ea9675c6325298a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r521198497
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1753,6 +1759,12 @@ case class Cast(child: Expression, dataType: DataType, timeZoneId: Option[String
copy(timeZoneId = Option(timeZoneId))
override protected val ansiEnabled: Boolean = SQLConf.get.ansiEnabled
+
+ override def canCast(from: DataType, to: DataType): Boolean = if (ansiEnabled) {
Review comment:
well, then we need to mention about the behavior of throwing overflow exceptions when ANSI flag enabled. I will add some content in the `sql-ref-ansi-compliance.md`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722210109
**[Test build #130642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130642/testReport)** for PR 30260 at commit [`fe0fe48`](https://github.com/apache/spark/commit/fe0fe4872c54550b44ab9458edb63d91d78b2ae0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726154161
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35609/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r521320971
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1396,14 +1337,125 @@ class CastSuite extends CastSuiteBase {
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
}
+
+ test("SPARK-22825 Cast array to string") {
Review comment:
I would prefer keeping them unchanged in this PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724144184
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-730042825
Thank you, @gengliangwang and all!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-730548920
@revans2 Thanks for reporting the problem. Actually, I am working on another PR to give suggestions for the disallowed ANSI cast. It will be sent out today and I will ping you in the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r522043455
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1770,6 +1782,110 @@ case class AnsiCast(child: Expression, dataType: DataType, timeZoneId: Option[St
copy(timeZoneId = Option(timeZoneId))
override protected val ansiEnabled: Boolean = true
+
+ override def canCast(from: DataType, to: DataType): Boolean = AnsiCast.canCast(from, to)
+}
+
+object AnsiCast {
+ /**
+ * As per section 6.13 "cast specification" in "Information technology — Database languages " +
Review comment:
Nice update. Thanks!
##########
File path: docs/sql-ref-ansi-compliance.md
##########
@@ -107,6 +107,22 @@ SELECT * FROM t;
+---+
```
+The valid combinations of target data type and source data type in a `Cast` expression are given by the following table.
+“Y” indicates that the combination is syntactically valid without restriction and “N” indicates that the combination is not valid.
+
+| From\To | Numeric | String | Date | Timestamp | Interval | Boolean | Binary | Array | Map | Struct |
+|-----------|---------|--------|------|-----------|----------|---------|--------|-------|-----|--------|
+| Numeric | Y | Y | N | N | N | Y | N | N | N | N |
+| String | Y | Y | Y | Y | Y | Y | Y | N | N | N |
+| Date | N | Y | Y | Y | N | N | N | N | N | N |
+| Timestamp | N | Y | Y | Y | N | N | N | N | N | N |
+| Interval | N | Y | N | N | Y | N | N | N | N | N |
+| Boolean | Y | Y | N | N | N | Y | N | N | N | N |
+| Binary | Y | N | N | N | N | N | Y | N | N | N |
+| Array | N | N | N | N | N | N | N | Y | N | N |
+| Map | N | N | N | N | N | N | N | N | Y | N |
+| Struct | N | N | N | N | N | N | N | N | N | Y |
Review comment:
How about following the wording in the data type document, e.g., `String` -> `StringType` https://spark.apache.org/docs/3.0.1/sql-ref-datatypes.html
##########
File path: docs/sql-ref-ansi-compliance.md
##########
@@ -107,6 +107,22 @@ SELECT * FROM t;
+---+
```
+The valid combinations of target data type and source data type in a `Cast` expression are given by the following table.
Review comment:
nit: `Cast` expression -> `CAST` syntax ?
##########
File path: docs/sql-ref-ansi-compliance.md
##########
@@ -107,6 +107,22 @@ SELECT * FROM t;
+---+
```
+The valid combinations of target data type and source data type in a `Cast` expression are given by the following table.
Review comment:
How about moving this statements into L61-62?
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -635,16 +585,74 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
checkEvaluation(cast("", BooleanType), null)
}
+ private def checkInvalidCastFromNumericType(to: DataType): Unit = {
+ assert(cast(1.toByte, to).checkInputDataTypes().isFailure)
+ assert(cast(1.toShort, to).checkInputDataTypes().isFailure)
+ assert(cast(1, to).checkInputDataTypes().isFailure)
+ assert(cast(1L, to).checkInputDataTypes().isFailure)
+ assert(cast(1.0.toFloat, to).checkInputDataTypes().isFailure)
+ assert(cast(1.0, to).checkInputDataTypes().isFailure)
+ }
+
test("SPARK-16729 type checking for casting to date type") {
assert(cast("1234", DateType).checkInputDataTypes().isSuccess)
assert(cast(new Timestamp(1), DateType).checkInputDataTypes().isSuccess)
assert(cast(false, DateType).checkInputDataTypes().isFailure)
- assert(cast(1.toByte, DateType).checkInputDataTypes().isFailure)
- assert(cast(1.toShort, DateType).checkInputDataTypes().isFailure)
- assert(cast(1, DateType).checkInputDataTypes().isFailure)
- assert(cast(1L, DateType).checkInputDataTypes().isFailure)
- assert(cast(1.0.toFloat, DateType).checkInputDataTypes().isFailure)
- assert(cast(1.0, DateType).checkInputDataTypes().isFailure)
+ checkInvalidCastFromNumericType(DateType)
+ }
+
+ test("ANSI mode: disallow type conversions between Numeric types and Timestamp type") {
+ import DataTypeTestUtils.numericTypes
+ withSQLConf(ansiConfForExceptionOnOverflow) {
Review comment:
Is this test related to overflow tests? (Seems the config name suggests so)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1770,6 +1782,110 @@ case class AnsiCast(child: Expression, dataType: DataType, timeZoneId: Option[St
copy(timeZoneId = Option(timeZoneId))
override protected val ansiEnabled: Boolean = true
+
+ override def canCast(from: DataType, to: DataType): Boolean = AnsiCast.canCast(from, to)
+}
+
+object AnsiCast {
+ /**
+ * As per section 6.13 "cast specification" in "Information technology — Database languages " +
+ * "- SQL — Part 2: Foundation (SQL/Foundation)":
+ * If the <cast operand> is a <value expression>, then the valid combinations of TD and SD
+ * in a <cast specification> are given by the following table. “Y” indicates that the
+ * combination is syntactically valid without restriction; “M” indicates that the combination
+ * is valid subject to other Syntax Rules in this Sub- clause being satisfied; and “N” indicates
+ * that the combination is not valid:
+ * SD TD
+ * EN AN C D T TS YM DT BO UDT B RT CT RW
+ * EN Y Y Y N N N M M N M N M N N
+ * AN Y Y Y N N N N N N M N M N N
+ * C Y Y Y Y Y Y Y Y Y M N M N N
+ * D N N Y Y N Y N N N M N M N N
+ * T N N Y N Y Y N N N M N M N N
+ * TS N N Y Y Y Y N N N M N M N N
+ * YM M N Y N N N Y N N M N M N N
+ * DT M N Y N N N N Y N M N M N N
+ * BO N N Y N N N N N Y M N M N N
+ * UDT M M M M M M M M M M M M M N
+ * B N N N N N N N N N M Y M N N
+ * RT M M M M M M M M M M M M N N
+ * CT N N N N N N N N N M N N M N
+ * RW N N N N N N N N N N N N N M
+ *
+ * Where:
+ * EN = Exact Numeric
+ * AN = Approximate Numeric
+ * C = Character (Fixed- or Variable-Length, or Character Large Object)
+ * D = Date
+ * T = Time
+ * TS = Timestamp
+ * YM = Year-Month Interval
+ * DT = Day-Time Interval
+ * BO = Boolean
+ * UDT = User-Defined Type
+ * B = Binary (Fixed- or Variable-Length or Binary Large Object)
+ * RT = Reference type
+ * CT = Collection type
+ * RW = Row type
+ *
+ * Spark's ANSI mode follows the syntax rules, except it specially allow the following
+ * straightforward type conversions which are disallowed as per the SQL standard:
+ * - Numeric <=> Boolean
+ * - String <=> Binary
Review comment:
How about describing this in `sql-ref-ansi-compliance.md`, too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-730024656
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] revans2 commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
revans2 commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-730556431
@gengliangwang sounds good. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729708199
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35888/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722385921
**[Test build #130650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130650/testReport)** for PR 30260 at commit [`d514f86`](https://github.com/apache/spark/commit/d514f86c268df53bd473009ef387564ddfef0bb6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725460099
@maropu thanks for the suggestions. I have updated the code. Please take another look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729466242
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35853/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725451186
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35543/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729681853
**[Test build #131283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131283/testReport)** for PR 30260 at commit [`6003bef`](https://github.com/apache/spark/commit/6003bef62c59d623ca59e51532b4fa5006c56fa2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-728638644
kindly ping @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725494760
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35546/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722231521
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35252/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724270228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729508843
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] revans2 commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
revans2 commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-730541975
I just noticed that the error messages when trying to cast between timestamps and numeric types is wrong now in ANSI mode.
```
org.apache.spark.sql.AnalysisException: cannot resolve 'CAST(`c0` AS TIMESTAMP)' due to data type mismatch: cannot cast int to timestamp,you can enable the casting by setting spark.sql.legacy.allowCastNumericToTimestamp to true,but we strongly recommend using function TIMESTAMP_SECONDS/TIMESTAMP_MILLIS/TIMESTAMP_MICROS instead.;;
```
`spark.sql.legacy.allowCastNumericToTimestamp` has no impact on ANSI casts after this change. It looks like a fairly simple fix.
https://github.com/apache/spark/blob/3695e997d5d436be086235505bbb030c87ae8eef/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L276
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725459751
**[Test build #130941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130941/testReport)** for PR 30260 at commit [`33452cd`](https://github.com/apache/spark/commit/33452cd11017efcedd4ef2ad2bdff5741e4f403a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724103719
**[Test build #130796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130796/testReport)** for PR 30260 at commit [`18b49bf`](https://github.com/apache/spark/commit/18b49bf7dbd50b6bbd47e5f5389f1ceff41a1be3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-730024656
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725547759
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726154183
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729466252
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729867859
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725459751
**[Test build #130941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130941/testReport)** for PR 30260 at commit [`33452cd`](https://github.com/apache/spark/commit/33452cd11017efcedd4ef2ad2bdff5741e4f403a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725482523
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35546/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729419446
**[Test build #131250 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131250/testReport)** for PR 30260 at commit [`1d57a24`](https://github.com/apache/spark/commit/1d57a24dfd457266a453f499f036456e0211ae09).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722211696
I am still adding more test case, mark it as WIP for now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724103719
**[Test build #130796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130796/testReport)** for PR 30260 at commit [`18b49bf`](https://github.com/apache/spark/commit/18b49bf7dbd50b6bbd47e5f5389f1ceff41a1be3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-730023716
**[Test build #131300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131300/testReport)** for PR 30260 at commit [`6003bef`](https://github.com/apache/spark/commit/6003bef62c59d623ca59e51532b4fa5006c56fa2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class AnsiCastSuiteWithAnsiModeOn extends AnsiCastSuiteBase `
* `class AnsiCastSuiteWithAnsiModeOff extends AnsiCastSuiteBase `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729681853
**[Test build #131283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131283/testReport)** for PR 30260 at commit [`6003bef`](https://github.com/apache/spark/commit/6003bef62c59d623ca59e51532b4fa5006c56fa2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722423256
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729896291
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35902/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726276564
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729872517
retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729508413
**[Test build #131250 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131250/testReport)** for PR 30260 at commit [`1d57a24`](https://github.com/apache/spark/commit/1d57a24dfd457266a453f499f036456e0211ae09).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `abstract class AnsiCastSuiteBase extends CastSuiteBase `
* `class CastSuiteWithAnsiModeOn extends AnsiCastSuiteBase `
* `class AnsiCastSuiteWithAnsiModeOn extends CastSuiteBase `
* `class AnsiCastSuiteWithAnsiModeOff extends CastSuiteBase `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-726154183
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729867872
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131283/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r525912257
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1396,93 +1355,199 @@ class CastSuite extends CastSuiteBase {
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
}
+
+ test("SPARK-22825 Cast array to string") {
+ val ret1 = cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType)
+ checkEvaluation(ret1, "[1, 2, 3, 4, 5]")
+ val ret2 = cast(Literal.create(Array("ab", "cde", "f")), StringType)
+ checkEvaluation(ret2, "[ab, cde, f]")
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret3 = cast(Literal.create(Array("ab", null, "c")), StringType)
+ checkEvaluation(ret3, s"[ab,${if (omitNull) "" else " null"}, c]")
+ }
+ }
+ val ret4 =
+ cast(Literal.create(Array("ab".getBytes, "cde".getBytes, "f".getBytes)), StringType)
+ checkEvaluation(ret4, "[ab, cde, f]")
+ val ret5 = cast(
+ Literal.create(Array("2014-12-03", "2014-12-04", "2014-12-06").map(Date.valueOf)),
+ StringType)
+ checkEvaluation(ret5, "[2014-12-03, 2014-12-04, 2014-12-06]")
+ val ret6 = cast(
+ Literal.create(Array("2014-12-03 13:01:00", "2014-12-04 15:05:00")
+ .map(Timestamp.valueOf)),
+ StringType)
+ checkEvaluation(ret6, "[2014-12-03 13:01:00, 2014-12-04 15:05:00]")
+ val ret7 = cast(Literal.create(Array(Array(1, 2, 3), Array(4, 5))), StringType)
+ checkEvaluation(ret7, "[[1, 2, 3], [4, 5]]")
+ val ret8 = cast(
+ Literal.create(Array(Array(Array("a"), Array("b", "c")), Array(Array("d")))),
+ StringType)
+ checkEvaluation(ret8, "[[[a], [b, c]], [[d]]]")
+ }
+
+ test("SPARK-33291: Cast array with null elements to string") {
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret1 = cast(Literal.create(Array(null, null)), StringType)
+ checkEvaluation(
+ ret1,
+ s"[${if (omitNull) "" else "null"},${if (omitNull) "" else " null"}]")
+ }
+ }
+ }
+
+ test("SPARK-22973 Cast map to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create(Map(1 -> "a", 2 -> "b", 3 -> "c")), StringType)
+ checkEvaluation(ret1, s"${lb}1 -> a, 2 -> b, 3 -> c$rb")
+ val ret2 = cast(
+ Literal.create(Map("1" -> "a".getBytes, "2" -> null, "3" -> "c".getBytes)),
+ StringType)
+ checkEvaluation(ret2, s"${lb}1 -> a, 2 ->${if (legacyCast) "" else " null"}, 3 -> c$rb")
+ val ret3 = cast(
+ Literal.create(Map(
+ 1 -> Date.valueOf("2014-12-03"),
+ 2 -> Date.valueOf("2014-12-04"),
+ 3 -> Date.valueOf("2014-12-05"))),
+ StringType)
+ checkEvaluation(ret3, s"${lb}1 -> 2014-12-03, 2 -> 2014-12-04, 3 -> 2014-12-05$rb")
+ val ret4 = cast(
+ Literal.create(Map(
+ 1 -> Timestamp.valueOf("2014-12-03 13:01:00"),
+ 2 -> Timestamp.valueOf("2014-12-04 15:05:00"))),
+ StringType)
+ checkEvaluation(ret4, s"${lb}1 -> 2014-12-03 13:01:00, 2 -> 2014-12-04 15:05:00$rb")
+ val ret5 = cast(
+ Literal.create(Map(
+ 1 -> Array(1, 2, 3),
+ 2 -> Array(4, 5, 6))),
+ StringType)
+ checkEvaluation(ret5, s"${lb}1 -> [1, 2, 3], 2 -> [4, 5, 6]$rb")
+ }
+ }
+ }
+
+ test("SPARK-22981 Cast struct to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create((1, "a", 0.1)), StringType)
+ checkEvaluation(ret1, s"${lb}1, a, 0.1$rb")
+ val ret2 = cast(Literal.create(Tuple3[Int, String, String](1, null, "a")), StringType)
+ checkEvaluation(ret2, s"${lb}1,${if (legacyCast) "" else " null"}, a$rb")
+ val ret3 = cast(Literal.create(
+ (Date.valueOf("2014-12-03"), Timestamp.valueOf("2014-12-03 15:05:00"))), StringType)
+ checkEvaluation(ret3, s"${lb}2014-12-03, 2014-12-03 15:05:00$rb")
+ val ret4 = cast(Literal.create(((1, "a"), 5, 0.1)), StringType)
+ checkEvaluation(ret4, s"$lb${lb}1, a$rb, 5, 0.1$rb")
+ val ret5 = cast(Literal.create((Seq(1, 2, 3), "a", 0.1)), StringType)
+ checkEvaluation(ret5, s"$lb[1, 2, 3], a, 0.1$rb")
+ val ret6 = cast(Literal.create((1, Map(1 -> "a", 2 -> "b", 3 -> "c"))), StringType)
+ checkEvaluation(ret6, s"${lb}1, ${lb}1 -> a, 2 -> b, 3 -> c$rb$rb")
+ }
+ }
+ }
+
+ test("SPARK-33291: Cast struct with null elements to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create(Tuple2[String, String](null, null)), StringType)
+ checkEvaluation(
+ ret1,
+ s"$lb${if (legacyCast) "" else "null"},${if (legacyCast) "" else " null"}$rb")
+ }
+ }
+ }
+
+ test("data type casting II") {
+ checkEvaluation(
+ cast(cast(cast(cast(cast(cast("5", ByteType), TimestampType),
+ DecimalType.SYSTEM_DEFAULT), LongType), StringType), ShortType),
+ 5.toShort)
+ checkEvaluation(
+ cast(cast(cast(cast(cast(cast("5", TimestampType, UTC_OPT), ByteType),
+ DecimalType.SYSTEM_DEFAULT), LongType), StringType), ShortType),
+ null)
+ checkEvaluation(cast(cast(cast(cast(cast(cast("5", DecimalType.SYSTEM_DEFAULT),
+ ByteType), TimestampType), LongType), StringType), ShortType),
+ 5.toShort)
+ }
+
+ test("Cast from double II") {
+ checkEvaluation(cast(cast(1.toDouble, TimestampType), DoubleType), 1.toDouble)
+ }
}
/**
- * Test suite for data type casting expression [[AnsiCast]].
+ * Test suite for data type casting expression [[Cast]] with ANSI mode disabled.
*/
-class AnsiCastSuite extends CastSuiteBase {
- // It is not required to set SQLConf.ANSI_ENABLED as true for testing numeric overflow.
- override protected def requiredAnsiEnabledForOverflowTestCases: Boolean = false
+class CastSuiteWithAnsiModeOn extends AnsiCastSuiteBase {
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ SQLConf.get.setConf(SQLConf.ANSI_ENABLED, true)
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ SQLConf.get.unsetConf(SQLConf.ANSI_ENABLED)
+ }
override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
v match {
- case lit: Expression => AnsiCast(lit, targetType, timeZoneId)
- case _ => AnsiCast(Literal(v), targetType, timeZoneId)
+ case lit: Expression => Cast(lit, targetType, timeZoneId)
+ case _ => Cast(Literal(v), targetType, timeZoneId)
}
}
+}
- test("cast from invalid string to numeric should throw NumberFormatException") {
- // cast to IntegerType
- Seq(IntegerType, ShortType, ByteType, LongType).foreach { dataType =>
- val array = Literal.create(Seq("123", "true", "f", null),
- ArrayType(StringType, containsNull = true))
- checkExceptionInExpression[NumberFormatException](
- cast(array, ArrayType(dataType, containsNull = true)),
- "invalid input syntax for type numeric: true")
- checkExceptionInExpression[NumberFormatException](
- cast("string", dataType), "invalid input syntax for type numeric: string")
- checkExceptionInExpression[NumberFormatException](
- cast("123-string", dataType), "invalid input syntax for type numeric: 123-string")
- checkExceptionInExpression[NumberFormatException](
- cast("2020-07-19", dataType), "invalid input syntax for type numeric: 2020-07-19")
- checkExceptionInExpression[NumberFormatException](
- cast("1.23", dataType), "invalid input syntax for type numeric: 1.23")
- }
+/**
+ * Test suite for data type casting expression [[AnsiCast]] with ANSI mode enabled.
+ */
+class AnsiCastSuiteWithAnsiModeOn extends CastSuiteBase {
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ SQLConf.get.setConf(SQLConf.ANSI_ENABLED, true)
Review comment:
It's fragile to change global conf. How about we override `test` method and wrap the test funnc with `withSQLConf`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r522043708
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1770,6 +1782,110 @@ case class AnsiCast(child: Expression, dataType: DataType, timeZoneId: Option[St
copy(timeZoneId = Option(timeZoneId))
override protected val ansiEnabled: Boolean = true
+
+ override def canCast(from: DataType, to: DataType): Boolean = AnsiCast.canCast(from, to)
+}
+
+object AnsiCast {
+ /**
+ * As per section 6.13 "cast specification" in "Information technology — Database languages " +
+ * "- SQL — Part 2: Foundation (SQL/Foundation)":
+ * If the <cast operand> is a <value expression>, then the valid combinations of TD and SD
+ * in a <cast specification> are given by the following table. “Y” indicates that the
+ * combination is syntactically valid without restriction; “M” indicates that the combination
+ * is valid subject to other Syntax Rules in this Sub- clause being satisfied; and “N” indicates
+ * that the combination is not valid:
+ * SD TD
+ * EN AN C D T TS YM DT BO UDT B RT CT RW
+ * EN Y Y Y N N N M M N M N M N N
+ * AN Y Y Y N N N N N N M N M N N
+ * C Y Y Y Y Y Y Y Y Y M N M N N
+ * D N N Y Y N Y N N N M N M N N
+ * T N N Y N Y Y N N N M N M N N
+ * TS N N Y Y Y Y N N N M N M N N
+ * YM M N Y N N N Y N N M N M N N
+ * DT M N Y N N N N Y N M N M N N
+ * BO N N Y N N N N N Y M N M N N
+ * UDT M M M M M M M M M M M M M N
+ * B N N N N N N N N N M Y M N N
+ * RT M M M M M M M M M M M M N N
+ * CT N N N N N N N N N M N N M N
+ * RW N N N N N N N N N N N N N M
+ *
+ * Where:
+ * EN = Exact Numeric
+ * AN = Approximate Numeric
+ * C = Character (Fixed- or Variable-Length, or Character Large Object)
+ * D = Date
+ * T = Time
+ * TS = Timestamp
+ * YM = Year-Month Interval
+ * DT = Day-Time Interval
+ * BO = Boolean
+ * UDT = User-Defined Type
+ * B = Binary (Fixed- or Variable-Length or Binary Large Object)
+ * RT = Reference type
+ * CT = Collection type
+ * RW = Row type
+ *
+ * Spark's ANSI mode follows the syntax rules, except it specially allow the following
+ * straightforward type conversions which are disallowed as per the SQL standard:
+ * - Numeric <=> Boolean
+ * - String <=> Binary
Review comment:
How about describing the special case above in `sql-ref-ansi-compliance.md`, too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722246653
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729913075
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722212972
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130642/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-724269144
**[Test build #130796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130796/testReport)** for PR 30260 at commit [`18b49bf`](https://github.com/apache/spark/commit/18b49bf7dbd50b6bbd47e5f5389f1ceff41a1be3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725546539
**[Test build #130937 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130937/testReport)** for PR 30260 at commit [`e6faf4b`](https://github.com/apache/spark/commit/e6faf4b64a32159116852d446ea9675c6325298a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722211696
I am still adding more test case, mark it as WIP for now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r520238225
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -38,8 +38,11 @@ import org.apache.spark.unsafe.types.UTF8String
abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
+ // Whether this test suite is for checking AnsiCast or default Cast.
+ protected def isAnsiCast: Boolean
+
// Whether it is required to set SQLConf.ANSI_ENABLED as true for testing numeric overflow.
Review comment:
I feel the name `requiredAnsiEnabledForOverflowTestCases` is confusing, so could you rename it according to the current usage and update the comment above?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1770,6 +1782,64 @@ case class AnsiCast(child: Expression, dataType: DataType, timeZoneId: Option[St
copy(timeZoneId = Option(timeZoneId))
override protected val ansiEnabled: Boolean = true
+
+ override def canCast(from: DataType, to: DataType): Boolean = AnsiCast.canCast(from, to)
+}
+
+object AnsiCast {
Review comment:
Could you leave some comments to summarize the current behaivour of the ANSI explicit cast as described in the PR description (references `section 6.13 of the ANSI SQL standard` and differences from the standard, e.g., `Numeric <=> Boolean`) ?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1753,6 +1759,12 @@ case class Cast(child: Expression, dataType: DataType, timeZoneId: Option[String
copy(timeZoneId = Option(timeZoneId))
override protected val ansiEnabled: Boolean = SQLConf.get.ansiEnabled
+
+ override def canCast(from: DataType, to: DataType): Boolean = if (ansiEnabled) {
Review comment:
How about describing this new behaviour in the usage above of `ExpressionDescription`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722212961
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725464020
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-725618693
**[Test build #130941 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130941/testReport)** for PR 30260 at commit [`33452cd`](https://github.com/apache/spark/commit/33452cd11017efcedd4ef2ad2bdff5741e4f403a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722423256
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r526112623
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -930,150 +733,254 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
}
}
- test("Throw exception on casting out-of-range value to byte type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(ByteType)
- Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
- checkExceptionInExpression[ArithmeticException](cast(value, ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
- }
+ test("ANSI mode: Throw exception on casting out-of-range value to byte type") {
+ testIntMaxAndMin(ByteType)
+ Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, ByteType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
+ }
- Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
- checkEvaluation(cast(value, ByteType), value)
- checkEvaluation(cast(value.toString, ByteType), value)
- checkEvaluation(cast(Decimal(value.toString), ByteType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType), value)
- checkEvaluation(cast(Literal(value.toInt, DateType), ByteType), null)
- checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType), value)
- checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType), value)
- }
+ Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
+ checkEvaluation(cast(value, ByteType), value)
+ checkEvaluation(cast(value.toString, ByteType), value)
+ checkEvaluation(cast(Decimal(value.toString), ByteType), value)
+ checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType), value)
+ checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType), value)
}
}
- test("Throw exception on casting out-of-range value to short type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(ShortType)
- Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
- checkExceptionInExpression[ArithmeticException](cast(value, ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
- }
+ test("ANSI mode: Throw exception on casting out-of-range value to short type") {
+ testIntMaxAndMin(ShortType)
+ Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, ShortType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
+ }
- Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
- checkEvaluation(cast(value, ShortType), value)
- checkEvaluation(cast(value.toString, ShortType), value)
- checkEvaluation(cast(Decimal(value.toString), ShortType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType), value)
- checkEvaluation(cast(Literal(value.toInt, DateType), ShortType), null)
- checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType), value)
- checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType), value)
- }
+ Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
+ checkEvaluation(cast(value, ShortType), value)
+ checkEvaluation(cast(value.toString, ShortType), value)
+ checkEvaluation(cast(Decimal(value.toString), ShortType), value)
+ checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType), value)
+ checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType), value)
}
}
- test("Throw exception on casting out-of-range value to int type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(IntegerType)
- testLongMaxAndMin(IntegerType)
+ test("ANSI mode: Throw exception on casting out-of-range value to int type") {
+ testIntMaxAndMin(IntegerType)
+ testLongMaxAndMin(IntegerType)
- Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
- checkEvaluation(cast(value, IntegerType), value)
- checkEvaluation(cast(value.toString, IntegerType), value)
- checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), IntegerType), value)
- checkEvaluation(cast(Literal(value * 1.0, DoubleType), IntegerType), value)
- }
- checkEvaluation(cast(Int.MaxValue + 0.9D, IntegerType), Int.MaxValue)
- checkEvaluation(cast(Int.MinValue - 0.9D, IntegerType), Int.MinValue)
+ Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
+ checkEvaluation(cast(value, IntegerType), value)
+ checkEvaluation(cast(value.toString, IntegerType), value)
+ checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
+ checkEvaluation(cast(Literal(value * 1.0, DoubleType), IntegerType), value)
}
+ checkEvaluation(cast(Int.MaxValue + 0.9D, IntegerType), Int.MaxValue)
+ checkEvaluation(cast(Int.MinValue - 0.9D, IntegerType), Int.MinValue)
}
- test("Throw exception on casting out-of-range value to long type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testLongMaxAndMin(LongType)
+ test("ANSI mode: Throw exception on casting out-of-range value to long type") {
+ testLongMaxAndMin(LongType)
- Seq(Long.MaxValue, 0, Long.MinValue).foreach { value =>
- checkEvaluation(cast(value, LongType), value)
- checkEvaluation(cast(value.toString, LongType), value)
- checkEvaluation(cast(Decimal(value.toString), LongType), value)
- checkEvaluation(cast(Literal(value, TimestampType), LongType),
- Math.floorDiv(value, MICROS_PER_SECOND))
- }
- checkEvaluation(cast(Long.MaxValue + 0.9F, LongType), Long.MaxValue)
- checkEvaluation(cast(Long.MinValue - 0.9F, LongType), Long.MinValue)
- checkEvaluation(cast(Long.MaxValue + 0.9D, LongType), Long.MaxValue)
- checkEvaluation(cast(Long.MinValue - 0.9D, LongType), Long.MinValue)
+ Seq(Long.MaxValue, 0, Long.MinValue).foreach { value =>
+ checkEvaluation(cast(value, LongType), value)
+ checkEvaluation(cast(value.toString, LongType), value)
+ checkEvaluation(cast(Decimal(value.toString), LongType), value)
}
+ checkEvaluation(cast(Long.MaxValue + 0.9F, LongType), Long.MaxValue)
+ checkEvaluation(cast(Long.MinValue - 0.9F, LongType), Long.MinValue)
+ checkEvaluation(cast(Long.MaxValue + 0.9D, LongType), Long.MaxValue)
+ checkEvaluation(cast(Long.MinValue - 0.9D, LongType), Long.MinValue)
}
-}
-/**
- * Test suite for data type casting expression [[Cast]].
- */
-class CastSuite extends CastSuiteBase {
- // It is required to set SQLConf.ANSI_ENABLED as true for testing numeric overflow.
- override protected def requiredAnsiEnabledForOverflowTestCases: Boolean = true
+ test("ANSI mode: Throw exception on casting out-of-range value to decimal type") {
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal("134.12"), DecimalType(3, 2)), "cannot be represented")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(BigDecimal(134.12)), DecimalType(3, 2)), "cannot be represented")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(134.12), DecimalType(3, 2)), "cannot be represented")
+ }
- override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
- v match {
- case lit: Expression => Cast(lit, targetType, timeZoneId)
- case _ => Cast(Literal(v), targetType, timeZoneId)
+ test("ANSI mode: disallow type conversions between Numeric types and Timestamp type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(TimestampType)
+ val timestampLiteral = Literal(1L, TimestampType)
+ numericTypes.foreach { numericType =>
+ assert(cast(timestampLiteral, numericType).checkInputDataTypes().isFailure)
}
}
- test("cast from int") {
- checkCast(0, false)
- checkCast(1, true)
- checkCast(-5, true)
- checkCast(1, 1.toByte)
- checkCast(1, 1.toShort)
- checkCast(1, 1)
- checkCast(1, 1.toLong)
- checkCast(1, 1.0f)
- checkCast(1, 1.0)
- checkCast(123, "123")
-
- checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
- checkEvaluation(cast(123, DecimalType(3, 0)), Decimal(123))
- checkEvaluation(cast(123, DecimalType(3, 1)), null)
- checkEvaluation(cast(123, DecimalType(2, 0)), null)
+ test("ANSI mode: disallow type conversions between Numeric types and Date type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(DateType)
+ val dateLiteral = Literal(1, DateType)
+ numericTypes.foreach { numericType =>
+ assert(cast(dateLiteral, numericType).checkInputDataTypes().isFailure)
+ }
}
- test("cast from long") {
- checkCast(0L, false)
- checkCast(1L, true)
- checkCast(-5L, true)
- checkCast(1L, 1.toByte)
- checkCast(1L, 1.toShort)
- checkCast(1L, 1)
- checkCast(1L, 1.toLong)
- checkCast(1L, 1.0f)
- checkCast(1L, 1.0)
- checkCast(123L, "123")
+ test("ANSI mode: disallow type conversions between Numeric types and Binary type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(BinaryType)
+ val binaryLiteral = Literal(new Array[Byte](1.toByte), BinaryType)
+ numericTypes.foreach { numericType =>
+ assert(cast(binaryLiteral, numericType).checkInputDataTypes().isFailure)
+ }
+ }
- checkEvaluation(cast(123L, DecimalType.USER_DEFAULT), Decimal(123))
- checkEvaluation(cast(123L, DecimalType(3, 0)), Decimal(123))
- checkEvaluation(cast(123L, DecimalType(3, 1)), null)
+ test("ANSI mode: disallow type conversions between Datatime types and Boolean types") {
+ val timestampLiteral = Literal(1L, TimestampType)
+ assert(cast(timestampLiteral, BooleanType).checkInputDataTypes().isFailure)
+ val dateLiteral = Literal(1, DateType)
+ assert(cast(dateLiteral, BooleanType).checkInputDataTypes().isFailure)
- checkEvaluation(cast(123L, DecimalType(2, 0)), null)
+ val booleanLiteral = Literal(true, BooleanType)
+ assert(cast(booleanLiteral, TimestampType).checkInputDataTypes().isFailure)
+ assert(cast(booleanLiteral, DateType).checkInputDataTypes().isFailure)
}
- test("cast from int 2") {
- checkEvaluation(cast(1, LongType), 1.toLong)
-
- checkEvaluation(cast(cast(1000, TimestampType), LongType), 1000.toLong)
- checkEvaluation(cast(cast(-1200, TimestampType), LongType), -1200.toLong)
+ test("ANSI mode: disallow casting complex types as String type") {
+ assert(cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType).checkInputDataTypes().isFailure)
+ assert(cast(Literal.create(Map(1 -> "a")), StringType).checkInputDataTypes().isFailure)
+ assert(cast(Literal.create((1, "a", 0.1)), StringType).checkInputDataTypes().isFailure)
+ }
- checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
+ test("cast from invalid string to numeric should throw NumberFormatException") {
+ // cast to IntegerType
+ Seq(IntegerType, ShortType, ByteType, LongType).foreach { dataType =>
+ val array = Literal.create(Seq("123", "true", "f", null),
+ ArrayType(StringType, containsNull = true))
+ checkExceptionInExpression[NumberFormatException](
+ cast(array, ArrayType(dataType, containsNull = true)),
+ "invalid input syntax for type numeric: true")
+ checkExceptionInExpression[NumberFormatException](
+ cast("string", dataType), "invalid input syntax for type numeric: string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("123-string", dataType), "invalid input syntax for type numeric: 123-string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("2020-07-19", dataType), "invalid input syntax for type numeric: 2020-07-19")
+ checkExceptionInExpression[NumberFormatException](
+ cast("1.23", dataType), "invalid input syntax for type numeric: 1.23")
+ }
+
+ Seq(DoubleType, FloatType, DecimalType.USER_DEFAULT).foreach { dataType =>
+ checkExceptionInExpression[NumberFormatException](
+ cast("string", dataType), "invalid input syntax for type numeric: string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("123.000.00", dataType), "invalid input syntax for type numeric: 123.000.00")
+ checkExceptionInExpression[NumberFormatException](
+ cast("abc.com", dataType), "invalid input syntax for type numeric: abc.com")
+ }
+ }
+
+ test("Fast fail for cast string type to decimal type in ansi mode") {
+ checkEvaluation(cast("12345678901234567890123456789012345678", DecimalType(38, 0)),
+ Decimal("12345678901234567890123456789012345678"))
+ checkExceptionInExpression[ArithmeticException](
+ cast("123456789012345678901234567890123456789", DecimalType(38, 0)),
+ "out of decimal type range")
+ checkExceptionInExpression[ArithmeticException](
+ cast("12345678901234567890123456789012345678", DecimalType(38, 1)),
+ "cannot be represented as Decimal(38, 1)")
+
+ checkEvaluation(cast("0.00000000000000000000000000000000000001", DecimalType(38, 0)),
+ Decimal("0"))
+ checkEvaluation(cast("0.00000000000000000000000000000000000000000001", DecimalType(38, 0)),
+ Decimal("0"))
+ checkEvaluation(cast("0.00000000000000000000000000000000000001", DecimalType(38, 18)),
+ Decimal("0E-18"))
+ checkEvaluation(cast("6E-120", DecimalType(38, 0)),
+ Decimal("0"))
+
+ checkEvaluation(cast("6E+37", DecimalType(38, 0)),
+ Decimal("60000000000000000000000000000000000000"))
+ checkExceptionInExpression[ArithmeticException](
+ cast("6E+38", DecimalType(38, 0)),
+ "out of decimal type range")
+ checkExceptionInExpression[ArithmeticException](
+ cast("6E+37", DecimalType(38, 1)),
+ "cannot be represented as Decimal(38, 1)")
+
+ checkExceptionInExpression[NumberFormatException](
+ cast("abcd", DecimalType(38, 1)),
+ "invalid input syntax for type numeric")
+ }
+}
+
+/**
+ * Test suite for data type casting expression [[Cast]].
+ */
+class CastSuite extends CastSuiteBase {
+
+ override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
+ v match {
+ case lit: Expression => Cast(lit, targetType, timeZoneId)
+ case _ => Cast(Literal(v), targetType, timeZoneId)
+ }
+ }
+
+ test("null cast II") {
+ import DataTypeTestUtils._
+
+ checkNullCast(DateType, BooleanType)
+ checkNullCast(TimestampType, BooleanType)
+ checkNullCast(BooleanType, TimestampType)
+ numericTypes.foreach(dt => checkNullCast(dt, TimestampType))
+ numericTypes.foreach(dt => checkNullCast(TimestampType, dt))
+ numericTypes.foreach(dt => checkNullCast(DateType, dt))
+ }
+
+ test("cast from int") {
+ checkCast(0, false)
+ checkCast(1, true)
+ checkCast(-5, true)
+ checkCast(1, 1.toByte)
+ checkCast(1, 1.toShort)
Review comment:
Good catch. I have updated the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729466252
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r525909826
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -930,150 +733,254 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
}
}
- test("Throw exception on casting out-of-range value to byte type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(ByteType)
- Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
- checkExceptionInExpression[ArithmeticException](cast(value, ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
- }
+ test("ANSI mode: Throw exception on casting out-of-range value to byte type") {
+ testIntMaxAndMin(ByteType)
+ Seq(Byte.MaxValue + 1, Byte.MinValue - 1).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, ByteType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ByteType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ByteType), "overflow")
+ }
- Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
- checkEvaluation(cast(value, ByteType), value)
- checkEvaluation(cast(value.toString, ByteType), value)
- checkEvaluation(cast(Decimal(value.toString), ByteType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), ByteType), value)
- checkEvaluation(cast(Literal(value.toInt, DateType), ByteType), null)
- checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType), value)
- checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType), value)
- }
+ Seq(Byte.MaxValue, 0.toByte, Byte.MinValue).foreach { value =>
+ checkEvaluation(cast(value, ByteType), value)
+ checkEvaluation(cast(value.toString, ByteType), value)
+ checkEvaluation(cast(Decimal(value.toString), ByteType), value)
+ checkEvaluation(cast(Literal(value.toFloat, FloatType), ByteType), value)
+ checkEvaluation(cast(Literal(value.toDouble, DoubleType), ByteType), value)
}
}
- test("Throw exception on casting out-of-range value to short type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(ShortType)
- Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
- checkExceptionInExpression[ArithmeticException](cast(value, ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
- checkExceptionInExpression[ArithmeticException](
- cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
- }
+ test("ANSI mode: Throw exception on casting out-of-range value to short type") {
+ testIntMaxAndMin(ShortType)
+ Seq(Short.MaxValue + 1, Short.MinValue - 1).foreach { value =>
+ checkExceptionInExpression[ArithmeticException](cast(value, ShortType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toFloat, FloatType), ShortType), "overflow")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(value.toDouble, DoubleType), ShortType), "overflow")
+ }
- Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
- checkEvaluation(cast(value, ShortType), value)
- checkEvaluation(cast(value.toString, ShortType), value)
- checkEvaluation(cast(Decimal(value.toString), ShortType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), ShortType), value)
- checkEvaluation(cast(Literal(value.toInt, DateType), ShortType), null)
- checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType), value)
- checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType), value)
- }
+ Seq(Short.MaxValue, 0.toShort, Short.MinValue).foreach { value =>
+ checkEvaluation(cast(value, ShortType), value)
+ checkEvaluation(cast(value.toString, ShortType), value)
+ checkEvaluation(cast(Decimal(value.toString), ShortType), value)
+ checkEvaluation(cast(Literal(value.toFloat, FloatType), ShortType), value)
+ checkEvaluation(cast(Literal(value.toDouble, DoubleType), ShortType), value)
}
}
- test("Throw exception on casting out-of-range value to int type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testIntMaxAndMin(IntegerType)
- testLongMaxAndMin(IntegerType)
+ test("ANSI mode: Throw exception on casting out-of-range value to int type") {
+ testIntMaxAndMin(IntegerType)
+ testLongMaxAndMin(IntegerType)
- Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
- checkEvaluation(cast(value, IntegerType), value)
- checkEvaluation(cast(value.toString, IntegerType), value)
- checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
- checkEvaluation(cast(Literal(value * MICROS_PER_SECOND, TimestampType), IntegerType), value)
- checkEvaluation(cast(Literal(value * 1.0, DoubleType), IntegerType), value)
- }
- checkEvaluation(cast(Int.MaxValue + 0.9D, IntegerType), Int.MaxValue)
- checkEvaluation(cast(Int.MinValue - 0.9D, IntegerType), Int.MinValue)
+ Seq(Int.MaxValue, 0, Int.MinValue).foreach { value =>
+ checkEvaluation(cast(value, IntegerType), value)
+ checkEvaluation(cast(value.toString, IntegerType), value)
+ checkEvaluation(cast(Decimal(value.toString), IntegerType), value)
+ checkEvaluation(cast(Literal(value * 1.0, DoubleType), IntegerType), value)
}
+ checkEvaluation(cast(Int.MaxValue + 0.9D, IntegerType), Int.MaxValue)
+ checkEvaluation(cast(Int.MinValue - 0.9D, IntegerType), Int.MinValue)
}
- test("Throw exception on casting out-of-range value to long type") {
- withSQLConf(SQLConf.ANSI_ENABLED.key -> requiredAnsiEnabledForOverflowTestCases.toString) {
- testLongMaxAndMin(LongType)
+ test("ANSI mode: Throw exception on casting out-of-range value to long type") {
+ testLongMaxAndMin(LongType)
- Seq(Long.MaxValue, 0, Long.MinValue).foreach { value =>
- checkEvaluation(cast(value, LongType), value)
- checkEvaluation(cast(value.toString, LongType), value)
- checkEvaluation(cast(Decimal(value.toString), LongType), value)
- checkEvaluation(cast(Literal(value, TimestampType), LongType),
- Math.floorDiv(value, MICROS_PER_SECOND))
- }
- checkEvaluation(cast(Long.MaxValue + 0.9F, LongType), Long.MaxValue)
- checkEvaluation(cast(Long.MinValue - 0.9F, LongType), Long.MinValue)
- checkEvaluation(cast(Long.MaxValue + 0.9D, LongType), Long.MaxValue)
- checkEvaluation(cast(Long.MinValue - 0.9D, LongType), Long.MinValue)
+ Seq(Long.MaxValue, 0, Long.MinValue).foreach { value =>
+ checkEvaluation(cast(value, LongType), value)
+ checkEvaluation(cast(value.toString, LongType), value)
+ checkEvaluation(cast(Decimal(value.toString), LongType), value)
}
+ checkEvaluation(cast(Long.MaxValue + 0.9F, LongType), Long.MaxValue)
+ checkEvaluation(cast(Long.MinValue - 0.9F, LongType), Long.MinValue)
+ checkEvaluation(cast(Long.MaxValue + 0.9D, LongType), Long.MaxValue)
+ checkEvaluation(cast(Long.MinValue - 0.9D, LongType), Long.MinValue)
}
-}
-/**
- * Test suite for data type casting expression [[Cast]].
- */
-class CastSuite extends CastSuiteBase {
- // It is required to set SQLConf.ANSI_ENABLED as true for testing numeric overflow.
- override protected def requiredAnsiEnabledForOverflowTestCases: Boolean = true
+ test("ANSI mode: Throw exception on casting out-of-range value to decimal type") {
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal("134.12"), DecimalType(3, 2)), "cannot be represented")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(BigDecimal(134.12)), DecimalType(3, 2)), "cannot be represented")
+ checkExceptionInExpression[ArithmeticException](
+ cast(Literal(134.12), DecimalType(3, 2)), "cannot be represented")
+ }
- override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
- v match {
- case lit: Expression => Cast(lit, targetType, timeZoneId)
- case _ => Cast(Literal(v), targetType, timeZoneId)
+ test("ANSI mode: disallow type conversions between Numeric types and Timestamp type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(TimestampType)
+ val timestampLiteral = Literal(1L, TimestampType)
+ numericTypes.foreach { numericType =>
+ assert(cast(timestampLiteral, numericType).checkInputDataTypes().isFailure)
}
}
- test("cast from int") {
- checkCast(0, false)
- checkCast(1, true)
- checkCast(-5, true)
- checkCast(1, 1.toByte)
- checkCast(1, 1.toShort)
- checkCast(1, 1)
- checkCast(1, 1.toLong)
- checkCast(1, 1.0f)
- checkCast(1, 1.0)
- checkCast(123, "123")
-
- checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
- checkEvaluation(cast(123, DecimalType(3, 0)), Decimal(123))
- checkEvaluation(cast(123, DecimalType(3, 1)), null)
- checkEvaluation(cast(123, DecimalType(2, 0)), null)
+ test("ANSI mode: disallow type conversions between Numeric types and Date type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(DateType)
+ val dateLiteral = Literal(1, DateType)
+ numericTypes.foreach { numericType =>
+ assert(cast(dateLiteral, numericType).checkInputDataTypes().isFailure)
+ }
}
- test("cast from long") {
- checkCast(0L, false)
- checkCast(1L, true)
- checkCast(-5L, true)
- checkCast(1L, 1.toByte)
- checkCast(1L, 1.toShort)
- checkCast(1L, 1)
- checkCast(1L, 1.toLong)
- checkCast(1L, 1.0f)
- checkCast(1L, 1.0)
- checkCast(123L, "123")
+ test("ANSI mode: disallow type conversions between Numeric types and Binary type") {
+ import DataTypeTestUtils.numericTypes
+ checkInvalidCastFromNumericType(BinaryType)
+ val binaryLiteral = Literal(new Array[Byte](1.toByte), BinaryType)
+ numericTypes.foreach { numericType =>
+ assert(cast(binaryLiteral, numericType).checkInputDataTypes().isFailure)
+ }
+ }
- checkEvaluation(cast(123L, DecimalType.USER_DEFAULT), Decimal(123))
- checkEvaluation(cast(123L, DecimalType(3, 0)), Decimal(123))
- checkEvaluation(cast(123L, DecimalType(3, 1)), null)
+ test("ANSI mode: disallow type conversions between Datatime types and Boolean types") {
+ val timestampLiteral = Literal(1L, TimestampType)
+ assert(cast(timestampLiteral, BooleanType).checkInputDataTypes().isFailure)
+ val dateLiteral = Literal(1, DateType)
+ assert(cast(dateLiteral, BooleanType).checkInputDataTypes().isFailure)
- checkEvaluation(cast(123L, DecimalType(2, 0)), null)
+ val booleanLiteral = Literal(true, BooleanType)
+ assert(cast(booleanLiteral, TimestampType).checkInputDataTypes().isFailure)
+ assert(cast(booleanLiteral, DateType).checkInputDataTypes().isFailure)
}
- test("cast from int 2") {
- checkEvaluation(cast(1, LongType), 1.toLong)
-
- checkEvaluation(cast(cast(1000, TimestampType), LongType), 1000.toLong)
- checkEvaluation(cast(cast(-1200, TimestampType), LongType), -1200.toLong)
+ test("ANSI mode: disallow casting complex types as String type") {
+ assert(cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType).checkInputDataTypes().isFailure)
+ assert(cast(Literal.create(Map(1 -> "a")), StringType).checkInputDataTypes().isFailure)
+ assert(cast(Literal.create((1, "a", 0.1)), StringType).checkInputDataTypes().isFailure)
+ }
- checkEvaluation(cast(123, DecimalType.USER_DEFAULT), Decimal(123))
+ test("cast from invalid string to numeric should throw NumberFormatException") {
+ // cast to IntegerType
+ Seq(IntegerType, ShortType, ByteType, LongType).foreach { dataType =>
+ val array = Literal.create(Seq("123", "true", "f", null),
+ ArrayType(StringType, containsNull = true))
+ checkExceptionInExpression[NumberFormatException](
+ cast(array, ArrayType(dataType, containsNull = true)),
+ "invalid input syntax for type numeric: true")
+ checkExceptionInExpression[NumberFormatException](
+ cast("string", dataType), "invalid input syntax for type numeric: string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("123-string", dataType), "invalid input syntax for type numeric: 123-string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("2020-07-19", dataType), "invalid input syntax for type numeric: 2020-07-19")
+ checkExceptionInExpression[NumberFormatException](
+ cast("1.23", dataType), "invalid input syntax for type numeric: 1.23")
+ }
+
+ Seq(DoubleType, FloatType, DecimalType.USER_DEFAULT).foreach { dataType =>
+ checkExceptionInExpression[NumberFormatException](
+ cast("string", dataType), "invalid input syntax for type numeric: string")
+ checkExceptionInExpression[NumberFormatException](
+ cast("123.000.00", dataType), "invalid input syntax for type numeric: 123.000.00")
+ checkExceptionInExpression[NumberFormatException](
+ cast("abc.com", dataType), "invalid input syntax for type numeric: abc.com")
+ }
+ }
+
+ test("Fast fail for cast string type to decimal type in ansi mode") {
+ checkEvaluation(cast("12345678901234567890123456789012345678", DecimalType(38, 0)),
+ Decimal("12345678901234567890123456789012345678"))
+ checkExceptionInExpression[ArithmeticException](
+ cast("123456789012345678901234567890123456789", DecimalType(38, 0)),
+ "out of decimal type range")
+ checkExceptionInExpression[ArithmeticException](
+ cast("12345678901234567890123456789012345678", DecimalType(38, 1)),
+ "cannot be represented as Decimal(38, 1)")
+
+ checkEvaluation(cast("0.00000000000000000000000000000000000001", DecimalType(38, 0)),
+ Decimal("0"))
+ checkEvaluation(cast("0.00000000000000000000000000000000000000000001", DecimalType(38, 0)),
+ Decimal("0"))
+ checkEvaluation(cast("0.00000000000000000000000000000000000001", DecimalType(38, 18)),
+ Decimal("0E-18"))
+ checkEvaluation(cast("6E-120", DecimalType(38, 0)),
+ Decimal("0"))
+
+ checkEvaluation(cast("6E+37", DecimalType(38, 0)),
+ Decimal("60000000000000000000000000000000000000"))
+ checkExceptionInExpression[ArithmeticException](
+ cast("6E+38", DecimalType(38, 0)),
+ "out of decimal type range")
+ checkExceptionInExpression[ArithmeticException](
+ cast("6E+37", DecimalType(38, 1)),
+ "cannot be represented as Decimal(38, 1)")
+
+ checkExceptionInExpression[NumberFormatException](
+ cast("abcd", DecimalType(38, 1)),
+ "invalid input syntax for type numeric")
+ }
+}
+
+/**
+ * Test suite for data type casting expression [[Cast]].
+ */
+class CastSuite extends CastSuiteBase {
+
+ override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
+ v match {
+ case lit: Expression => Cast(lit, targetType, timeZoneId)
+ case _ => Cast(Literal(v), targetType, timeZoneId)
+ }
+ }
+
+ test("null cast II") {
Review comment:
nit: `null cast #2`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30260: [WIP][SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-722246636
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35252/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #30260:
URL: https://github.com/apache/spark/pull/30260#discussion_r526112368
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1396,93 +1355,199 @@ class CastSuite extends CastSuiteBase {
checkEvaluation(cast("abcd", DecimalType(38, 1)), null)
}
+
+ test("SPARK-22825 Cast array to string") {
+ val ret1 = cast(Literal.create(Array(1, 2, 3, 4, 5)), StringType)
+ checkEvaluation(ret1, "[1, 2, 3, 4, 5]")
+ val ret2 = cast(Literal.create(Array("ab", "cde", "f")), StringType)
+ checkEvaluation(ret2, "[ab, cde, f]")
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret3 = cast(Literal.create(Array("ab", null, "c")), StringType)
+ checkEvaluation(ret3, s"[ab,${if (omitNull) "" else " null"}, c]")
+ }
+ }
+ val ret4 =
+ cast(Literal.create(Array("ab".getBytes, "cde".getBytes, "f".getBytes)), StringType)
+ checkEvaluation(ret4, "[ab, cde, f]")
+ val ret5 = cast(
+ Literal.create(Array("2014-12-03", "2014-12-04", "2014-12-06").map(Date.valueOf)),
+ StringType)
+ checkEvaluation(ret5, "[2014-12-03, 2014-12-04, 2014-12-06]")
+ val ret6 = cast(
+ Literal.create(Array("2014-12-03 13:01:00", "2014-12-04 15:05:00")
+ .map(Timestamp.valueOf)),
+ StringType)
+ checkEvaluation(ret6, "[2014-12-03 13:01:00, 2014-12-04 15:05:00]")
+ val ret7 = cast(Literal.create(Array(Array(1, 2, 3), Array(4, 5))), StringType)
+ checkEvaluation(ret7, "[[1, 2, 3], [4, 5]]")
+ val ret8 = cast(
+ Literal.create(Array(Array(Array("a"), Array("b", "c")), Array(Array("d")))),
+ StringType)
+ checkEvaluation(ret8, "[[[a], [b, c]], [[d]]]")
+ }
+
+ test("SPARK-33291: Cast array with null elements to string") {
+ Seq(false, true).foreach { omitNull =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> omitNull.toString) {
+ val ret1 = cast(Literal.create(Array(null, null)), StringType)
+ checkEvaluation(
+ ret1,
+ s"[${if (omitNull) "" else "null"},${if (omitNull) "" else " null"}]")
+ }
+ }
+ }
+
+ test("SPARK-22973 Cast map to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create(Map(1 -> "a", 2 -> "b", 3 -> "c")), StringType)
+ checkEvaluation(ret1, s"${lb}1 -> a, 2 -> b, 3 -> c$rb")
+ val ret2 = cast(
+ Literal.create(Map("1" -> "a".getBytes, "2" -> null, "3" -> "c".getBytes)),
+ StringType)
+ checkEvaluation(ret2, s"${lb}1 -> a, 2 ->${if (legacyCast) "" else " null"}, 3 -> c$rb")
+ val ret3 = cast(
+ Literal.create(Map(
+ 1 -> Date.valueOf("2014-12-03"),
+ 2 -> Date.valueOf("2014-12-04"),
+ 3 -> Date.valueOf("2014-12-05"))),
+ StringType)
+ checkEvaluation(ret3, s"${lb}1 -> 2014-12-03, 2 -> 2014-12-04, 3 -> 2014-12-05$rb")
+ val ret4 = cast(
+ Literal.create(Map(
+ 1 -> Timestamp.valueOf("2014-12-03 13:01:00"),
+ 2 -> Timestamp.valueOf("2014-12-04 15:05:00"))),
+ StringType)
+ checkEvaluation(ret4, s"${lb}1 -> 2014-12-03 13:01:00, 2 -> 2014-12-04 15:05:00$rb")
+ val ret5 = cast(
+ Literal.create(Map(
+ 1 -> Array(1, 2, 3),
+ 2 -> Array(4, 5, 6))),
+ StringType)
+ checkEvaluation(ret5, s"${lb}1 -> [1, 2, 3], 2 -> [4, 5, 6]$rb")
+ }
+ }
+ }
+
+ test("SPARK-22981 Cast struct to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create((1, "a", 0.1)), StringType)
+ checkEvaluation(ret1, s"${lb}1, a, 0.1$rb")
+ val ret2 = cast(Literal.create(Tuple3[Int, String, String](1, null, "a")), StringType)
+ checkEvaluation(ret2, s"${lb}1,${if (legacyCast) "" else " null"}, a$rb")
+ val ret3 = cast(Literal.create(
+ (Date.valueOf("2014-12-03"), Timestamp.valueOf("2014-12-03 15:05:00"))), StringType)
+ checkEvaluation(ret3, s"${lb}2014-12-03, 2014-12-03 15:05:00$rb")
+ val ret4 = cast(Literal.create(((1, "a"), 5, 0.1)), StringType)
+ checkEvaluation(ret4, s"$lb${lb}1, a$rb, 5, 0.1$rb")
+ val ret5 = cast(Literal.create((Seq(1, 2, 3), "a", 0.1)), StringType)
+ checkEvaluation(ret5, s"$lb[1, 2, 3], a, 0.1$rb")
+ val ret6 = cast(Literal.create((1, Map(1 -> "a", 2 -> "b", 3 -> "c"))), StringType)
+ checkEvaluation(ret6, s"${lb}1, ${lb}1 -> a, 2 -> b, 3 -> c$rb$rb")
+ }
+ }
+ }
+
+ test("SPARK-33291: Cast struct with null elements to string") {
+ Seq(
+ false -> ("{", "}"),
+ true -> ("[", "]")).foreach { case (legacyCast, (lb, rb)) =>
+ withSQLConf(SQLConf.LEGACY_COMPLEX_TYPES_TO_STRING.key -> legacyCast.toString) {
+ val ret1 = cast(Literal.create(Tuple2[String, String](null, null)), StringType)
+ checkEvaluation(
+ ret1,
+ s"$lb${if (legacyCast) "" else "null"},${if (legacyCast) "" else " null"}$rb")
+ }
+ }
+ }
+
+ test("data type casting II") {
+ checkEvaluation(
+ cast(cast(cast(cast(cast(cast("5", ByteType), TimestampType),
+ DecimalType.SYSTEM_DEFAULT), LongType), StringType), ShortType),
+ 5.toShort)
+ checkEvaluation(
+ cast(cast(cast(cast(cast(cast("5", TimestampType, UTC_OPT), ByteType),
+ DecimalType.SYSTEM_DEFAULT), LongType), StringType), ShortType),
+ null)
+ checkEvaluation(cast(cast(cast(cast(cast(cast("5", DecimalType.SYSTEM_DEFAULT),
+ ByteType), TimestampType), LongType), StringType), ShortType),
+ 5.toShort)
+ }
+
+ test("Cast from double II") {
+ checkEvaluation(cast(cast(1.toDouble, TimestampType), DoubleType), 1.toDouble)
+ }
}
/**
- * Test suite for data type casting expression [[AnsiCast]].
+ * Test suite for data type casting expression [[Cast]] with ANSI mode disabled.
*/
-class AnsiCastSuite extends CastSuiteBase {
- // It is not required to set SQLConf.ANSI_ENABLED as true for testing numeric overflow.
- override protected def requiredAnsiEnabledForOverflowTestCases: Boolean = false
+class CastSuiteWithAnsiModeOn extends AnsiCastSuiteBase {
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ SQLConf.get.setConf(SQLConf.ANSI_ENABLED, true)
+ }
+
+ override def afterAll(): Unit = {
+ super.afterAll()
+ SQLConf.get.unsetConf(SQLConf.ANSI_ENABLED)
+ }
override def cast(v: Any, targetType: DataType, timeZoneId: Option[String] = None): CastBase = {
v match {
- case lit: Expression => AnsiCast(lit, targetType, timeZoneId)
- case _ => AnsiCast(Literal(v), targetType, timeZoneId)
+ case lit: Expression => Cast(lit, targetType, timeZoneId)
+ case _ => Cast(Literal(v), targetType, timeZoneId)
}
}
+}
- test("cast from invalid string to numeric should throw NumberFormatException") {
- // cast to IntegerType
- Seq(IntegerType, ShortType, ByteType, LongType).foreach { dataType =>
- val array = Literal.create(Seq("123", "true", "f", null),
- ArrayType(StringType, containsNull = true))
- checkExceptionInExpression[NumberFormatException](
- cast(array, ArrayType(dataType, containsNull = true)),
- "invalid input syntax for type numeric: true")
- checkExceptionInExpression[NumberFormatException](
- cast("string", dataType), "invalid input syntax for type numeric: string")
- checkExceptionInExpression[NumberFormatException](
- cast("123-string", dataType), "invalid input syntax for type numeric: 123-string")
- checkExceptionInExpression[NumberFormatException](
- cast("2020-07-19", dataType), "invalid input syntax for type numeric: 2020-07-19")
- checkExceptionInExpression[NumberFormatException](
- cast("1.23", dataType), "invalid input syntax for type numeric: 1.23")
- }
+/**
+ * Test suite for data type casting expression [[AnsiCast]] with ANSI mode enabled.
+ */
+class AnsiCastSuiteWithAnsiModeOn extends CastSuiteBase {
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ SQLConf.get.setConf(SQLConf.ANSI_ENABLED, true)
Review comment:
Essentially, `withSQLConf` also uses `SQLConf`. There is no much difference here.
This is a common issue for test suites of the Catalyst module. We should figure out a proper way for setting SQL conf without impacts on other concurrent tests later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30260: [SPARK-33354][SQL] New explicit cast syntax rules in ANSI mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30260:
URL: https://github.com/apache/spark/pull/30260#issuecomment-729724925
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org