You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/20 09:41:24 UTC

[GitHub] [spark] leanken opened a new pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

leanken opened a new pull request #30442:
URL: https://github.com/apache/spark/pull/30442


   ### What changes were proposed in this pull request?
   
   Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid, when ANSI mode is enable. This patch should update GetTimeStamp, UnixTimeStamp, ToUnixTimeStamp and Cast.
   
   ### Why are the changes needed?
   
   For ANSI mode.
   
   ### Does this PR introduce any user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Added UT and Existing UT.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734683691


   **[Test build #131865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131865/testReport)** for PR 30442 at commit [`4706576`](https://github.com/apache/spark/commit/47065761665cbc281159242e511822b3f4fa499d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731922512


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731922512






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r530367654



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -945,6 +934,36 @@ abstract class AnsiCastSuiteBase extends CastSuiteBase {
       cast("abcd", DecimalType(38, 1)),
       "invalid input syntax for type numeric")
   }
+
+  test("ANSI mode: cast string to timestamp with parse error") {
+    val currentAnsiEnabled = SQLConf.get.ansiEnabled
+    new ParVector(ALL_TIMEZONES.toVector).foreach { zid =>
+      def checkCastWithParseError(str: String): Unit = {
+        checkExceptionInExpression[DateTimeException](
+          cast(Literal(str), TimestampType, Option(zid.getId)),
+          s"Cannot cast $str to TimestampType.")
+      }
+
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> currentAnsiEnabled.toString) {
+        checkCastWithParseError("123")
+        checkCastWithParseError("2015-03-18 123142")
+        checkCastWithParseError("2015-03-18T123123")
+        checkCastWithParseError("2015-03-18X")
+        checkCastWithParseError("2015/03/18")
+        checkCastWithParseError("2015.03.18")
+        checkCastWithParseError("20150318")
+        checkCastWithParseError("2015-031-8")
+        checkCastWithParseError("2015-03-18T12:03:17-0:70")
+      }
+    }
+  }
+
+  test("ANSI mode: timestamp type casting with parse error") {

Review comment:
       Why is this a separate test?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733038378






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734760121


   **[Test build #131879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131879/testReport)** for PR 30442 at commit [`e4fe5ee`](https://github.com/apache/spark/commit/e4fe5ee34b47bb5f57aad76fa41caa2c27eee53d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] leanken edited a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
leanken edited a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731061016


   @cloud-fan FYI.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r530366120



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1254,10 +1260,11 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
         ctx.addReferenceObj("zoneId", zoneId, zoneIdClass.getName),
         zoneIdClass)
       val longOpt = ctx.freshVariable("longOpt", classOf[Option[Long]])
+      val stringToTimestampFunc = if (ansiEnabled) "stringToTimestampAnsi" else "stringToTimestamp"

Review comment:
       If `stringToTimestampAnsi`  returns `Long` directly, the codegen for it will be largely simplified.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732919060


   **[Test build #131656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131656/testReport)** for PR 30442 at commit [`9b76d6a`](https://github.com/apache/spark/commit/9b76d6ac046f3bd8ea67b7628ed5b48659b73f46).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732516802






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733637314






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731113427


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r531599042



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -945,6 +934,34 @@ abstract class AnsiCastSuiteBase extends CastSuiteBase {
       cast("abcd", DecimalType(38, 1)),
       "invalid input syntax for type numeric")
   }
+
+  test("ANSI mode: cast string to timestamp with parse error") {
+    val activeConf = conf
+    new ParVector(ALL_TIMEZONES.toVector).foreach { zid =>
+      def checkCastWithParseError(str: String): Unit = {
+        checkExceptionInExpression[DateTimeException](
+          cast(Literal(str), TimestampType, Option(zid.getId)),
+          s"Cannot cast $str to TimestampType.")
+      }
+
+      SQLConf.withExistingConf(activeConf) {
+        checkCastWithParseError("123")
+        checkCastWithParseError("2015-03-18 123142")
+        checkCastWithParseError("2015-03-18T123123")
+        checkCastWithParseError("2015-03-18X")
+        checkCastWithParseError("2015/03/18")
+        checkCastWithParseError("2015.03.18")
+        checkCastWithParseError("20150318")
+        checkCastWithParseError("2015-031-8")
+        checkCastWithParseError("2015-03-18T12:03:17-0:70")
+
+        val input = "abdef"
+        checkExceptionInExpression[DateTimeException](

Review comment:
       is it just `checkCastWithParseError("abdef")`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732799105






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733694230






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] leanken commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r531412245



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -945,6 +934,36 @@ abstract class AnsiCastSuiteBase extends CastSuiteBase {
       cast("abcd", DecimalType(38, 1)),
       "invalid input syntax for type numeric")
   }
+
+  test("ANSI mode: cast string to timestamp with parse error") {
+    val currentAnsiEnabled = SQLConf.get.ansiEnabled
+    new ParVector(ALL_TIMEZONES.toVector).foreach { zid =>
+      def checkCastWithParseError(str: String): Unit = {
+        checkExceptionInExpression[DateTimeException](
+          cast(Literal(str), TimestampType, Option(zid.getId)),
+          s"Cannot cast $str to TimestampType.")
+      }
+
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> currentAnsiEnabled.toString) {
+        checkCastWithParseError("123")
+        checkCastWithParseError("2015-03-18 123142")
+        checkCastWithParseError("2015-03-18T123123")
+        checkCastWithParseError("2015-03-18X")
+        checkCastWithParseError("2015/03/18")
+        checkCastWithParseError("2015.03.18")
+        checkCastWithParseError("20150318")
+        checkCastWithParseError("2015-031-8")
+        checkCastWithParseError("2015-03-18T12:03:17-0:70")
+      }
+    }
+  }
+
+  test("ANSI mode: timestamp type casting with parse error") {

Review comment:
       merged in one test

##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -945,6 +934,36 @@ abstract class AnsiCastSuiteBase extends CastSuiteBase {
       cast("abcd", DecimalType(38, 1)),
       "invalid input syntax for type numeric")
   }
+
+  test("ANSI mode: cast string to timestamp with parse error") {
+    val currentAnsiEnabled = SQLConf.get.ansiEnabled
+    new ParVector(ALL_TIMEZONES.toVector).foreach { zid =>
+      def checkCastWithParseError(str: String): Unit = {
+        checkExceptionInExpression[DateTimeException](
+          cast(Literal(str), TimestampType, Option(zid.getId)),
+          s"Cannot cast $str to TimestampType.")
+      }
+
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> currentAnsiEnabled.toString) {

Review comment:
       done

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -720,10 +720,12 @@ case class DateFormatClass(left: Expression, right: Expression, timeZoneId: Opti
 case class ToUnixTimestamp(
     timeExp: Expression,
     format: Expression,
-    timeZoneId: Option[String] = None)
+    timeZoneId: Option[String] = None,
+    failOnError: Boolean = SQLConf.get.ansiEnabled)
   extends UnixTime {

Review comment:
       done.

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1254,10 +1260,11 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
         ctx.addReferenceObj("zoneId", zoneId, zoneIdClass.getName),
         zoneIdClass)
       val longOpt = ctx.freshVariable("longOpt", classOf[Option[Long]])
+      val stringToTimestampFunc = if (ansiEnabled) "stringToTimestampAnsi" else "stringToTimestamp"

Review comment:
       done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731922518


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131529/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731133207


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36024/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732919060


   **[Test build #131656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131656/testReport)** for PR 30442 at commit [`9b76d6a`](https://github.com/apache/spark/commit/9b76d6ac046f3bd8ea67b7628ed5b48659b73f46).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733472027


   **[Test build #131739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131739/testReport)** for PR 30442 at commit [`140081d`](https://github.com/apache/spark/commit/140081d593f443054cf62776b20443e981b4c0c5).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733230959






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r529621021



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -453,7 +453,8 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
   // TimestampConverter
   private[this] def castToTimestamp(from: DataType): Any => Any = from match {
     case StringType =>
-      buildCast[UTF8String](_, utfs => DateTimeUtils.stringToTimestamp(utfs, zoneId).orNull)
+      buildCast[UTF8String](_,
+        utfs => DateTimeUtils.stringToTimestampAnsi(utfs, zoneId, ansiEnabled).orNull)

Review comment:
       We should call `stringToTimestampAnsi` when ansi mode is on, and call the original method when ansi mode is off.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r530365453



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
##########
@@ -720,10 +720,12 @@ case class DateFormatClass(left: Expression, right: Expression, timeZoneId: Opti
 case class ToUnixTimestamp(
     timeExp: Expression,
     format: Expression,
-    timeZoneId: Option[String] = None)
+    timeZoneId: Option[String] = None,
+    failOnError: Boolean = SQLConf.get.ansiEnabled)
   extends UnixTime {

Review comment:
       can we update the `nullability` based on the `failOnError` flag?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734757830






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r530367089



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -945,6 +934,36 @@ abstract class AnsiCastSuiteBase extends CastSuiteBase {
       cast("abcd", DecimalType(38, 1)),
       "invalid input syntax for type numeric")
   }
+
+  test("ANSI mode: cast string to timestamp with parse error") {
+    val currentAnsiEnabled = SQLConf.get.ansiEnabled
+    new ParVector(ALL_TIMEZONES.toVector).foreach { zid =>
+      def checkCastWithParseError(str: String): Unit = {
+        checkExceptionInExpression[DateTimeException](
+          cast(Literal(str), TimestampType, Option(zid.getId)),
+          s"Cannot cast $str to TimestampType.")
+      }
+
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> currentAnsiEnabled.toString) {

Review comment:
       to be more robust:
   ```
   val activeConf = conf
   new ParVector...
     SQLConf.withExistingConf(activeConf) ...
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] leanken commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733639230


   @cloud-fan Test passed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731113427






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733029355






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732653732






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733639401






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733637314






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731121843


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36024/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732503292


   **[Test build #131592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131592/testReport)** for PR 30442 at commit [`67f2d41`](https://github.com/apache/spark/commit/67f2d41c2784d47e7ad7a8985ab811fc5151b877).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731064034


   **[Test build #131418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131418/testReport)** for PR 30442 at commit [`e6f5634`](https://github.com/apache/spark/commit/e6f56348538fd62f890de646e6223d1ed00abc82).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731113433


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131418/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734757830






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733038378






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734708203






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731064034


   **[Test build #131418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131418/testReport)** for PR 30442 at commit [`e6f5634`](https://github.com/apache/spark/commit/e6f56348538fd62f890de646e6223d1ed00abc82).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732799105






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731113092


   **[Test build #131418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131418/testReport)** for PR 30442 at commit [`e6f5634`](https://github.com/apache/spark/commit/e6f56348538fd62f890de646e6223d1ed00abc82).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `case class UnixTimestamp(`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734760121


   **[Test build #131879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131879/testReport)** for PR 30442 at commit [`e4fe5ee`](https://github.com/apache/spark/commit/e4fe5ee34b47bb5f57aad76fa41caa2c27eee53d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733230959






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733488754


   **[Test build #131744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131744/testReport)** for PR 30442 at commit [`5ab59d4`](https://github.com/apache/spark/commit/5ab59d41cf840f524d37dbb581fabcbbba1c56a4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r528557886



##########
File path: docs/sql-ref-ansi-compliance.md
##########
@@ -136,12 +136,17 @@ The behavior of some SQL functions can be different under ANSI mode (`spark.sql.
   - `element_at`: This function throws `NoSuchElementException` if key does not exist in map. 
   - `elt`: This function throws `ArrayIndexOutOfBoundsException` if using invalid indices.
   - `parse_url`: This function throws `IllegalArgumentException` if an input string is not a valid url.
+  - `to_date` This function should fail with an exception if the input string can't be parsed, or the pattern string is invalid.
+  - `to_timestamp` This function should fail with an exception if the input string can't be parsed, or the pattern string is invalid.
+  - `unix_timestamp` This function should fail with an exception if the input string can't be parsed, or the pattern string is invalid.
+  - `to_unix_timestamp` This function should fail with an exception if the input string can't be parsed, or the pattern string is invalid.
 
 ### SQL Operators
 
 The behavior of some SQL operators can be different under ANSI mode (`spark.sql.ansi.enabled=true`).
   - `array_col[index]`: This operator throws `ArrayIndexOutOfBoundsException` if using invalid indices.
   - `map_col[key]`: This operator throws `NoSuchElementException` if key does not exist in map.
+  - `cast to timestamp`: This operator should fail with an exception if the input string can't be parsed.

Review comment:
       nit: `CAST(string_col AS TIMESTAMP)` to be user-facing.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734867997


   **[Test build #131879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131879/testReport)** for PR 30442 at commit [`e4fe5ee`](https://github.com/apache/spark/commit/e4fe5ee34b47bb5f57aad76fa41caa2c27eee53d).
    * This patch **fails SparkR unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732756404


   **[Test build #131635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131635/testReport)** for PR 30442 at commit [`0417977`](https://github.com/apache/spark/commit/0417977c5cffa824e0e684f94ef62a2179018f14).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731924628






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731924628






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733639401






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] leanken commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
leanken commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731061016


   @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r527690304



##########
File path: docs/sql-ref-ansi-compliance.md
##########
@@ -135,12 +135,17 @@ The behavior of some SQL functions can be different under ANSI mode (`spark.sql.
   - `element_at`: This function throws `ArrayIndexOutOfBoundsException` if using invalid indices. 
   - `element_at`: This function throws `NoSuchElementException` if key does not exist in map. 
   - `elt`: This function throws `ArrayIndexOutOfBoundsException` if using invalid indices.
+  - `to_date` This function should fail with Exception if the input string can't be parsed, or the pattern string is invalid.

Review comment:
       I would call it `an exception` instead of `Exception`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r530364835



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -364,6 +364,15 @@ object DateTimeUtils {
     }
   }
 
+  def stringToTimestampAnsi(s: UTF8String, timeZoneId: ZoneId): Option[Long] = {

Review comment:
       does this method still need to return option?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan closed pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30442:
URL: https://github.com/apache/spark/pull/30442


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733694230






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r528565942



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -1286,4 +1287,47 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
     testIntegralFunc(Long.MaxValue)
     testIntegralFunc(Long.MinValue)
   }
+
+  test("SPARK-33498: GetTimestamp,UnixTimestamp,ToUnixTimestamp with parseError") {
+    Seq(true, false).foreach { ansiEnabled =>
+      Seq("LEGACY", "CORRECTED", "EXCEPTION").foreach { policy =>
+        withSQLConf(SQLConf.LEGACY_TIME_PARSER_POLICY.key -> policy,
+          SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+
+          val exprSeq = Seq[Expression](
+            GetTimestamp(Literal("2020-01-27T20:06:11.847"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            GetTimestamp(Literal("Unparseable"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            UnixTimestamp(Literal("2020-01-27T20:06:11.847"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            UnixTimestamp(Literal("Unparseable"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            ToUnixTimestamp(Literal("2020-01-27T20:06:11.847"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            ToUnixTimestamp(Literal("Unparseable"), Literal("yyyy-MM-dd HH:mm:ss.SSS"))
+          )
+
+          if (!ansiEnabled) {
+            exprSeq.foreach(checkEvaluation(_, null))
+          } else if (policy == "LEGACY") {
+            exprSeq.foreach(checkExceptionInExpression[ParseException](_, "Unparseable"))
+          } else {
+            exprSeq.foreach(
+              checkExceptionInExpression[DateTimeParseException](_, "could not be parsed"))
+          }
+        }
+      }
+    }
+  }
+
+  test("SPARK-33498: TimestampType cast with parseError") {

Review comment:
       is it duplicated with `CastSuiteBase`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734708203






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731919440


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36131/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732516802






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733567807






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731911257


   **[Test build #131529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131529/testReport)** for PR 30442 at commit [`67f2d41`](https://github.com/apache/spark/commit/67f2d41c2784d47e7ad7a8985ab811fc5151b877).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734782649






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734782649






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733488754


   **[Test build #131744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131744/testReport)** for PR 30442 at commit [`5ab59d4`](https://github.com/apache/spark/commit/5ab59d41cf840f524d37dbb581fabcbbba1c56a4).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733567807






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733472027


   **[Test build #131739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131739/testReport)** for PR 30442 at commit [`140081d`](https://github.com/apache/spark/commit/140081d593f443054cf62776b20443e981b4c0c5).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733633867


   **[Test build #131744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131744/testReport)** for PR 30442 at commit [`5ab59d4`](https://github.com/apache/spark/commit/5ab59d41cf840f524d37dbb581fabcbbba1c56a4).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733029355






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732503292


   **[Test build #131592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131592/testReport)** for PR 30442 at commit [`67f2d41`](https://github.com/apache/spark/commit/67f2d41c2784d47e7ad7a8985ab811fc5151b877).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r528559834



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -364,6 +364,15 @@ object DateTimeUtils {
     }
   }
 
+  def stringToTimestamp(s: UTF8String, timeZoneId: ZoneId, ansiEnabled: Boolean): Option[Long] = {

Review comment:
       It's better to add a `stringToTimestampAnsi` method. Non-ansi code path should call the original `stringToTimestamp` method.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731133226






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731911257


   **[Test build #131529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131529/testReport)** for PR 30442 at commit [`67f2d41`](https://github.com/apache/spark/commit/67f2d41c2784d47e7ad7a8985ab811fc5151b877).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731924616


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36131/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732653732






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734683691


   **[Test build #131865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131865/testReport)** for PR 30442 at commit [`4706576`](https://github.com/apache/spark/commit/47065761665cbc281159242e511822b3f4fa499d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734757421


   **[Test build #131865 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131865/testReport)** for PR 30442 at commit [`4706576`](https://github.com/apache/spark/commit/47065761665cbc281159242e511822b3f4fa499d).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734879651






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732516563


   **[Test build #131592 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131592/testReport)** for PR 30442 at commit [`67f2d41`](https://github.com/apache/spark/commit/67f2d41c2784d47e7ad7a8985ab811fc5151b877).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734833120


   GA passed, merging to master!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] leanken commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
leanken commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r530904343



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/window_part3.sql.out
##########
@@ -71,7 +71,8 @@ insert into datetimes values
 -- !query schema
 struct<>
 -- !query output
-
+org.apache.spark.sql.AnalysisException

Review comment:
       seems this datatimes table was not used at all. I could refine this test case in new PR.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732756404


   **[Test build #131635 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131635/testReport)** for PR 30442 at commit [`0417977`](https://github.com/apache/spark/commit/0417977c5cffa824e0e684f94ef62a2179018f14).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-732798784


   **[Test build #131635 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131635/testReport)** for PR 30442 at commit [`0417977`](https://github.com/apache/spark/commit/0417977c5cffa824e0e684f94ef62a2179018f14).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733021920


   **[Test build #131656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131656/testReport)** for PR 30442 at commit [`9b76d6a`](https://github.com/apache/spark/commit/9b76d6ac046f3bd8ea67b7628ed5b48659b73f46).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731169064


   cc @MaxGekk too FYI


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-733564271


   **[Test build #131739 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131739/testReport)** for PR 30442 at commit [`140081d`](https://github.com/apache/spark/commit/140081d593f443054cf62776b20443e981b4c0c5).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-734879651






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r528567293



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
##########
@@ -1286,4 +1287,47 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
     testIntegralFunc(Long.MaxValue)
     testIntegralFunc(Long.MinValue)
   }
+
+  test("SPARK-33498: GetTimestamp,UnixTimestamp,ToUnixTimestamp with parseError") {
+    Seq(true, false).foreach { ansiEnabled =>
+      Seq("LEGACY", "CORRECTED", "EXCEPTION").foreach { policy =>
+        withSQLConf(SQLConf.LEGACY_TIME_PARSER_POLICY.key -> policy,
+          SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+
+          val exprSeq = Seq[Expression](
+            GetTimestamp(Literal("2020-01-27T20:06:11.847"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            GetTimestamp(Literal("Unparseable"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            UnixTimestamp(Literal("2020-01-27T20:06:11.847"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            UnixTimestamp(Literal("Unparseable"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            ToUnixTimestamp(Literal("2020-01-27T20:06:11.847"), Literal("yyyy-MM-dd HH:mm:ss.SSS")),
+            ToUnixTimestamp(Literal("Unparseable"), Literal("yyyy-MM-dd HH:mm:ss.SSS"))

Review comment:
       can we test one more case that throws `SparkUpgradeException`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731922480


   **[Test build #131529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131529/testReport)** for PR 30442 at commit [`67f2d41`](https://github.com/apache/spark/commit/67f2d41c2784d47e7ad7a8985ab811fc5151b877).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] cloud-fan commented on a change in pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30442:
URL: https://github.com/apache/spark/pull/30442#discussion_r530370757



##########
File path: sql/core/src/test/resources/sql-tests/results/postgreSQL/window_part3.sql.out
##########
@@ -71,7 +71,8 @@ insert into datetimes values
 -- !query schema
 struct<>
 -- !query output
-
+org.apache.spark.sql.AnalysisException

Review comment:
       Not related to this PR but it's weird that the data preparation step for a test is broken. We should fix it later.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30442:
URL: https://github.com/apache/spark/pull/30442#issuecomment-731133226






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org