You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/21 08:21:08 UTC
[GitHub] [spark] AngersZhuuuu opened a new pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
AngersZhuuuu opened a new pull request #32266:
URL: https://github.com/apache/spark/pull/32266
### What changes were proposed in this pull request?
Support Cast string to year-month interval
### Why are the changes needed?
Support Cast string to year-month interval
### Does this PR introduce _any_ user-facing change?
User can cast year month interval string to YearMonthIntervalType
### How was this patch tested?
Added UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623239339
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,25 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): Int = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
Review comment:
Just wonder why don't you trim whitespaces by regular expression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r620939247
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,13 +95,22 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def fromYearMonthString(input: UTF8String): CalendarInterval = {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
Review comment:
cast null to interval should return null, this is incorrect.
we can add `assert(input != null)`, as `Cast` should guarantee it before calling this function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824088616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137725/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823904257
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137721/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623553192
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,25 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): Int = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
Review comment:
> You don't see any tests that check proper spaces handling - I mean `trimAll()`. Could you add such checks.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622784209
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,30 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
+ case yearMonthPattern("-", year, month) => toYMInterval(year, month, -1)
+ case yearMonthPattern(_, year, month) => toYMInterval(year, month, 1)
+ case yearMonthStringPattern(_, prefixSign, _, suffixSign, year, month, _, _) =>
Review comment:
`prefixSign` `suffixSign` looks weird. how about `firstSign` and `secondSign`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825687669
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42393/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826251054
**[Test build #137905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137905/testReport)** for PR 32266 at commit [`3b84baa`](https://github.com/apache/spark/commit/3b84baae46936929212b7f1b57dc27e08fd9c564).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824694001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829038268
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824712740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823922267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824792808
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825683474
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42393/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825687669
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42393/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622206825
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +92,36 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "(INTERVAL )?([+|-])*?(')?([+|-])?(\\d+)-(\\d+)(')?( YEAR TO MONTH)?".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ // scalastyle:off caselocale .toUpperCase
+ input.trimAll().toUpperCase.toString match {
Review comment:
toUpperCase(Locale.Root)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824974848
**[Test build #137803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137803/testReport)** for PR 32266 at commit [`6d14414`](https://github.com/apache/spark/commit/6d144147721001253cf515dc13b4cee615948463).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829186341
**[Test build #138075 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138075/testReport)** for PR 32266 at commit [`2c8785b`](https://github.com/apache/spark/commit/2c8785b2a4515ca7a30bf2ef8c5312e81ff3e5f6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622787481
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,50 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("interval '10-1' year TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create(null, StringType), YearMonthIntervalType), null)
+
+ Seq("INTERVAL '0-0' YEAR TO MONTH", "INTERVAL '10-1' YEAR TO MONTH",
Review comment:
We can merge most of the tests above into this loop
```
val hiveStyleInterval = interval.split("'")(1)
// test with both interval and hiveStyleInterval
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829139042
**[Test build #138072 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138072/testReport)** for PR 32266 at commit [`ca19c09`](https://github.com/apache/spark/commit/ca19c09e71afafc3d11fd752e828e1207e27ed8e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824839668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622202234
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +92,36 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "(INTERVAL )?([+|-])*?(')?([+|-])?(\\d+)-(\\d+)(')?( YEAR TO MONTH)?".r
Review comment:
`INTERVAL\s+`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826251054
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618197487
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,28 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
+
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
Review comment:
> Why not just call `fromYearMonthString`?
Null safe? like `safeStringToInterval` we can avoid NPE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823942131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623228675
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,19 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
Review comment:
Please, check the input in lower case. For example:
```scala
checkEvaluation(cast(Literal.create(" interval '1-0' YEAR TO MONTH "),
YearMonthIntervalType), 12)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823878736
**[Test build #137721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137721/testReport)** for PR 32266 at commit [`9546a0f`](https://github.com/apache/spark/commit/9546a0f22f62e7017d73530cc691223e0ca85587).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829782930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42616/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829782021
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42616/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622797208
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -127,20 +123,21 @@ object IntervalUtils {
require(input != null, "Interval year-month string must be not null")
input.trim match {
case yearMonthPattern("-", yearStr, monthStr) =>
- toYMInterval(yearStr, monthStr, -1)
+ new CalendarInterval(toYMInterval(yearStr, monthStr, -1), 0, 0)
case yearMonthPattern(_, yearStr, monthStr) =>
- toYMInterval(yearStr, monthStr, 1)
+ new CalendarInterval(toYMInterval(yearStr, monthStr, 1), 0, 0)
case _ =>
throw new IllegalArgumentException(
s"Interval string does not match year-month format of 'y-m': $input")
}
}
- def toYMInterval(yearStr: String, monthStr: String, sign: Int): CalendarInterval = {
+ def toYMInterval(yearStr: String, monthStr: String, sign: Int): Int = {
try {
val years = toLongWithRange(YEAR, yearStr, 0, Integer.MAX_VALUE / MONTHS_PER_YEAR)
val totalMonths = sign * (years * MONTHS_PER_YEAR + toLongWithRange(MONTH, monthStr, 0, 11))
- new CalendarInterval(Math.toIntExact(totalMonths), 0, 0)
+// new CalendarInterval(Math.toIntExact(totalMonths), 0, 0)
Review comment:
remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829076714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826251054
**[Test build #137905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137905/testReport)** for PR 32266 at commit [`3b84baa`](https://github.com/apache/spark/commit/3b84baae46936929212b7f1b57dc27e08fd9c564).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619258363
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,23 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
Review comment:
what do we want to support here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619183754
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
@MaxGekk Can you help to list the case we need to support here,
```
INTERVAL [-|+]'[+|-]0-0' YEAR TO MONTH
[+|-]0-0
```
Any more case need to support here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r620960351
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,45 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '0-0' YEAR TO MONTH"),
Review comment:
This is covered by the `Seq("INTERVAL '0-0' YEAR TO MONTH" ...` below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829201800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622238616
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +92,36 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "(INTERVAL )?([+|-])*?(')?([+|-])?(\\d+)-(\\d+)(')?( YEAR TO MONTH)?".r
Review comment:
> `([+|-])*`? The sign can only appear once, right?
This is written to match as right as possibleļ¼ as past pattern `-0-0`, the sign will be match to prefix sign, but as current pattern, we don't need to do this now.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +92,36 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "(INTERVAL )?([+|-])*?(')?([+|-])?(\\d+)-(\\d+)(')?( YEAR TO MONTH)?".r
Review comment:
DOne
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +92,36 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "(INTERVAL )?([+|-])*?(')?([+|-])?(\\d+)-(\\d+)(')?( YEAR TO MONTH)?".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ // scalastyle:off caselocale .toUpperCase
+ input.trimAll().toUpperCase.toString match {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824636620
**[Test build #137791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137791/testReport)** for PR 32266 at commit [`62d175b`](https://github.com/apache/spark/commit/62d175bf9a9ae89a0b5a150e14583f5d850c4cda).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622798441
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,51 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828381339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829759396
**[Test build #138096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138096/testReport)** for PR 32266 at commit [`5ca83ab`](https://github.com/apache/spark/commit/5ca83abe001897216575fad9f00906de553626c6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623228675
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,19 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
Review comment:
Please, check the input in lower case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829005840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829086259
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42599/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825798307
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829243915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622784689
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,30 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
+ case yearMonthPattern("-", year, month) => toYMInterval(year, month, -1)
+ case yearMonthPattern(_, year, month) => toYMInterval(year, month, 1)
+ case yearMonthStringPattern(_, prefixSign, _, suffixSign, year, month, _, _) =>
+ (prefixSign, suffixSign) match {
+ case ("-", "-") => toYMInterval(year, month, 1)
+ case ("-", _) => toYMInterval(year, month, -1)
+ case (_, "-") => toYMInterval(year, month, -1)
+ case (_, _) => toYMInterval(year, month, 1)
+ }
+ case yearMonthStringPattern(_, null, _, "-", year, month, _, _) =>
+ toYMInterval(year, month, -1)
+ case yearMonthStringPattern(_, null, _, _, year, month, _, _) =>
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618593263
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1353,6 +1361,13 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
+ private[this] def castToYearMonthIntervalCode(from: DataType): CastFunction = from match {
+ case StringType =>
+ val util = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
+ (c, evPrim, evNull) =>
+ code"""$evPrim = (Integer)$util.fromYearMonthString($c).months;""".stripMargin
Review comment:
Why do we need to box the result?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829052738
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42596/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825625763
**[Test build #137862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137862/testReport)** for PR 32266 at commit [`d19bbc8`](https://github.com/apache/spark/commit/d19bbc82d33ced40c93e00954610a521566642ac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829048907
**[Test build #138079 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138079/testReport)** for PR 32266 at commit [`9c70b88`](https://github.com/apache/spark/commit/9c70b88f0a3a1bdfd094589a7e2582f20ddd7ff7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829780838
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42616/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622206277
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +92,36 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "(INTERVAL )?([+|-])*?(')?([+|-])?(\\d+)-(\\d+)(')?( YEAR TO MONTH)?".r
Review comment:
Maybe we can match by `yearMonthPattern` first, and fall back to this regex later. Then this regex doesn't need to have so many optional parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824977520
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137803/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828977050
**[Test build #138072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138072/testReport)** for PR 32266 at commit [`ca19c09`](https://github.com/apache/spark/commit/ca19c09e71afafc3d11fd752e828e1207e27ed8e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829835952
+1, LGTM. GA passed. Merging to master.
Thank you @AngersZhuuuu , and @cloud-fan for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828507671
**[Test build #138047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138047/testReport)** for PR 32266 at commit [`092d01a`](https://github.com/apache/spark/commit/092d01acb4e412f09f58270678438ca903201e37).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622204367
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +92,36 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "(INTERVAL )?([+|-])*?(')?([+|-])?(\\d+)-(\\d+)(')?( YEAR TO MONTH)?".r
Review comment:
we also need `^` and `$`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829201800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829046266
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42595/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829906468
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823940132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618166304
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
> Regarding to the pattern `'([-|+]?[0-9]+-[-|+]?[0-9]+)'`, should we handle the second sign before month? Take a look at the val above `yearMonthPattern`
How about current
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824837786
**[Test build #137791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137791/testReport)** for PR 32266 at commit [`62d175b`](https://github.com/apache/spark/commit/62d175bf9a9ae89a0b5a150e14583f5d850c4cda).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823906967
**[Test build #137725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137725/testReport)** for PR 32266 at commit [`691c1f4`](https://github.com/apache/spark/commit/691c1f406d2d13a87c5748c6be758b91e9d844e2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622028747
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,45 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '0-0' YEAR TO MONTH"),
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r617316490
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,19 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
Review comment:
Could you add round trip tests: string -> year-month interval -> string, and year-month interval -> string -> year-month interval
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
Also could you move the regexp out of the method. So, we can avoid its "compilation" per every call.
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,19 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
Review comment:
Could you test corner cases when arithmetic overflow happens. Also test upper and lower cases.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
The keywords `INTERVAL` and `YEAR TO MONTH` are not mandatory by SQL standard. We should be able to parse only payload too.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
Regarding to the pattern `'([-|+]?[0-9]+-[-|+]?[0-9]+)'`, should we handle the second sign before month? Take a look at the val above `yearMonthPattern`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828507671
**[Test build #138047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138047/testReport)** for PR 32266 at commit [`092d01a`](https://github.com/apache/spark/commit/092d01acb4e412f09f58270678438ca903201e37).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828553397
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42566/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826289678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137905/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829052709
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42596/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622041722
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,13 +95,22 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def fromYearMonthString(input: UTF8String): CalendarInterval = {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ fromYearMonthString(input.trimAll().toString)
+ }
+ }
+
/**
* Parse YearMonth string in form: [+|-]YYYY-MM
*
* adapted from HiveIntervalYearMonth.valueOf
*/
def fromYearMonthString(input: String): CalendarInterval = {
Review comment:
Doneļ¼ how about current?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619736559
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,23 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
Review comment:
I am discussing this here https://github.com/apache/spark/pull/32266#discussion_r619231878 with @MaxGekk ,
Do you have any other supplementary suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618287141
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,28 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
+
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
Review comment:
How about current?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825680373
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42393/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826284117
**[Test build #137905 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137905/testReport)** for PR 32266 at commit [`3b84baa`](https://github.com/apache/spark/commit/3b84baae46936929212b7f1b57dc27e08fd9c564).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r620092134
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
For parsing I think it's OK to be always case insensitive. The Spark SQL parser is case insensitive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829906468
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138096/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829154887
**[Test build #138073 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138073/testReport)** for PR 32266 at commit [`3adde87`](https://github.com/apache/spark/commit/3adde873e1f0b4ad1ea5e468642ef4b76da2aaf4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618194841
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,28 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
+
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
Review comment:
Why not just call `fromYearMonthString`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828977050
**[Test build #138072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138072/testReport)** for PR 32266 at commit [`ca19c09`](https://github.com/apache/spark/commit/ca19c09e71afafc3d11fd752e828e1207e27ed8e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619134509
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,23 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
Review comment:
> why do we need a new regex?
Since yearMonthPattern only support `[+|-]yy-mmmm`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623231412
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,25 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
Review comment:
```suggestion
"(?i)^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
```
The (?i) is placed at the beginning of the pattern to enable case-insensitivity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618593520
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,23 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
Review comment:
why do we need a new regex?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619183754
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
Can you help to list the case we need to support here,
```
INTERVAL [-|+]'[+|-]0-0' YEAR TO MONTH
[+|-]0-0
```
Any more case need to support here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622334501
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,31 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])*?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR TO MONTH)$".r
Review comment:
`YEAR TO MONTH` should use `\s+`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829894606
**[Test build #138096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138096/testReport)** for PR 32266 at commit [`5ca83ab`](https://github.com/apache/spark/commit/5ca83abe001897216575fad9f00906de553626c6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825798307
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829007853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824712740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42320/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829155252
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138072/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622203816
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +92,36 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "(INTERVAL )?([+|-])*?(')?([+|-])?(\\d+)-(\\d+)(')?( YEAR TO MONTH)?".r
Review comment:
`([+|-])*`? The sign can only appear once, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622032465
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,45 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '0-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("interval '10-1' year TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create(null, StringType), YearMonthIntervalType), null)
+
+ Seq("INTERVAL '0-0' YEAR TO MONTH" -> "INTERVAL '0-0' YEAR TO MONTH",
+ "INTERVAL '10-1' YEAR TO MONTH" -> "INTERVAL '10-1' YEAR TO MONTH",
+ "INTERVAL '-178956970-7' YEAR TO MONTH" -> "INTERVAL '-178956970-7' YEAR TO MONTH",
+ "INTERVAL '178956970-7' YEAR TO MONTH" -> "INTERVAL '178956970-7' YEAR TO MONTH",
+ "INTERVAL '-178956970-8' YEAR TO MONTH" -> "INTERVAL '-178956970-8' YEAR TO MONTH"
Review comment:
Yea, handled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823906967
**[Test build #137725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137725/testReport)** for PR 32266 at commit [`691c1f4`](https://github.com/apache/spark/commit/691c1f406d2d13a87c5748c6be758b91e9d844e2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823878736
**[Test build #137721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137721/testReport)** for PR 32266 at commit [`9546a0f`](https://github.com/apache/spark/commit/9546a0f22f62e7017d73530cc691223e0ca85587).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829048863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829013446
**[Test build #138076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138076/testReport)** for PR 32266 at commit [`253c70e`](https://github.com/apache/spark/commit/253c70ea166a8e133dccef2cbe1c589be77aa861).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828348780
**[Test build #138044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138044/testReport)** for PR 32266 at commit [`ce69004`](https://github.com/apache/spark/commit/ce690041d1ee9dd26c1ced6087ff23cd45d2c0ef).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826257657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619745533
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
> but ideally we should respect the SQL config `spark.sql.caseSensitive`):
Spark Catalyst parser not respect `spark.sql.caseSensitive` too, should we respect it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829086259
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42599/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828553397
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42566/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829009391
**[Test build #138075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138075/testReport)** for PR 32266 at commit [`2c8785b`](https://github.com/apache/spark/commit/2c8785b2a4515ca7a30bf2ef8c5312e81ff3e5f6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829046266
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42595/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828545361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623233056
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,25 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): Int = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
Review comment:
You don't see any tests that check proper spaces handling - I mean `trimAll()`. Could you add such checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829189478
**[Test build #138076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138076/testReport)** for PR 32266 at commit [`253c70e`](https://github.com/apache/spark/commit/253c70ea166a8e133dccef2cbe1c589be77aa861).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829241166
**[Test build #138079 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138079/testReport)** for PR 32266 at commit [`9c70b88`](https://github.com/apache/spark/commit/9c70b88f0a3a1bdfd094589a7e2582f20ddd7ff7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826257657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824088616
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137725/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829155252
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138072/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622029023
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,13 +95,22 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def fromYearMonthString(input: UTF8String): CalendarInterval = {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
Review comment:
Removeļ¼ `Cast` can keep null safe here
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1354,6 +1362,13 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
+ private[this] def castToYearMonthIntervalCode(from: DataType): CastFunction = from match {
+ case StringType =>
+ val util = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
+ (c, evPrim, evNull) =>
+ code"""$evPrim = $util.fromYearMonthString($c).months;""".stripMargin
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823904257
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137721/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618991491
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1353,6 +1361,13 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
+ private[this] def castToYearMonthIntervalCode(from: DataType): CastFunction = from match {
+ case StringType =>
+ val util = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
+ (c, evPrim, evNull) =>
+ code"""$evPrim = (Integer)$util.fromYearMonthString($c).months;""".stripMargin
Review comment:
Hm, a mistake, forgot to remove this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824748308
**[Test build #137803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137803/testReport)** for PR 32266 at commit [`6d14414`](https://github.com/apache/spark/commit/6d144147721001253cf515dc13b4cee615948463).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829052738
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42596/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824792853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42332/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619231878
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
First of all, we must support our own output (in case insensitive manner by default but ideally we should respect the SQL config `spark.sql.caseSensitive`):
1. ANSI_STYLE, like
INTERVAL -'-10-1' YEAR TO MONTH
2. HIVE_STYLE like
10-1 or -10-1
Rules from the SQL standard:
```
<interval literal> ::=
INTERVAL [ <sign> ] <interval string> <interval qualifier>
<interval string> ::=
<quote> <unquoted interval string> <quote>
<unquoted interval string> ::=
[ <sign> ] { <year-month literal> | <day-time literal> }
<year-month literal> ::=
<years value> [ <minus sign> <months value> ]
| <months value>
<years value> ::=
<datetime value>
<months value> ::=
<datetime value>
<datetime value> ::=
<unsigned integer>
<unsigned integer> ::= <digit>...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622767866
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,31 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])*?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR TO MONTH)$".r
Review comment:
> I still don't understand `([+|-])*?`. Do we support `INTERVAL ++'1-1' YEAR TO MONTH`?
Updated
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,31 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])*?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR TO MONTH)$".r
Review comment:
> `YEAR TO MONTH` should use `\s+`
Done
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,31 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])*?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR TO MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
+ case yearMonthPattern("-", year, month) =>
+ fromYearMonthString(s"-$year-$month")
Review comment:
> in `fromYearMonthString`, there is an inner method `toInterval`. Can we move it out and call it here?
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829006869
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42592/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829007853
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823942130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824062976
**[Test build #137725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137725/testReport)** for PR 32266 at commit [`691c1f4`](https://github.com/apache/spark/commit/691c1f406d2d13a87c5748c6be758b91e9d844e2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622798208
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,51 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
Review comment:
this is duplicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829004026
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42592/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826289678
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137905/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622798566
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,51 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
Review comment:
ditto
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,51 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("interval '10-1' year TO MONTH"),
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824977520
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137803/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623553133
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,25 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
Review comment:
Thanks for you share
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1354,6 +1362,12 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
+ private[this] def castToYearMonthIntervalCode(from: DataType): CastFunction = from match {
+ case StringType =>
+ val util = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
+ (c, evPrim, evNull) => code"$evPrim = $util.castStringToYMInterval($c);"
Review comment:
yea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622336292
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,31 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])*?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR TO MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
+ case yearMonthPattern("-", year, month) =>
+ fromYearMonthString(s"-$year-$month")
Review comment:
in `fromYearMonthString`, there is an inner method `toInterval`. Can we move it out and call it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828723505
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138047/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826257657
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42429/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824839668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828348780
**[Test build #138044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138044/testReport)** for PR 32266 at commit [`ce69004`](https://github.com/apache/spark/commit/ce690041d1ee9dd26c1ced6087ff23cd45d2c0ef).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r620147289
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
Ok, how about current approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824792853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42332/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622806738
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,51 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("interval '10-1' year TO MONTH"),
Review comment:
Done
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,51 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
Review comment:
Done
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,51 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
Review comment:
Done
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,51 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
Review comment:
Done
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -127,20 +123,21 @@ object IntervalUtils {
require(input != null, "Interval year-month string must be not null")
input.trim match {
case yearMonthPattern("-", yearStr, monthStr) =>
- toYMInterval(yearStr, monthStr, -1)
+ new CalendarInterval(toYMInterval(yearStr, monthStr, -1), 0, 0)
case yearMonthPattern(_, yearStr, monthStr) =>
- toYMInterval(yearStr, monthStr, 1)
+ new CalendarInterval(toYMInterval(yearStr, monthStr, 1), 0, 0)
case _ =>
throw new IllegalArgumentException(
s"Interval string does not match year-month format of 'y-m': $input")
}
}
- def toYMInterval(yearStr: String, monthStr: String, sign: Int): CalendarInterval = {
+ def toYMInterval(yearStr: String, monthStr: String, sign: Int): Int = {
try {
val years = toLongWithRange(YEAR, yearStr, 0, Integer.MAX_VALUE / MONTHS_PER_YEAR)
val totalMonths = sign * (years * MONTHS_PER_YEAR + toLongWithRange(MONTH, monthStr, 0, 11))
- new CalendarInterval(Math.toIntExact(totalMonths), 0, 0)
+// new CalendarInterval(Math.toIntExact(totalMonths), 0, 0)
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826251054
**[Test build #137905 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137905/testReport)** for PR 32266 at commit [`3b84baa`](https://github.com/apache/spark/commit/3b84baae46936929212b7f1b57dc27e08fd9c564).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825625763
**[Test build #137862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137862/testReport)** for PR 32266 at commit [`d19bbc8`](https://github.com/apache/spark/commit/d19bbc82d33ced40c93e00954610a521566642ac).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826254842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622787481
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,50 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("interval '10-1' year TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create(null, StringType), YearMonthIntervalType), null)
+
+ Seq("INTERVAL '0-0' YEAR TO MONTH", "INTERVAL '10-1' YEAR TO MONTH",
Review comment:
We can merge the tests above into this loop
```
val hiveStyleInterval = interval.split("'")(1)
// test with both interval and hiveStyleInterval
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828565578
**[Test build #138044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138044/testReport)** for PR 32266 at commit [`ce69004`](https://github.com/apache/spark/commit/ce690041d1ee9dd26c1ced6087ff23cd45d2c0ef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828381373
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42563/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r620960081
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,45 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '0-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("interval '10-1' year TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create(null, StringType), YearMonthIntervalType), null)
+
+ Seq("INTERVAL '0-0' YEAR TO MONTH" -> "INTERVAL '0-0' YEAR TO MONTH",
+ "INTERVAL '10-1' YEAR TO MONTH" -> "INTERVAL '10-1' YEAR TO MONTH",
+ "INTERVAL '-178956970-7' YEAR TO MONTH" -> "INTERVAL '-178956970-7' YEAR TO MONTH",
+ "INTERVAL '178956970-7' YEAR TO MONTH" -> "INTERVAL '178956970-7' YEAR TO MONTH",
+ "INTERVAL '-178956970-8' YEAR TO MONTH" -> "INTERVAL '-178956970-8' YEAR TO MONTH"
Review comment:
they key and value are always the same?
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,45 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '0-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("interval '10-1' year TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create(null, StringType), YearMonthIntervalType), null)
+
+ Seq("INTERVAL '0-0' YEAR TO MONTH" -> "INTERVAL '0-0' YEAR TO MONTH",
+ "INTERVAL '10-1' YEAR TO MONTH" -> "INTERVAL '10-1' YEAR TO MONTH",
+ "INTERVAL '-178956970-7' YEAR TO MONTH" -> "INTERVAL '-178956970-7' YEAR TO MONTH",
+ "INTERVAL '178956970-7' YEAR TO MONTH" -> "INTERVAL '178956970-7' YEAR TO MONTH",
+ "INTERVAL '-178956970-8' YEAR TO MONTH" -> "INTERVAL '-178956970-8' YEAR TO MONTH"
Review comment:
the key and value are always the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623230819
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,25 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): Int = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
Review comment:
You can specify that your pattern is case insensitive, and remove `.toUpperCase(Locale.ROOT)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829155964
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138073/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829243915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138079/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829759396
**[Test build #138096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138096/testReport)** for PR 32266 at commit [`5ca83ab`](https://github.com/apache/spark/commit/5ca83abe001897216575fad9f00906de553626c6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622790178
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,30 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
+ case yearMonthPattern("-", year, month) => toYMInterval(year, month, -1)
+ case yearMonthPattern(_, year, month) => toYMInterval(year, month, 1)
+ case yearMonthStringPattern(_, prefixSign, _, suffixSign, year, month, _, _) =>
+ (prefixSign, suffixSign) match {
+ case ("-", "-") => toYMInterval(year, month, 1)
+ case ("-", _) => toYMInterval(year, month, -1)
+ case (_, "-") => toYMInterval(year, month, -1)
+ case (_, _) => toYMInterval(year, month, 1)
+ }
+ case yearMonthStringPattern(_, null, _, "-", year, month, _, _) =>
Review comment:
yea, after change the code, we don't need this case now.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,30 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
+ case yearMonthPattern("-", year, month) => toYMInterval(year, month, -1)
+ case yearMonthPattern(_, year, month) => toYMInterval(year, month, 1)
+ case yearMonthStringPattern(_, prefixSign, _, suffixSign, year, month, _, _) =>
+ (prefixSign, suffixSign) match {
+ case ("-", "-") => toYMInterval(year, month, 1)
+ case ("-", _) => toYMInterval(year, month, -1)
+ case (_, "-") => toYMInterval(year, month, -1)
+ case (_, _) => toYMInterval(year, month, 1)
+ }
+ case yearMonthStringPattern(_, null, _, "-", year, month, _, _) =>
+ toYMInterval(year, month, -1)
+ case yearMonthStringPattern(_, null, _, _, year, month, _, _) =>
Review comment:
Done
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,30 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
+ case yearMonthPattern("-", year, month) => toYMInterval(year, month, -1)
+ case yearMonthPattern(_, year, month) => toYMInterval(year, month, 1)
+ case yearMonthStringPattern(_, prefixSign, _, suffixSign, year, month, _, _) =>
Review comment:
DOne
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -100,28 +125,29 @@ object IntervalUtils {
*/
def fromYearMonthString(input: String): CalendarInterval = {
require(input != null, "Interval year-month string must be not null")
- def toInterval(yearStr: String, monthStr: String, sign: Int): CalendarInterval = {
- try {
- val years = toLongWithRange(YEAR, yearStr, 0, Integer.MAX_VALUE / MONTHS_PER_YEAR)
- val totalMonths = sign * (years * MONTHS_PER_YEAR + toLongWithRange(MONTH, monthStr, 0, 11))
- new CalendarInterval(Math.toIntExact(totalMonths), 0, 0)
- } catch {
- case NonFatal(e) =>
- throw new IllegalArgumentException(
- s"Error parsing interval year-month string: ${e.getMessage}", e)
- }
- }
input.trim match {
case yearMonthPattern("-", yearStr, monthStr) =>
- toInterval(yearStr, monthStr, -1)
+ toYMInterval(yearStr, monthStr, -1)
case yearMonthPattern(_, yearStr, monthStr) =>
- toInterval(yearStr, monthStr, 1)
+ toYMInterval(yearStr, monthStr, 1)
case _ =>
throw new IllegalArgumentException(
s"Interval string does not match year-month format of 'y-m': $input")
}
}
+ def toYMInterval(yearStr: String, monthStr: String, sign: Int): CalendarInterval = {
Review comment:
DOne
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824636620
**[Test build #137791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137791/testReport)** for PR 32266 at commit [`62d175b`](https://github.com/apache/spark/commit/62d175bf9a9ae89a0b5a150e14583f5d850c4cda).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828715750
**[Test build #138047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138047/testReport)** for PR 32266 at commit [`092d01a`](https://github.com/apache/spark/commit/092d01acb4e412f09f58270678438ca903201e37).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829782930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42616/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829013446
**[Test build #138076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138076/testReport)** for PR 32266 at commit [`253c70e`](https://github.com/apache/spark/commit/253c70ea166a8e133dccef2cbe1c589be77aa861).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r620938213
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1354,6 +1362,13 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
+ private[this] def castToYearMonthIntervalCode(from: DataType): CastFunction = from match {
+ case StringType =>
+ val util = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
+ (c, evPrim, evNull) =>
+ code"""$evPrim = $util.fromYearMonthString($c).months;""".stripMargin
Review comment:
this is a single line, we don't need to use the multiline string format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622784637
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,30 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): CalendarInterval = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
+ case yearMonthPattern("-", year, month) => toYMInterval(year, month, -1)
+ case yearMonthPattern(_, year, month) => toYMInterval(year, month, 1)
+ case yearMonthStringPattern(_, prefixSign, _, suffixSign, year, month, _, _) =>
+ (prefixSign, suffixSign) match {
+ case ("-", "-") => toYMInterval(year, month, 1)
+ case ("-", _) => toYMInterval(year, month, -1)
+ case (_, "-") => toYMInterval(year, month, -1)
+ case (_, _) => toYMInterval(year, month, 1)
+ }
+ case yearMonthStringPattern(_, null, _, "-", year, month, _, _) =>
Review comment:
isn't it covered by the case above?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619736559
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,23 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
Review comment:
I am discussing this here https://github.com/apache/spark/pull/32266#discussion_r619231878 with @MaxGekk ,
Do you have any other supplementary suggestions?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,24 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
+ try {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ val regex = "INTERVAL '([-|+]?[0-9]+-[-|+]?[0-9]+)' YEAR TO MONTH".r
Review comment:
> but ideally we should respect the SQL config `spark.sql.caseSensitive`):
Spark Catalyst parser not respect `spark.sql.caseSensitive` too, should we respect it here?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -108,17 +119,32 @@ object IntervalUtils {
} catch {
case NonFatal(e) =>
throw new IllegalArgumentException(
- s"Error parsing interval year-month string: ${e.getMessage}", e)
+ s"$errorPrefix: ${e.getMessage}", e)
}
}
+
input.trim match {
case yearMonthPattern("-", yearStr, monthStr) =>
toInterval(yearStr, monthStr, -1)
case yearMonthPattern(_, yearStr, monthStr) =>
toInterval(yearStr, monthStr, 1)
case _ =>
- throw new IllegalArgumentException(
- s"Interval string does not match year-month format of 'y-m': $input")
+ try {
+ CatalystSqlParser.parseExpression(input) match {
+ case Literal(value: Int, _: YearMonthIntervalType) => new CalendarInterval(value, 0, 0)
+ case Literal(value: CalendarInterval, _: CalendarIntervalType) => value
+ case _ => throw new IllegalArgumentException(
+ s"Interval string does not match year-month format of 'y-m': $input")
+ }
+ } catch {
+ case NonFatal(e) =>
+ if (e.getMessage.contains(errorPrefix)) {
+ throw new IllegalArgumentException(e.getMessage)
Review comment:
One weird thing here is the real message is
```
Error parsing interval year-month string: integer overflow(line 1, pos 9)
== SQL ==
INTERVAL '-178956970-9' YEAR TO MONTH
---------^^^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618224464
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,28 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
+
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
Review comment:
> It's a new feature, we can use the ANSI behavior directly.
So just let it throw exception? Update later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618217375
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,6 +93,28 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthFuzzyPattern = "[^-|+]*?([+|-]?[\\d]+-[\\d]+).*".r
+
+ def safeFromYearMonthString(input: UTF8String): Option[Int] = {
Review comment:
It's a new feature, we can use the ANSI behavior directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823876777
FYI @MaxGekk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622783760
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -100,28 +125,29 @@ object IntervalUtils {
*/
def fromYearMonthString(input: String): CalendarInterval = {
require(input != null, "Interval year-month string must be not null")
- def toInterval(yearStr: String, monthStr: String, sign: Int): CalendarInterval = {
- try {
- val years = toLongWithRange(YEAR, yearStr, 0, Integer.MAX_VALUE / MONTHS_PER_YEAR)
- val totalMonths = sign * (years * MONTHS_PER_YEAR + toLongWithRange(MONTH, monthStr, 0, 11))
- new CalendarInterval(Math.toIntExact(totalMonths), 0, 0)
- } catch {
- case NonFatal(e) =>
- throw new IllegalArgumentException(
- s"Error parsing interval year-month string: ${e.getMessage}", e)
- }
- }
input.trim match {
case yearMonthPattern("-", yearStr, monthStr) =>
- toInterval(yearStr, monthStr, -1)
+ toYMInterval(yearStr, monthStr, -1)
case yearMonthPattern(_, yearStr, monthStr) =>
- toInterval(yearStr, monthStr, 1)
+ toYMInterval(yearStr, monthStr, 1)
case _ =>
throw new IllegalArgumentException(
s"Interval string does not match year-month format of 'y-m': $input")
}
}
+ def toYMInterval(yearStr: String, monthStr: String, sign: Int): CalendarInterval = {
Review comment:
This can return Int directly. `fromYearMonthString` should wrap the int with `CalendarInterval`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622792833
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,50 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
+ checkEvaluation(cast(Literal.create("INTERVAL '1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("0-0"), YearMonthIntervalType), 0)
+ checkEvaluation(cast(Literal.create("INTERVAL '-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL -'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'-1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'+1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("INTERVAL +'1-0' YEAR TO MONTH"),
+ YearMonthIntervalType), 12)
+ checkEvaluation(cast(Literal.create("-1-0"), YearMonthIntervalType), -12)
+ checkEvaluation(cast(Literal.create("INTERVAL '10-1' YEAR TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("10-1"), YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create("interval '10-1' year TO MONTH"),
+ YearMonthIntervalType), 121)
+ checkEvaluation(cast(Literal.create(null, StringType), YearMonthIntervalType), null)
+
+ Seq("INTERVAL '0-0' YEAR TO MONTH", "INTERVAL '10-1' YEAR TO MONTH",
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r618166216
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##########
@@ -1774,6 +1774,19 @@ class CastSuite extends CastSuiteBase {
assert(e3.contains("Casting 2147483648 to int causes overflow"))
}
}
+
+ test("SPARK-35111: Cast string to year-month interval") {
Review comment:
> Could you test corner cases when arithmetic overflow happens. Also test upper and lower cases.
Current test is not correct according to https://github.com/apache/spark/pull/32281, will update after https://github.com/apache/spark/pull/32281 merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828978710
**[Test build #138073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138073/testReport)** for PR 32266 at commit [`3adde87`](https://github.com/apache/spark/commit/3adde873e1f0b4ad1ea5e468642ef4b76da2aaf4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828978710
**[Test build #138073 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138073/testReport)** for PR 32266 at commit [`3adde87`](https://github.com/apache/spark/commit/3adde873e1f0b4ad1ea5e468642ef4b76da2aaf4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-824748308
**[Test build #137803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137803/testReport)** for PR 32266 at commit [`6d14414`](https://github.com/apache/spark/commit/6d144147721001253cf515dc13b4cee615948463).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk closed pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk closed pull request #32266:
URL: https://github.com/apache/spark/pull/32266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-823880542
**[Test build #137721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137721/testReport)** for PR 32266 at commit [`9546a0f`](https://github.com/apache/spark/commit/9546a0f22f62e7017d73530cc691223e0ca85587).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-825791851
**[Test build #137862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137862/testReport)** for PR 32266 at commit [`d19bbc8`](https://github.com/apache/spark/commit/d19bbc82d33ced40c93e00954610a521566642ac).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829009391
**[Test build #138075 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138075/testReport)** for PR 32266 at commit [`2c8785b`](https://github.com/apache/spark/commit/2c8785b2a4515ca7a30bf2ef8c5312e81ff3e5f6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r619745692
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -108,17 +119,32 @@ object IntervalUtils {
} catch {
case NonFatal(e) =>
throw new IllegalArgumentException(
- s"Error parsing interval year-month string: ${e.getMessage}", e)
+ s"$errorPrefix: ${e.getMessage}", e)
}
}
+
input.trim match {
case yearMonthPattern("-", yearStr, monthStr) =>
toInterval(yearStr, monthStr, -1)
case yearMonthPattern(_, yearStr, monthStr) =>
toInterval(yearStr, monthStr, 1)
case _ =>
- throw new IllegalArgumentException(
- s"Interval string does not match year-month format of 'y-m': $input")
+ try {
+ CatalystSqlParser.parseExpression(input) match {
+ case Literal(value: Int, _: YearMonthIntervalType) => new CalendarInterval(value, 0, 0)
+ case Literal(value: CalendarInterval, _: CalendarIntervalType) => value
+ case _ => throw new IllegalArgumentException(
+ s"Interval string does not match year-month format of 'y-m': $input")
+ }
+ } catch {
+ case NonFatal(e) =>
+ if (e.getMessage.contains(errorPrefix)) {
+ throw new IllegalArgumentException(e.getMessage)
Review comment:
One weird thing here is the real message is
```
Error parsing interval year-month string: integer overflow(line 1, pos 9)
== SQL ==
INTERVAL '-178956970-9' YEAR TO MONTH
---------^^^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r620963245
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -93,13 +95,22 @@ object IntervalUtils {
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ def fromYearMonthString(input: UTF8String): CalendarInterval = {
+ if (input == null || input.toString == null) {
+ throw new IllegalArgumentException("Interval year-month string must be not null")
+ } else {
+ fromYearMonthString(input.trimAll().toString)
+ }
+ }
+
/**
* Parse YearMonth string in form: [+|-]YYYY-MM
*
* adapted from HiveIntervalYearMonth.valueOf
*/
def fromYearMonthString(input: String): CalendarInterval = {
Review comment:
Now I understand why we need a new version of `fromYearMonthString`, as we need to allow `INTERVAL ... YEAR TO MONTH`. Let's make it clear in the PR description.
Let's add a new method `castStringToYMInterval`, and define a new regex for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r622334319
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,31 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])*?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR TO MONTH)$".r
Review comment:
I still don't understand `([+|-])*?`. Do we support `INTERVAL ++'1-1' YEAR TO MONTH`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-829155964
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138073/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623553106
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
##########
@@ -92,6 +93,25 @@ object IntervalUtils {
}
private val yearMonthPattern = "^([+|-])?(\\d+)-(\\d+)$".r
+ private val yearMonthStringPattern =
+ "^(INTERVAL\\s+)([+|-])?(')([+|-])?(\\d+)-(\\d+)(')(\\s+YEAR\\s+TO\\s+MONTH)$".r
+
+ def castStringToYMInterval(input: UTF8String): Int = {
+ input.trimAll().toString.toUpperCase(Locale.ROOT) match {
Review comment:
> regular
Match case ` -1-0 `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828567902
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138044/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828567902
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #32266:
URL: https://github.com/apache/spark/pull/32266#discussion_r623236295
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##########
@@ -1354,6 +1362,12 @@ abstract class CastBase extends UnaryExpression with TimeZoneAwareExpression wit
}
+ private[this] def castToYearMonthIntervalCode(from: DataType): CastFunction = from match {
+ case StringType =>
+ val util = IntervalUtils.getClass.getCanonicalName.stripSuffix("$")
+ (c, evPrim, evNull) => code"$evPrim = $util.castStringToYMInterval($c);"
Review comment:
nit: evNull is not used
```suggestion
(c, evPrim, _) => code"$evPrim = $util.castStringToYMInterval($c);"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-828381373
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42563/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32266: [SPARK-35111][SQL] Support Cast string to year-month interval
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32266:
URL: https://github.com/apache/spark/pull/32266#issuecomment-826257657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org