You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/03 09:36:36 UTC
[GitHub] [spark] yaooqinn opened a new pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
yaooqinn opened a new pull request #29635:
URL: https://github.com/apache/spark/pull/29635
### What changes were proposed in this pull request?
bugfix for incomplete interval values, e.g. interval '1', interval '1 day 2', currently these cases will result null, but actually we should fail them with IllegalArgumentsException
### Why are the changes needed?
correctness
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
yes, incomplete intervals will throw exception now
#### before
```
bin/spark-sql -S -e "select interval '1', interval '+', interval '1 day -'"
NULL NULL NULL
```
#### after
```
-- !query
select interval '1'
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException
Cannot parse the INTERVAL value: 1(line 1, pos 7)
== SQL ==
select interval '1'
```
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
unit tests added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686404729
**[Test build #128254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128254/testReport)** for PR 29635 at commit [`eb99bd5`](https://github.com/apache/spark/commit/eb99bd53b2050e13c4fa92f8faa06693e3b6556c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686375775
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686374656
cc @cloud-fan @MaxGekk @maropu @HyukjinKwon thanks for reviewing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29635:
URL: https://github.com/apache/spark/pull/29635#discussion_r482926096
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala
##########
@@ -77,6 +77,19 @@ class IntervalUtilsSuite extends SparkFunSuite with SQLHelper {
}
}
+ test("string to interval: interval with dangling parts should not results null") {
+ checkFromInvalidString("+", "dangling interval value sign '+'")
+ checkFromInvalidString("-", "dangling interval value sign '-'")
+ checkFromInvalidString("+ 2", "dangling interval value '2'")
+ checkFromInvalidString("- 1", "dangling interval value '1'")
+ checkFromInvalidString("1", "dangling interval value '1'")
Review comment:
how about `Expect a unit after '1' but hit EOL`
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala
##########
@@ -77,6 +77,19 @@ class IntervalUtilsSuite extends SparkFunSuite with SQLHelper {
}
}
+ test("string to interval: interval with dangling parts should not results null") {
+ checkFromInvalidString("+", "dangling interval value sign '+'")
+ checkFromInvalidString("-", "dangling interval value sign '-'")
+ checkFromInvalidString("+ 2", "dangling interval value '2'")
+ checkFromInvalidString("- 1", "dangling interval value '1'")
+ checkFromInvalidString("1", "dangling interval value '1'")
Review comment:
how about `Expect a unit name after '1' but hit EOL`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-687206192
**[Test build #128293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128293/testReport)** for PR 29635 at commit [`daed4e6`](https://github.com/apache/spark/commit/daed4e68fa8b9ce44ef88078d1cb0405ec6d3fe5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #29635:
URL: https://github.com/apache/spark/pull/29635#discussion_r507241899
##########
File path: docs/sql-migration-guide.md
##########
@@ -40,6 +40,8 @@ license: |
- In Spark 3.1, `path` option cannot coexist when the following methods are called with path parameter(s): `DataFrameReader.load()`, `DataFrameWriter.save()`, `DataStreamReader.load()`, or `DataStreamWriter.start()`. In addition, `paths` option cannot coexist for `DataFrameReader.load()`. For example, `spark.read.format("csv").option("path", "/tmp").load("/tmp2")` or `spark.read.option("path", "/tmp").csv("/tmp2")` will throw `org.apache.spark.sql.AnalysisException`. In Spark version 3.0 and below, `path` option is overwritten if one path parameter is passed to above methods; `path` option is added to the overall paths if multiple path parameters are passed to `DataFrameReader.load()`. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.pathOptionBehavior.enabled` to `true`.
+ - In Spark 3.1, incomplete interval literals, e.g. `INTERVAL '1'`, `INTERVAL '1 DAY 2'` will fail with IllegalArgumentException. In Spark 3.0, they result `NULL`s.
Review comment:
> In Spark 3.1, incomplete interval literals, e.g. `INTERVAL '1'`, `INTERVAL '1 DAY 2'` will fail with IllegalArgumentException. In Spark 3.0, they result `NULL`s.
=>
> In Spark 3.1, IllegalArgumentException is returned for the incomplete interval literals, e.g. `INTERVAL '1'`, `INTERVAL '1 DAY 2'`, which are invalid. In Spark 3.0, these literals result in `NULL`s.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686572456
**[Test build #128254 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128254/testReport)** for PR 29635 at commit [`eb99bd5`](https://github.com/apache/spark/commit/eb99bd53b2050e13c4fa92f8faa06693e3b6556c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29635:
URL: https://github.com/apache/spark/pull/29635#discussion_r482925854
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala
##########
@@ -77,6 +77,19 @@ class IntervalUtilsSuite extends SparkFunSuite with SQLHelper {
}
}
+ test("string to interval: interval with dangling parts should not results null") {
+ checkFromInvalidString("+", "dangling interval value sign '+'")
+ checkFromInvalidString("-", "dangling interval value sign '-'")
Review comment:
how about `Expect a number after '-' but hit EOL`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686982439
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686394052
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128248/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686405271
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #29635:
URL: https://github.com/apache/spark/pull/29635#discussion_r483040796
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala
##########
@@ -77,6 +77,19 @@ class IntervalUtilsSuite extends SparkFunSuite with SQLHelper {
}
}
+ test("string to interval: interval with dangling parts should not results null") {
+ checkFromInvalidString("+", "dangling interval value sign '+'")
+ checkFromInvalidString("-", "dangling interval value sign '-'")
+ checkFromInvalidString("+ 2", "dangling interval value '2'")
+ checkFromInvalidString("- 1", "dangling interval value '1'")
+ checkFromInvalidString("1", "dangling interval value '1'")
Review comment:
done, but looks a bit weird for `interval '-.'` to expect a unit name
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-687042645
**[Test build #128293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128293/testReport)** for PR 29635 at commit [`daed4e6`](https://github.com/apache/spark/commit/daed4e68fa8b9ce44ef88078d1cb0405ec6d3fe5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686547718
**[Test build #128261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128261/testReport)** for PR 29635 at commit [`ae6d9be`](https://github.com/apache/spark/commit/ae6d9beb118cab81fe7f2273ed76405d73c710d7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686574428
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-687028437
yea we should target 3.0. Let's add migration guide for 3.1 in this PR, and do the same for 3.0.2 in the backport PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686548517
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-687207847
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-687043161
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686394046
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686394046
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686724454
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-687042645
**[Test build #128293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128293/testReport)** for PR 29635 at commit [`daed4e6`](https://github.com/apache/spark/commit/daed4e68fa8b9ce44ef88078d1cb0405ec6d3fe5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686405271
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686548517
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686375045
**[Test build #128248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128248/testReport)** for PR 29635 at commit [`6d27597`](https://github.com/apache/spark/commit/6d2759715ba0375dfe40d4143eaf05de011afee2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686574428
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686981169
can we add a migration guide?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686404729
**[Test build #128254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128254/testReport)** for PR 29635 at commit [`eb99bd5`](https://github.com/apache/spark/commit/eb99bd53b2050e13c4fa92f8faa06693e3b6556c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #29635:
URL: https://github.com/apache/spark/pull/29635#discussion_r508950659
##########
File path: docs/sql-migration-guide.md
##########
@@ -40,6 +40,8 @@ license: |
- In Spark 3.1, `path` option cannot coexist when the following methods are called with path parameter(s): `DataFrameReader.load()`, `DataFrameWriter.save()`, `DataStreamReader.load()`, or `DataStreamWriter.start()`. In addition, `paths` option cannot coexist for `DataFrameReader.load()`. For example, `spark.read.format("csv").option("path", "/tmp").load("/tmp2")` or `spark.read.option("path", "/tmp").csv("/tmp2")` will throw `org.apache.spark.sql.AnalysisException`. In Spark version 3.0 and below, `path` option is overwritten if one path parameter is passed to above methods; `path` option is added to the overall paths if multiple path parameters are passed to `DataFrameReader.load()`. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.pathOptionBehavior.enabled` to `true`.
+ - In Spark 3.1, incomplete interval literals, e.g. `INTERVAL '1'`, `INTERVAL '1 DAY 2'` will fail with IllegalArgumentException. In Spark 3.0, they result `NULL`s.
Review comment:
OK, I will raise a PR today~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-688039775
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686375045
**[Test build #128248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128248/testReport)** for PR 29635 at commit [`6d27597`](https://github.com/apache/spark/commit/6d2759715ba0375dfe40d4143eaf05de011afee2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686723629
**[Test build #128261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128261/testReport)** for PR 29635 at commit [`ae6d9be`](https://github.com/apache/spark/commit/ae6d9beb118cab81fe7f2273ed76405d73c710d7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686547718
**[Test build #128261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128261/testReport)** for PR 29635 at commit [`ae6d9be`](https://github.com/apache/spark/commit/ae6d9beb118cab81fe7f2273ed76405d73c710d7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686724443
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-687207847
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
yaooqinn removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686982450
OK, are you targeting this to 3.0 or not?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686393991
**[Test build #128248 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128248/testReport)** for PR 29635 at commit [`6d27597`](https://github.com/apache/spark/commit/6d2759715ba0375dfe40d4143eaf05de011afee2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-686375775
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #29635:
URL: https://github.com/apache/spark/pull/29635
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29635:
URL: https://github.com/apache/spark/pull/29635#issuecomment-687043161
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29635: [SPARK-32785][SQL] Interval with dangling parts should not results null
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29635:
URL: https://github.com/apache/spark/pull/29635#discussion_r508941830
##########
File path: docs/sql-migration-guide.md
##########
@@ -40,6 +40,8 @@ license: |
- In Spark 3.1, `path` option cannot coexist when the following methods are called with path parameter(s): `DataFrameReader.load()`, `DataFrameWriter.save()`, `DataStreamReader.load()`, or `DataStreamWriter.start()`. In addition, `paths` option cannot coexist for `DataFrameReader.load()`. For example, `spark.read.format("csv").option("path", "/tmp").load("/tmp2")` or `spark.read.option("path", "/tmp").csv("/tmp2")` will throw `org.apache.spark.sql.AnalysisException`. In Spark version 3.0 and below, `path` option is overwritten if one path parameter is passed to above methods; `path` option is added to the overall paths if multiple path parameters are passed to `DataFrameReader.load()`. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.pathOptionBehavior.enabled` to `true`.
+ - In Spark 3.1, incomplete interval literals, e.g. `INTERVAL '1'`, `INTERVAL '1 DAY 2'` will fail with IllegalArgumentException. In Spark 3.0, they result `NULL`s.
Review comment:
Could you fix it in followup, @yaooqinn ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org