You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/03 10:59:11 UTC
[GitHub] [spark] MaxGekk opened a new pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
MaxGekk opened a new pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441
### What changes were proposed in this pull request?
In the PR, I propose to partially revert the commit https://github.com/apache/spark/commit/51a6ba0181a013f2b62b47184785a8b6f6a78f12, and provide a legacy parser based on `FastDateFormat` which is compatible to `SimpleDateFormat`.
To enable the legacy parser, set `spark.sql.legacy.timeParser.enabled` to `true`.
### Why are the changes needed?
To allow users to restore old behavior in parsing timestamps/dates using `SimpleDateFormat` patterns. The main reason for restoring is `DateTimeFormatter`'s patterns are not fully compatible to `SimpleDateFormat` patterns, see https://issues.apache.org/jira/browse/SPARK-30668
### Does this PR introduce any user-facing change?
Yes
### How was this patch tested?
- Added new test to `DateFunctionsSuite`
- Restored additional test cases in `JsonInferSchemaSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581836545
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r376025257
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -789,4 +789,18 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
Row(Timestamp.valueOf("2015-07-24 07:00:00")),
Row(Timestamp.valueOf("2015-07-24 22:00:00"))))
}
+
+ test("SPARK-30668: use legacy timestamp parser in to_timestamp") {
+ def checkTimeZoneParsing(expected: Any): Unit = {
+ val df = Seq("2020-01-27T20:06:11.847-0800").toDF("ts")
+ checkAnswer(df.select(to_timestamp(col("ts"), "yyyy-MM-dd'T'HH:mm:ss.SSSz")),
+ Row(expected))
+ }
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> "true") {
+ checkTimeZoneParsing(Timestamp.valueOf("2020-01-27 20:06:11.847"))
+ }
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> "false") {
+ checkTimeZoneParsing(null)
Review comment:
Silent fallback to old parser can lead to mixed values in the same column - some in combined calendar Julian+Gregorian another in Proleptic Gregorian calendar.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581360845
**[Test build #117770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117770/testReport)** for PR 27441 at commit [`af38b87`](https://github.com/apache/spark/commit/af38b879dab37b5a1bad62ab528eae6fa008916e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581854198
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581728336
a separate item sounds good, but don't forget to remove the existing item.
> TODO: Need to update all docs like
I don't think so. Legacy configs are interval and are not expected to be turned on in most cases. We don't need to mention the legacy config in user-facing documents.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581853679
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581458867
**[Test build #117770 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117770/testReport)** for PR 27441 at commit [`af38b87`](https://github.com/apache/spark/commit/af38b879dab37b5a1bad62ab528eae6fa008916e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581784141
> Legacy configs are interval ...
At the moment, the config is not internal. I will make it as internal one. Also I reviewed other legacy configs, and made some of them internal: https://github.com/apache/spark/pull/27448
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581851365
jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581459729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117770/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581836553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22589/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546977
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581358363
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546994
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22544/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581357747
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546994
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22544/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581853684
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117830/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582205149
**[Test build #117861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117861/testReport)** for PR 27441 at commit [`19a47c1`](https://github.com/apache/spark/commit/19a47c1cd08ce6e2e00e37abdc8fd6c4b4311b4e).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class PlanAdaptiveSubqueries(subqueryMap: Map[Long, SubqueryExec]) extends Rule[SparkPlan] `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581357747
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582134406
**[Test build #117861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117861/testReport)** for PR 27441 at commit [`19a47c1`](https://github.com/apache/spark/commit/19a47c1cd08ce6e2e00e37abdc8fd6c4b4311b4e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546397
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117782/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581358262
jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582390520
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581835986
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117827/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582205512
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581357753
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117769/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582205521
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117861/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581835977
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581784141
> Legacy configs are interval ...
At the moment, the config is not internal. I will make it as internal one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581358363
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582286704
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22671/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581836545
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581522914
> since we use the new formatter by default, shall we try the old formatter if the new formatter doesn't work? Then users don't need to enable the legacy config for some cases.
This may have some downsides like some values could be parsed by old parser which is based on combined calendar (Julian + Gregorian) but another values by new one which uses Proleptic Gregorian calendar. This could make user life harder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581522173
Hi, @gatorsmile . Since you are the issue reporter, could you confirm that you are okay with this solution?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581361258
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581853679
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546385
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r376021523
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -789,4 +789,18 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
Row(Timestamp.valueOf("2015-07-24 07:00:00")),
Row(Timestamp.valueOf("2015-07-24 22:00:00"))))
}
+
+ test("SPARK-30668: use legacy timestamp parser in to_timestamp") {
+ def checkTimeZoneParsing(expected: Any): Unit = {
+ val df = Seq("2020-01-27T20:06:11.847-0800").toDF("ts")
+ checkAnswer(df.select(to_timestamp(col("ts"), "yyyy-MM-dd'T'HH:mm:ss.SSSz")),
+ Row(expected))
+ }
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> "true") {
+ checkTimeZoneParsing(Timestamp.valueOf("2020-01-27 20:06:11.847"))
+ }
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> "false") {
+ checkTimeZoneParsing(null)
Review comment:
fallback to the old parser?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r374062911
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2146,6 +2146,13 @@ object SQLConf {
.checkValue(_ > 0, "The value of spark.sql.addPartitionInBatch.size must be positive")
.createWithDefault(100)
+ val LEGACY_TIME_PARSER_ENABLED = buildConf("spark.sql.legacy.timeParser.enabled")
+ .doc("When set to true, java.text.SimpleDateFormat is used for formatting and parsing " +
+ " dates/timestamps in a locale-sensitive manner. When set to false, classes from " +
Review comment:
nit: extra space at the beginning
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582284712
jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582286322
**[Test build #117909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117909/testReport)** for PR 27441 at commit [`19a47c1`](https://github.com/apache/spark/commit/19a47c1cd08ce6e2e00e37abdc8fd6c4b4311b4e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582286704
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22671/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r374255072
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -789,4 +789,12 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
Row(Timestamp.valueOf("2015-07-24 07:00:00")),
Row(Timestamp.valueOf("2015-07-24 22:00:00"))))
}
+
+ test("SPARK-30668: use legacy timestamp parser in to_timestamp") {
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> "true") {
+ val df = Seq("2020-01-27T20:06:11.847-0800").toDF("ts")
+ checkAnswer(df.select(to_timestamp(col("ts"), "yyyy-MM-dd'T'HH:mm:ss.SSSz")),
+ Row(Timestamp.valueOf("2020-01-27 20:06:11.847")))
+ }
Review comment:
I will add the check
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r374062911
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2146,6 +2146,13 @@ object SQLConf {
.checkValue(_ > 0, "The value of spark.sql.addPartitionInBatch.size must be positive")
.createWithDefault(100)
+ val LEGACY_TIME_PARSER_ENABLED = buildConf("spark.sql.legacy.timeParser.enabled")
+ .doc("When set to true, java.text.SimpleDateFormat is used for formatting and parsing " +
+ " dates/timestamps in a locale-sensitive manner. When set to false, classes from " +
Review comment:
nit: no extra space at the beginning
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582205512
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581853684
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117830/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582389759
**[Test build #117909 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117909/testReport)** for PR 27441 at commit [`19a47c1`](https://github.com/apache/spark/commit/19a47c1cd08ce6e2e00e37abdc8fd6c4b4311b4e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class PlanAdaptiveSubqueries(subqueryMap: Map[Long, SubqueryExec]) extends Rule[SparkPlan] `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r376026192
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -789,4 +789,18 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
Row(Timestamp.valueOf("2015-07-24 07:00:00")),
Row(Timestamp.valueOf("2015-07-24 22:00:00"))))
}
+
+ test("SPARK-30668: use legacy timestamp parser in to_timestamp") {
+ def checkTimeZoneParsing(expected: Any): Unit = {
+ val df = Seq("2020-01-27T20:06:11.847-0800").toDF("ts")
+ checkAnswer(df.select(to_timestamp(col("ts"), "yyyy-MM-dd'T'HH:mm:ss.SSSz")),
+ Row(expected))
+ }
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> "true") {
+ checkTimeZoneParsing(Timestamp.valueOf("2020-01-27 20:06:11.847"))
+ }
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> "false") {
+ checkTimeZoneParsing(null)
Review comment:
Strings parsed in different calendars may have difference of dozen days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582205521
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117861/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581854209
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22592/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546977
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581835977
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582286694
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582134406
**[Test build #117861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117861/testReport)** for PR 27441 at commit [`19a47c1`](https://github.com/apache/spark/commit/19a47c1cd08ce6e2e00e37abdc8fd6c4b4311b4e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546397
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117782/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582286694
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581836553
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22589/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582390528
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117909/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581552860
> I will update the SQL migration guide soon.
> pending migration guide updates
I am considering 2 options for updating the SQL migration guide:
1. Write a separate item regarding `SimpleDateFormat` and `DateTimeFomatter` in parsing timestamps/dates using a pattern
2. Or to modify existing items https://github.com/apache/spark/blame/cd5f03a3ba18ae455f93abc5e5d04f098fa8f046/docs/sql-migration-guide.md#L70-L72
The first option will clearly highlight the changes. @cloud-fan @dongjoon-hyun WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581357753
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117769/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581361263
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22532/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581360845
**[Test build #117770 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117770/testReport)** for PR 27441 at commit [`af38b87`](https://github.com/apache/spark/commit/af38b879dab37b5a1bad62ab528eae6fa008916e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546949
There are some problems with jenkins:
```
ERROR: [WS-CLEANUP] Cannot delete workspace: remote file operation failed: /home/jenkins/workspace/SparkPullRequestBuilder at hudson.remoting.Channel@35d245c0:amp-jenkins-worker-02: java.nio.file.FileSystemException: /home/jenkins/workspace/SparkPullRequestBuilder: Read-only file system
ERROR: Cannot delete workspace: remote file operation failed: /home/jenkins/workspace/SparkPullRequestBuilder at hudson.remoting.Channel@35d245c0:amp-jenkins-worker-02: java.nio.file.FileSystemException: /home/jenkins/workspace/SparkPullRequestBuilder: Read-only file system
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581459729
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117770/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581854198
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581358376
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22531/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582135008
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22623/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582135008
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22623/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581546385
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581835986
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117827/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581459714
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582132530
jenkins, retest this, please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581555007
TODO: Need to update all docs like https://github.com/apache/spark/blob/e5abbab0ed5d9d70522c1e19c53e95c631dd1565/python/pyspark/sql/readwriter.py#L224
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582286322
**[Test build #117909 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117909/testReport)** for PR 27441 at commit [`19a47c1`](https://github.com/apache/spark/commit/19a47c1cd08ce6e2e00e37abdc8fd6c4b4311b4e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r375121655
##########
File path: docs/sql-migration-guide.md
##########
@@ -67,9 +67,7 @@ license: |
- Since Spark 3.0, Proleptic Gregorian calendar is used in parsing, formatting, and converting dates and timestamps as well as in extracting sub-components like years, days and etc. Spark 3.0 uses Java 8 API classes from the java.time packages that based on ISO chronology (https://docs.oracle.com/javase/8/docs/api/java/time/chrono/IsoChronology.html). In Spark version 2.4 and earlier, those operations are performed by using the hybrid calendar (Julian + Gregorian, see https://docs.oracle.com/javase/7/docs/api/java/util/GregorianCalendar.html). The changes impact on the results for dates before October 15, 1582 (Gregorian) and affect on the following Spark 3.0 API:
- - CSV/JSON datasources use java.time API for parsing and generating CSV/JSON content. In Spark version 2.4 and earlier, java.text.SimpleDateFormat is used for the same purpose with fallbacks to the parsing mechanisms of Spark 2.0 and 1.x. For example, `2018-12-08 10:39:21.123` with the pattern `yyyy-MM-dd'T'HH:mm:ss.SSS` cannot be parsed since Spark 3.0 because the timestamp does not match to the pattern but it can be parsed by earlier Spark versions due to a fallback to `Timestamp.valueOf`. To parse the same timestamp since Spark 3.0, the pattern should be `yyyy-MM-dd HH:mm:ss.SSS`.
-
- - The `unix_timestamp`, `date_format`, `to_unix_timestamp`, `from_unixtime`, `to_date`, `to_timestamp` functions. New implementation supports pattern formats as described here https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html and performs strict checking of its input. For example, the `2015-07-22 10:00:00` timestamp cannot be parse if pattern is `yyyy-MM-dd` because the parser does not consume whole input. Another example is the `31/01/2015 00:00` input cannot be parsed by the `dd/MM/yyyy hh:mm` pattern because `hh` supposes hours in the range `1-12`.
+ - Parsing/formatting of timestamp/date strings. This effects on CSV/JSON datasources and on the `unix_timestamp`, `date_format`, `to_unix_timestamp`, `from_unixtime`, `to_date`, `to_timestamp` functions when patterns specified by users is used for parsing and formatting. Since Spark 3.0, the conversions are based on `java.time.format.DateTimeFormatter`, see https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html. New implementation performs strict checking of its input. For example, the `2015-07-22 10:00:00` timestamp cannot be parse if pattern is `yyyy-MM-dd` because the parser does not consume whole input. Another example is the `31/01/2015 00:00` input cannot be parsed by the `dd/MM/yyyy hh:mm` pattern because `hh` supposes hours in the range `1-12`. In Spark version 2.4 and earlier, `java.text.SimpleDateFormat` is used for timestamp/date string conversions, and the supported patterns are described in https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html. The old behavior can be restored by setting `spark.sql.legacy.timeParser.enabled` to `true`.
Review comment:
is this really related to the Proleptic Gregorian calendar switch? It looks to me that we just switch to a better pattern string implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581361263
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22532/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581358188
I will update the SQL migration guide soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582134999
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582390528
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117909/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582134999
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581358376
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22531/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r375131044
##########
File path: docs/sql-migration-guide.md
##########
@@ -67,9 +67,7 @@ license: |
- Since Spark 3.0, Proleptic Gregorian calendar is used in parsing, formatting, and converting dates and timestamps as well as in extracting sub-components like years, days and etc. Spark 3.0 uses Java 8 API classes from the java.time packages that based on ISO chronology (https://docs.oracle.com/javase/8/docs/api/java/time/chrono/IsoChronology.html). In Spark version 2.4 and earlier, those operations are performed by using the hybrid calendar (Julian + Gregorian, see https://docs.oracle.com/javase/7/docs/api/java/util/GregorianCalendar.html). The changes impact on the results for dates before October 15, 1582 (Gregorian) and affect on the following Spark 3.0 API:
- - CSV/JSON datasources use java.time API for parsing and generating CSV/JSON content. In Spark version 2.4 and earlier, java.text.SimpleDateFormat is used for the same purpose with fallbacks to the parsing mechanisms of Spark 2.0 and 1.x. For example, `2018-12-08 10:39:21.123` with the pattern `yyyy-MM-dd'T'HH:mm:ss.SSS` cannot be parsed since Spark 3.0 because the timestamp does not match to the pattern but it can be parsed by earlier Spark versions due to a fallback to `Timestamp.valueOf`. To parse the same timestamp since Spark 3.0, the pattern should be `yyyy-MM-dd HH:mm:ss.SSS`.
-
- - The `unix_timestamp`, `date_format`, `to_unix_timestamp`, `from_unixtime`, `to_date`, `to_timestamp` functions. New implementation supports pattern formats as described here https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html and performs strict checking of its input. For example, the `2015-07-22 10:00:00` timestamp cannot be parse if pattern is `yyyy-MM-dd` because the parser does not consume whole input. Another example is the `31/01/2015 00:00` input cannot be parsed by the `dd/MM/yyyy hh:mm` pattern because `hh` supposes hours in the range `1-12`.
+ - Parsing/formatting of timestamp/date strings. This effects on CSV/JSON datasources and on the `unix_timestamp`, `date_format`, `to_unix_timestamp`, `from_unixtime`, `to_date`, `to_timestamp` functions when patterns specified by users is used for parsing and formatting. Since Spark 3.0, the conversions are based on `java.time.format.DateTimeFormatter`, see https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html. New implementation performs strict checking of its input. For example, the `2015-07-22 10:00:00` timestamp cannot be parse if pattern is `yyyy-MM-dd` because the parser does not consume whole input. Another example is the `31/01/2015 00:00` input cannot be parsed by the `dd/MM/yyyy hh:mm` pattern because `hh` supposes hours in the range `1-12`. In Spark version 2.4 and earlier, `java.text.SimpleDateFormat` is used for timestamp/date string conversions, and the supported patterns are described in https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html. The old behavior can be restored by setting `spark.sql.legacy.timeParser.enabled` to `true`.
Review comment:
Yes, it is related because `SimpleDateFormat` and `DateTimeFormatter` use different calendars underneath. Slightly different patterns are just a consequence of switching.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581381291
This patch LGTM as it is (pending migration guide updates). Another concern is: since we use the new formatter by default, shall we try the old formatter if the new formatter doesn't work? Then users don't need to enable the legacy config for some cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582300990
LGTM except one question.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r374231390
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -789,4 +789,12 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
Row(Timestamp.valueOf("2015-07-24 07:00:00")),
Row(Timestamp.valueOf("2015-07-24 22:00:00"))))
}
+
+ test("SPARK-30668: use legacy timestamp parser in to_timestamp") {
+ withSQLConf(SQLConf.LEGACY_TIME_PARSER_ENABLED.key -> "true") {
+ val df = Seq("2020-01-27T20:06:11.847-0800").toDF("ts")
+ checkAnswer(df.select(to_timestamp(col("ts"), "yyyy-MM-dd'T'HH:mm:ss.SSSz")),
+ Row(Timestamp.valueOf("2020-01-27 20:06:11.847")))
+ }
Review comment:
Shall we check `NULL` case together when `SQLConf.LEGACY_TIME_PARSER_ENABLED.key=false`? That will detect when we fix the regression issue in the default configuration (`SQLConf.LEGACY_TIME_PARSER_ENABLED.key=true`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582390520
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581361258
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581854209
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22592/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27441: [SPARK-30668][SQL]
Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-582349683
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#discussion_r374229889
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DateFunctionsSuite.scala
##########
@@ -789,4 +789,12 @@ class DateFunctionsSuite extends QueryTest with SharedSparkSession {
Row(Timestamp.valueOf("2015-07-24 07:00:00")),
Row(Timestamp.valueOf("2015-07-24 22:00:00"))))
}
+
+ test("SPARK-30668: use legacy timestamp parser in to_timestamp") {
Review comment:
Thank you for adding this test coverage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581459714
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk edited a comment on issue #27441:
[SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing
timestamps/dates strings
Posted by GitBox <gi...@apache.org>.
MaxGekk edited a comment on issue #27441: [SPARK-30668][SQL] Support `SimpleDateFormat` patterns in parsing timestamps/dates strings
URL: https://github.com/apache/spark/pull/27441#issuecomment-581522914
> since we use the new formatter by default, shall we try the old formatter if the new formatter doesn't work? Then users don't need to enable the legacy config for some cases.
This may have some downsides like some values in a column could be parsed by old parser which is based on combined calendar (Julian + Gregorian) but another values **in the same column** by new one which uses Proleptic Gregorian calendar. This could make user life harder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org