You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/06 08:08:16 UTC
[GitHub] [spark] xuanyuanking opened a new pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
xuanyuanking opened a new pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830
### What changes were proposed in this pull request?
In Spark version 2.4 and earlier, datetime parsing, formatting and conversion are performed by using the hybrid calendar (Julian + Gregorian).
Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes (the java.time packages that are based on ISO chronology ). The switching job is completed in SPARK-26651.
But after the switching, there are some patterns not compatible between Java 8 and Java 7, Spark needs its own definition on the patterns rather than depends on Java API.
In this PR, we achieve this by writing the document and shadow the incompatible letters. See more details in [SPARK-31030](https://issues.apache.org/jira/browse/SPARK-31030)
### Why are the changes needed?
For backward compatibility.
### Does this PR introduce any user-facing change?
No.
After we define our own datetime parsing and formatting patterns, it's same to old Spark version.
### How was this patch tested?
Existing and new added UT.
Locally document test:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595653274
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390157654
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
Review comment:
Thanks, done in e846fbb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596537551
**[Test build #119558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119558/testReport)** for PR 27830 at commit [`82aa515`](https://github.com/apache/spark/commit/82aa515416ffdfca3b5c4565af7dfcab56b04155).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595955150
**[Test build #119477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119477/testReport)** for PR 27830 at commit [`b3b5ee4`](https://github.com/apache/spark/commit/b3b5ee4aef23dc25e03bff2f2dc4711cde725448).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596410329
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24290/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596525648
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119557/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389007936
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,35 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in SPARK-31030.
+ *
+ * @param pattern - the input pattern.
Review comment:
input and output? and nit: `@param pattern The input pattern.`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390299797
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,45 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using DateTimeFormatter classes.
+ * However, the breaking changes between SimpleDateFormat and DateTimeFormatter pattern string
+ * will also breaks the backward compatibility of Spark 2.4 and earlier when parsing datetime.
+ * This function converts all incompatible pattern for the new parser in Spark 3.0. See more
+ * details in SPARK-31030.
+ *
+ * @param pattern The input pattern.
+ * @return The pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ val eraDesignatorContained = pattern.split("'").zipWithIndex.exists {
+ case (patternPart, index) =>
+ // Text can be quoted using single quotes, we only check the non-quote parts.
+ index % 2 == 0 && patternPart.contains("G")
+ }
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ // The meaning of 'u' was day number of week in Java 7, it was changed to year in Java 8.
+ // Substitute 'u' to 'e' and use Java 8 parser to parse the string. If parsable, return
+ // the result; otherwise, fall back to 'u', and then use the legacy Java 7 parser to
+ // parse. When it is successfully parsed, throw an exception and ask users to change
+ // the pattern strings or turn on the legacy mode; otherwise, return NULL as what Spark
+ // 2.4 does.
+ val res = patternPart.replace("u", "e")
+ // In Java 8 API, 'u' supports negative years. We substitute 'y' to 'u' here for keeping
Review comment:
Thanks, done in 5382508.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595652769
**[Test build #119451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119451/testReport)** for PR 27830 at commit [`462c63c`](https://github.com/apache/spark/commit/462c63c30330325a4c2a828794ffc6e4d7b682c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595955345
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119477/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595712911
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596402915
**[Test build #119557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119557/testReport)** for PR 27830 at commit [`621a00e`](https://github.com/apache/spark/commit/621a00e56afbafdfe8dfa32ad728e6a326e98e5e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596403366
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597607495
Thanks for the review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389593943
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>H</b> </td>
+ <td> hour-of-day (0-23) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>m</b> </td>
+ <td> minute-of-hour </td>
+ <td> number </td>
+ <td> 30 </td>
+</tr>
+<tr>
+ <td> <b>s</b> </td>
+ <td> second-of-minute </td>
+ <td> number </td>
+ <td> 55 </td>
+</tr>
+<tr>
+ <td> <b>S</b> </td>
+ <td> fraction-of-second </td>
+ <td> fraction </td>
+ <td> 978 </td>
+</tr>
+<tr>
+ <td> <b>z</b> </td>
+ <td> time-zone name </td>
+ <td> zone-name </td>
+ <td> Pacific Standard Time; PST </td>
+</tr>
+<tr>
+ <td> <b>O</b> </td>
+ <td> localized zone-offset </td>
+ <td> offset-O </td>
+ <td> GMT+8; GMT+08:00; UTC-08:00; </td>
+</tr>
+<tr>
+ <td> <b>X</b> </td>
+ <td> zone-offset 'Z' for zero </td>
+ <td> offset-X </td>
+ <td> Z; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>x</b> </td>
+ <td> zone-offset </td>
+ <td> offset-x </td>
+ <td> +0000; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>Z</b> </td>
+ <td> zone-offset </td>
+ <td> offset-Z </td>
+ <td> +0000; -0800; -08:00; </td>
+</tr>
+<tr>
+ <td> <b>'</b> </td>
+ <td> escape for text </td>
+ <td> delimiter </td>
+ <td></td>
+</tr>
+<tr>
+ <td> <b>''</b> </td>
+ <td> single quote </td>
+ <td> literal </td>
+ <td> ' </td>
+</tr>
+</table>
+
+The count of pattern letters determines the format.
+
+- Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. Exactly 4 pattern letters will use the full form.
+
+- Number: If the count of letters is one, then the value is output using the minimum number of digits and without padding. Otherwise, the count of digits is used as the width of the output field, with the value zero-padded as necessary. The following pattern letters have constraints on the count of letters. Only one letter 'F' can be specified. Up to two letters of 'd', 'H', 'h', 'K', 'k', 'm', and 's' can be specified. Up to three letters of 'D' can be specified.
+
+- Number/Text: If the count of pattern letters is 3 or greater, use the Text rules above. Otherwise use the Number rules above.
+
+- Fraction: Outputs the micro-of-second field as a fraction-of-second. The micro-of-second value has six digits, thus the count of pattern letters is from 1 to 6. If it is less than 6, then the micro-of-second value is truncated, with only the most significant digits being output.
+
+- Year: The count of letters determines the minimum field width below which padding is used. If the count of letters is two, then a reduced two digit form is used. For printing, this outputs the rightmost two digits. For parsing, this will parse using the base value of 2000, resulting in a year within the range 2000 to 2099 inclusive. If the count of letters is less than four (but not two), then the sign is only output for negative years. Otherwise, the sign is output if the pad width is exceeded.
Review comment:
> Otherwise, the sign is output if the pad width is exceeded.
This is not true when G is present, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389590856
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,44 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
Review comment:
It's a bit confusing to say java 7 & 8 as the old APIs are also available in java 8.
How about `SimpleDateFormat` and `DateTimeFormatter`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597079500
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390149628
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>H</b> </td>
+ <td> hour-of-day (0-23) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>m</b> </td>
+ <td> minute-of-hour </td>
+ <td> number </td>
+ <td> 30 </td>
+</tr>
+<tr>
+ <td> <b>s</b> </td>
+ <td> second-of-minute </td>
+ <td> number </td>
+ <td> 55 </td>
+</tr>
+<tr>
+ <td> <b>S</b> </td>
+ <td> fraction-of-second </td>
+ <td> fraction </td>
+ <td> 978 </td>
+</tr>
+<tr>
+ <td> <b>z</b> </td>
+ <td> time-zone name </td>
+ <td> zone-name </td>
+ <td> Pacific Standard Time; PST </td>
+</tr>
+<tr>
+ <td> <b>O</b> </td>
+ <td> localized zone-offset </td>
+ <td> offset-O </td>
+ <td> GMT+8; GMT+08:00; UTC-08:00; </td>
+</tr>
+<tr>
+ <td> <b>X</b> </td>
+ <td> zone-offset 'Z' for zero </td>
+ <td> offset-X </td>
+ <td> Z; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>x</b> </td>
+ <td> zone-offset </td>
+ <td> offset-x </td>
+ <td> +0000; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>Z</b> </td>
+ <td> zone-offset </td>
+ <td> offset-Z </td>
+ <td> +0000; -0800; -08:00; </td>
+</tr>
+<tr>
+ <td> <b>'</b> </td>
+ <td> escape for text </td>
+ <td> delimiter </td>
+ <td></td>
+</tr>
+<tr>
+ <td> <b>''</b> </td>
+ <td> single quote </td>
+ <td> literal </td>
+ <td> ' </td>
+</tr>
+</table>
+
+The count of pattern letters determines the format.
+
+- Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. Exactly 4 pattern letters will use the full form.
Review comment:
We'll get IllegalArgumentException.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595955342
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389590981
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,44 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in SPARK-31030.
+ *
+ * @param pattern The input pattern.
+ * @return The pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ val eraDesignatorContained = pattern.split("'").zipWithIndex.exists {
+ case (patternPart, index) =>
+ // Text can be quoted using single quotes, we only check the non-quote parts.
+ index % 2 == 0 && patternPart.contains("G")
+ }
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ // The meaning of 'u' was day number of week in Java 7, it changed to year in Java 8.
Review comment:
`it changed` -> `it was changed`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595893960
**[Test build #119477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119477/testReport)** for PR 27830 at commit [`b3b5ee4`](https://github.com/apache/spark/commit/b3b5ee4aef23dc25e03bff2f2dc4711cde725448).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595712920
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119451/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597227162
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390228108
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,45 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
Review comment:
In Spark 3.0, we switch to the Proleptic Gregorian calendar and use `DateTimeFormatter` for parsing/formatting datetime values. The pattern string is incompatible with the one defined by `SimpleDateFormat` in Spark 2.4 and earlier. This function ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595653283
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24186/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-598242853
so I only skimmed this but I ran into the config: val LEGACY_TIME_PARSER_ENABLED = buildConf("spark.sql.legacy.timeParser.enabled") in SQLConf.
I assume that can be removed with this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596964992
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24342/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596409702
**[Test build #119558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119558/testReport)** for PR 27830 at commit [`82aa515`](https://github.com/apache/spark/commit/82aa515416ffdfca3b5c4565af7dfcab56b04155).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597078813
**[Test build #119620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119620/testReport)** for PR 27830 at commit [`5382508`](https://github.com/apache/spark/commit/53825089b0e553fe912bd7bd16fc453cc94c36c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596964992
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24342/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596964987
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389060387
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,35 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in SPARK-31030.
+ *
+ * @param pattern - the input pattern.
+ * @return the pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ patternPart
+ // The meaning of 'u' was day number of week in Java 7, it changed to year in Java 8.
+ // Substitute 'u' to 'e' and use Java 8 parser to parse the string. If parsable, return
+ // the result; otherwise, fall back to 'u', and then use the legacy Java 7 parser to
+ // parse. When it is successfully parsed, throw an exception and ask users to change
+ // the pattern strings or turn on the legacy mode; otherwise, return NULL as what Spark
+ // 2.4 does.
+ .replace("u", "e")
Review comment:
Actually the quoted text has been considered, let me add comments to emphasize.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597086735
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119613/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595955345
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119477/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596967720
**[Test build #119613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119613/testReport)** for PR 27830 at commit [`e846fbb`](https://github.com/apache/spark/commit/e846fbb5950e302897bd73a0e7eb2eb054c8194e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389006226
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the blow letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>H</b> </td>
+ <td> hour-of-day (0-23) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>m</b> </td>
+ <td> minute-of-hour </td>
+ <td> number </td>
+ <td> 30 </td>
+</tr>
+<tr>
+ <td> <b>s</b> </td>
+ <td> second-of-minute </td>
+ <td> number </td>
+ <td> 55 </td>
+</tr>
+<tr>
+ <td> <b>S</b> </td>
+ <td> fraction-of-second </td>
+ <td> fraction </td>
+ <td> 978 </td>
+</tr>
+<tr>
+ <td> <b>z</b> </td>
+ <td> time-zone name </td>
+ <td> zone-name </td>
+ <td> Pacific Standard Time; PST </td>
+</tr>
+<tr>
+ <td> <b>O</b> </td>
+ <td> localized zone-offset </td>
+ <td> offset-O </td>
+ <td> GMT+8; GMT+08:00; UTC-08:00; </td>
+</tr>
+<tr>
+ <td> <b>X</b> </td>
+ <td> zone-offset 'Z' for zero </td>
+ <td> offset-X </td>
+ <td> Z; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>x</b> </td>
+ <td> zone-offset </td>
+ <td> offset-x </td>
+ <td> +0000; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>Z</b> </td>
+ <td> zone-offset </td>
+ <td> offset-Z </td>
+ <td> +0000; -0800; -08:00; </td>
+</tr>
+<tr>
+ <td> <b>'</b> </td>
+ <td> escape for text </td>
+ <td> delimiter </td>
+ <td></td>
+</tr>
+<tr>
+ <td> <b>''</b> </td>
+ <td> single quote </td>
+ <td> literal </td>
+ <td> ' </td>
+</tr>
+</table>
+
+The count of pattern letters determines the format.
+
+- Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. Exactly 4 pattern letters will use the full form.
+
+- Number: If the count of letters is one, then the value is output using the minimum number of digits and without padding. Otherwise, the count of digits is used as the width of the output field, with the value zero-padded as necessary. The following pattern letters have constraints on the count of letters. Only one letter 'F' can be specified. Up to two letters of 'd', 'H', 'h', 'K', 'k', 'm', and 's' can be specified. Up to three letters of 'D' can be specified.
+
+- Number/Text: If the count of pattern letters is 3 or greater, use the Text rules above. Otherwise use the Number rules above.
+
+- Fraction: Outputs the nano-of-second field as a fraction-of-second. The nano-of-second value has nine digits, thus the count of pattern letters is from 1 to 9. If it is less than 9, then the nano-of-second value is truncated, with only the most significant digits being output.
Review comment:
Currently, Spark doesn't support fraction in nanosecond precision. It can mislead users.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595894381
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24207/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597226156
**[Test build #119620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119620/testReport)** for PR 27830 at commit [`5382508`](https://github.com/apache/spark/commit/53825089b0e553fe912bd7bd16fc453cc94c36c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596421731
cc @MaxGekk @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595653274
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595653283
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24186/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597227176
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119620/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595893960
**[Test build #119477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119477/testReport)** for PR 27830 at commit [`b3b5ee4`](https://github.com/apache/spark/commit/b3b5ee4aef23dc25e03bff2f2dc4711cde725448).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596403375
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24289/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390157715
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,44 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in SPARK-31030.
+ *
+ * @param pattern The input pattern.
+ * @return The pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ val eraDesignatorContained = pattern.split("'").zipWithIndex.exists {
+ case (patternPart, index) =>
+ // Text can be quoted using single quotes, we only check the non-quote parts.
+ index % 2 == 0 && patternPart.contains("G")
+ }
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ // The meaning of 'u' was day number of week in Java 7, it changed to year in Java 8.
+ // Substitute 'u' to 'e' and use Java 8 parser to parse the string. If parsable, return
+ // the result; otherwise, fall back to 'u', and then use the legacy Java 7 parser to
+ // parse. When it is successfully parsed, throw an exception and ask users to change
+ // the pattern strings or turn on the legacy mode; otherwise, return NULL as what Spark
+ // 2.4 does.
+ val res = patternPart.replace("u", "e")
+ // In Java 8 API, 'u' supports negative years. We substitute 'y' to 'u' here for keeping
+ // the support in Spark 3.0. If parse failed in Spark 3.0, fall back to 'y'. We only do
+ // this substitution when there is no era designator found in the pattern.
+ if (!eraDesignatorContained) {
+ res.replace("y", "u")
+ } else {
+ res
+ }
+ } else {
+ patternPart
Review comment:
Thanks, done in e846fbb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596525633
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389591877
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
Review comment:
`clock-hour-of-day`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389545058
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,35 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in SPARK-31030.
+ *
+ * @param pattern - the input pattern.
Review comment:
Thanks, done in 82aa515.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597086735
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119613/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-598305261
yea it has been removed in https://github.com/apache/spark/pull/27889
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596409702
**[Test build #119558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119558/testReport)** for PR 27830 at commit [`82aa515`](https://github.com/apache/spark/commit/82aa515416ffdfca3b5c4565af7dfcab56b04155).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595894381
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24207/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596410329
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24290/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389588564
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,44 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in SPARK-31030.
+ *
+ * @param pattern The input pattern.
+ * @return The pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ val eraDesignatorContained = pattern.split("'").zipWithIndex.exists {
+ case (patternPart, index) =>
+ // Text can be quoted using single quotes, we only check the non-quote parts.
+ index % 2 == 0 && patternPart.contains("G")
+ }
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ // The meaning of 'u' was day number of week in Java 7, it changed to year in Java 8.
+ // Substitute 'u' to 'e' and use Java 8 parser to parse the string. If parsable, return
+ // the result; otherwise, fall back to 'u', and then use the legacy Java 7 parser to
+ // parse. When it is successfully parsed, throw an exception and ask users to change
+ // the pattern strings or turn on the legacy mode; otherwise, return NULL as what Spark
+ // 2.4 does.
+ val res = patternPart.replace("u", "e")
+ // In Java 8 API, 'u' supports negative years. We substitute 'y' to 'u' here for keeping
+ // the support in Spark 3.0. If parse failed in Spark 3.0, fall back to 'y'. We only do
+ // this substitution when there is no era designator found in the pattern.
+ if (!eraDesignatorContained) {
+ res.replace("y", "u")
+ } else {
+ res
+ }
+ } else {
+ patternPart
Review comment:
nit: indentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389592332
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>H</b> </td>
+ <td> hour-of-day (0-23) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>m</b> </td>
+ <td> minute-of-hour </td>
+ <td> number </td>
+ <td> 30 </td>
+</tr>
+<tr>
+ <td> <b>s</b> </td>
+ <td> second-of-minute </td>
+ <td> number </td>
+ <td> 55 </td>
+</tr>
+<tr>
+ <td> <b>S</b> </td>
+ <td> fraction-of-second </td>
+ <td> fraction </td>
+ <td> 978 </td>
+</tr>
+<tr>
+ <td> <b>z</b> </td>
+ <td> time-zone name </td>
+ <td> zone-name </td>
+ <td> Pacific Standard Time; PST </td>
+</tr>
+<tr>
+ <td> <b>O</b> </td>
+ <td> localized zone-offset </td>
+ <td> offset-O </td>
+ <td> GMT+8; GMT+08:00; UTC-08:00; </td>
+</tr>
+<tr>
+ <td> <b>X</b> </td>
+ <td> zone-offset 'Z' for zero </td>
+ <td> offset-X </td>
+ <td> Z; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>x</b> </td>
+ <td> zone-offset </td>
+ <td> offset-x </td>
+ <td> +0000; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>Z</b> </td>
+ <td> zone-offset </td>
+ <td> offset-Z </td>
+ <td> +0000; -0800; -08:00; </td>
+</tr>
+<tr>
+ <td> <b>'</b> </td>
+ <td> escape for text </td>
+ <td> delimiter </td>
+ <td></td>
+</tr>
+<tr>
+ <td> <b>''</b> </td>
+ <td> single quote </td>
+ <td> literal </td>
+ <td> ' </td>
+</tr>
+</table>
+
+The count of pattern letters determines the format.
+
+- Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. Exactly 4 pattern letters will use the full form.
Review comment:
how about more than 5 letters?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596403366
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595712625
**[Test build #119451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119451/testReport)** for PR 27830 at commit [`462c63c`](https://github.com/apache/spark/commit/462c63c30330325a4c2a828794ffc6e4d7b682c8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597461388
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597227162
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597079510
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24349/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597085740
**[Test build #119613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119613/testReport)** for PR 27830 at commit [`e846fbb`](https://github.com/apache/spark/commit/e846fbb5950e302897bd73a0e7eb2eb054c8194e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597078813
**[Test build #119620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119620/testReport)** for PR 27830 at commit [`5382508`](https://github.com/apache/spark/commit/53825089b0e553fe912bd7bd16fc453cc94c36c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390157958
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,44 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in SPARK-31030.
+ *
+ * @param pattern The input pattern.
+ * @return The pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ val eraDesignatorContained = pattern.split("'").zipWithIndex.exists {
+ case (patternPart, index) =>
+ // Text can be quoted using single quotes, we only check the non-quote parts.
+ index % 2 == 0 && patternPart.contains("G")
+ }
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ // The meaning of 'u' was day number of week in Java 7, it changed to year in Java 8.
Review comment:
Thanks, fixed in e846fbb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596967720
**[Test build #119613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119613/testReport)** for PR 27830 at commit [`e846fbb`](https://github.com/apache/spark/commit/e846fbb5950e302897bd73a0e7eb2eb054c8194e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596538738
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119558/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390299314
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,45 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
Review comment:
Thanks, done in 5382508
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595894367
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596403375
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24289/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390157824
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,44 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
Review comment:
Thanks, done in e846fbb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390158239
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>H</b> </td>
+ <td> hour-of-day (0-23) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>m</b> </td>
+ <td> minute-of-hour </td>
+ <td> number </td>
+ <td> 30 </td>
+</tr>
+<tr>
+ <td> <b>s</b> </td>
+ <td> second-of-minute </td>
+ <td> number </td>
+ <td> 55 </td>
+</tr>
+<tr>
+ <td> <b>S</b> </td>
+ <td> fraction-of-second </td>
+ <td> fraction </td>
+ <td> 978 </td>
+</tr>
+<tr>
+ <td> <b>z</b> </td>
+ <td> time-zone name </td>
+ <td> zone-name </td>
+ <td> Pacific Standard Time; PST </td>
+</tr>
+<tr>
+ <td> <b>O</b> </td>
+ <td> localized zone-offset </td>
+ <td> offset-O </td>
+ <td> GMT+8; GMT+08:00; UTC-08:00; </td>
+</tr>
+<tr>
+ <td> <b>X</b> </td>
+ <td> zone-offset 'Z' for zero </td>
+ <td> offset-X </td>
+ <td> Z; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>x</b> </td>
+ <td> zone-offset </td>
+ <td> offset-x </td>
+ <td> +0000; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>Z</b> </td>
+ <td> zone-offset </td>
+ <td> offset-Z </td>
+ <td> +0000; -0800; -08:00; </td>
+</tr>
+<tr>
+ <td> <b>'</b> </td>
+ <td> escape for text </td>
+ <td> delimiter </td>
+ <td></td>
+</tr>
+<tr>
+ <td> <b>''</b> </td>
+ <td> single quote </td>
+ <td> literal </td>
+ <td> ' </td>
+</tr>
+</table>
+
+The count of pattern letters determines the format.
+
+- Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. Exactly 4 pattern letters will use the full form.
+
+- Number: If the count of letters is one, then the value is output using the minimum number of digits and without padding. Otherwise, the count of digits is used as the width of the output field, with the value zero-padded as necessary. The following pattern letters have constraints on the count of letters. Only one letter 'F' can be specified. Up to two letters of 'd', 'H', 'h', 'K', 'k', 'm', and 's' can be specified. Up to three letters of 'D' can be specified.
+
+- Number/Text: If the count of pattern letters is 3 or greater, use the Text rules above. Otherwise use the Number rules above.
+
+- Fraction: Outputs the micro-of-second field as a fraction-of-second. The micro-of-second value has six digits, thus the count of pattern letters is from 1 to 6. If it is less than 6, then the micro-of-second value is truncated, with only the most significant digits being output.
+
+- Year: The count of letters determines the minimum field width below which padding is used. If the count of letters is two, then a reduced two digit form is used. For printing, this outputs the rightmost two digits. For parsing, this will parse using the base value of 2000, resulting in a year within the range 2000 to 2099 inclusive. If the count of letters is less than four (but not two), then the sign is only output for negative years. Otherwise, the sign is output if the pad width is exceeded.
Review comment:
Right, emphasize in e846fbb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596538738
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119558/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390158121
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>H</b> </td>
+ <td> hour-of-day (0-23) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>m</b> </td>
+ <td> minute-of-hour </td>
+ <td> number </td>
+ <td> 30 </td>
+</tr>
+<tr>
+ <td> <b>s</b> </td>
+ <td> second-of-minute </td>
+ <td> number </td>
+ <td> 55 </td>
+</tr>
+<tr>
+ <td> <b>S</b> </td>
+ <td> fraction-of-second </td>
+ <td> fraction </td>
+ <td> 978 </td>
+</tr>
+<tr>
+ <td> <b>z</b> </td>
+ <td> time-zone name </td>
+ <td> zone-name </td>
+ <td> Pacific Standard Time; PST </td>
+</tr>
+<tr>
+ <td> <b>O</b> </td>
+ <td> localized zone-offset </td>
+ <td> offset-O </td>
+ <td> GMT+8; GMT+08:00; UTC-08:00; </td>
+</tr>
+<tr>
+ <td> <b>X</b> </td>
+ <td> zone-offset 'Z' for zero </td>
+ <td> offset-X </td>
+ <td> Z; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>x</b> </td>
+ <td> zone-offset </td>
+ <td> offset-x </td>
+ <td> +0000; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>Z</b> </td>
+ <td> zone-offset </td>
+ <td> offset-Z </td>
+ <td> +0000; -0800; -08:00; </td>
+</tr>
+<tr>
+ <td> <b>'</b> </td>
+ <td> escape for text </td>
+ <td> delimiter </td>
+ <td></td>
+</tr>
+<tr>
+ <td> <b>''</b> </td>
+ <td> single quote </td>
+ <td> literal </td>
+ <td> ' </td>
+</tr>
+</table>
+
+The count of pattern letters determines the format.
+
+- Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. Exactly 4 pattern letters will use the full form.
Review comment:
Sure, done in e846fbb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389078519
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the blow letters in date and timestamp parsing and formatting:
Review comment:
Thanks! The typo fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596402915
**[Test build #119557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119557/testReport)** for PR 27830 at commit [`621a00e`](https://github.com/apache/spark/commit/621a00e56afbafdfe8dfa32ad728e6a326e98e5e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390158063
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
Review comment:
Thanks, done in e846fbb.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596410319
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597086726
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596538729
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596538729
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389004634
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the blow letters in date and timestamp parsing and formatting:
Review comment:
`blow`? Maybe `below`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390152472
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>H</b> </td>
+ <td> hour-of-day (0-23) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>m</b> </td>
+ <td> minute-of-hour </td>
+ <td> number </td>
+ <td> 30 </td>
+</tr>
+<tr>
+ <td> <b>s</b> </td>
+ <td> second-of-minute </td>
+ <td> number </td>
+ <td> 55 </td>
+</tr>
+<tr>
+ <td> <b>S</b> </td>
+ <td> fraction-of-second </td>
+ <td> fraction </td>
+ <td> 978 </td>
+</tr>
+<tr>
+ <td> <b>z</b> </td>
+ <td> time-zone name </td>
+ <td> zone-name </td>
+ <td> Pacific Standard Time; PST </td>
+</tr>
+<tr>
+ <td> <b>O</b> </td>
+ <td> localized zone-offset </td>
+ <td> offset-O </td>
+ <td> GMT+8; GMT+08:00; UTC-08:00; </td>
+</tr>
+<tr>
+ <td> <b>X</b> </td>
+ <td> zone-offset 'Z' for zero </td>
+ <td> offset-X </td>
+ <td> Z; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>x</b> </td>
+ <td> zone-offset </td>
+ <td> offset-x </td>
+ <td> +0000; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>Z</b> </td>
+ <td> zone-offset </td>
+ <td> offset-Z </td>
+ <td> +0000; -0800; -08:00; </td>
+</tr>
+<tr>
+ <td> <b>'</b> </td>
+ <td> escape for text </td>
+ <td> delimiter </td>
+ <td></td>
+</tr>
+<tr>
+ <td> <b>''</b> </td>
+ <td> single quote </td>
+ <td> literal </td>
+ <td> ' </td>
+</tr>
+</table>
+
+The count of pattern letters determines the format.
+
+- Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. Exactly 4 pattern letters will use the full form.
Review comment:
Let's document it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597079500
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390228562
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,45 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using DateTimeFormatter classes.
+ * However, the breaking changes between SimpleDateFormat and DateTimeFormatter pattern string
+ * will also breaks the backward compatibility of Spark 2.4 and earlier when parsing datetime.
+ * This function converts all incompatible pattern for the new parser in Spark 3.0. See more
+ * details in SPARK-31030.
+ *
+ * @param pattern The input pattern.
+ * @return The pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ val eraDesignatorContained = pattern.split("'").zipWithIndex.exists {
+ case (patternPart, index) =>
+ // Text can be quoted using single quotes, we only check the non-quote parts.
+ index % 2 == 0 && patternPart.contains("G")
+ }
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ // The meaning of 'u' was day number of week in Java 7, it was changed to year in Java 8.
+ // Substitute 'u' to 'e' and use Java 8 parser to parse the string. If parsable, return
+ // the result; otherwise, fall back to 'u', and then use the legacy Java 7 parser to
+ // parse. When it is successfully parsed, throw an exception and ask users to change
+ // the pattern strings or turn on the legacy mode; otherwise, return NULL as what Spark
+ // 2.4 does.
+ val res = patternPart.replace("u", "e")
+ // In Java 8 API, 'u' supports negative years. We substitute 'y' to 'u' here for keeping
Review comment:
ditto
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390228521
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,45 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using DateTimeFormatter classes.
+ * However, the breaking changes between SimpleDateFormat and DateTimeFormatter pattern string
+ * will also breaks the backward compatibility of Spark 2.4 and earlier when parsing datetime.
+ * This function converts all incompatible pattern for the new parser in Spark 3.0. See more
+ * details in SPARK-31030.
+ *
+ * @param pattern The input pattern.
+ * @return The pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ val eraDesignatorContained = pattern.split("'").zipWithIndex.exists {
+ case (patternPart, index) =>
+ // Text can be quoted using single quotes, we only check the non-quote parts.
+ index % 2 == 0 && patternPart.contains("G")
+ }
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ // The meaning of 'u' was day number of week in Java 7, it was changed to year in Java 8.
Review comment:
Java 8 -> DateTimeFormatter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595955342
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389010389
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,35 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using Java 8 API classes. However, the
+ * breaking changes between Java 7 and Java 8 pattern string will also breaks the backward
+ * compatibility of Spark 2.4 and earlier when parsing datetime. This function converts all
+ * incompatible pattern for the new parser in Spark 3.0. See more details in SPARK-31030.
+ *
+ * @param pattern - the input pattern.
+ * @return the pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ patternPart
+ // The meaning of 'u' was day number of week in Java 7, it changed to year in Java 8.
+ // Substitute 'u' to 'e' and use Java 8 parser to parse the string. If parsable, return
+ // the result; otherwise, fall back to 'u', and then use the legacy Java 7 parser to
+ // parse. When it is successfully parsed, throw an exception and ask users to change
+ // the pattern strings or turn on the legacy mode; otherwise, return NULL as what Spark
+ // 2.4 does.
+ .replace("u", "e")
Review comment:
'u' can be escaped in the pattern like `'update time' uuuu-MM-dd`. Replacing every 'u' will lead to wrong pattern, and nothing matches to it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596525633
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596524703
**[Test build #119557 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119557/testReport)** for PR 27830 at commit [`621a00e`](https://github.com/apache/spark/commit/621a00e56afbafdfe8dfa32ad728e6a326e98e5e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r390299707
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
##########
@@ -906,4 +906,45 @@ object DateTimeUtils {
val days = period.getDays
new CalendarInterval(months, days, 0)
}
+
+ /**
+ * Since the Proleptic Gregorian calendar is de-facto calendar worldwide, as well as the chosen
+ * one in ANSI SQL standard, Spark 3.0 switches to it by using DateTimeFormatter classes.
+ * However, the breaking changes between SimpleDateFormat and DateTimeFormatter pattern string
+ * will also breaks the backward compatibility of Spark 2.4 and earlier when parsing datetime.
+ * This function converts all incompatible pattern for the new parser in Spark 3.0. See more
+ * details in SPARK-31030.
+ *
+ * @param pattern The input pattern.
+ * @return The pattern for new parser
+ */
+ def convertIncompatiblePattern(pattern: String): String = {
+ val eraDesignatorContained = pattern.split("'").zipWithIndex.exists {
+ case (patternPart, index) =>
+ // Text can be quoted using single quotes, we only check the non-quote parts.
+ index % 2 == 0 && patternPart.contains("G")
+ }
+ pattern.split("'").zipWithIndex.map {
+ case (patternPart, index) =>
+ if (index % 2 == 0) {
+ // The meaning of 'u' was day number of week in Java 7, it was changed to year in Java 8.
Review comment:
Thanks, also rephrase the whole comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389586615
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the below letters in date and timestamp parsing and formatting:
Review comment:
better to mention some places users need the pattern string, like some SQL functions and CSV/JSON options.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595894367
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595652769
**[Test build #119451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119451/testReport)** for PR 27830 at commit [`462c63c`](https://github.com/apache/spark/commit/462c63c30330325a4c2a828794ffc6e4d7b682c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595712920
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119451/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596964987
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596410319
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#discussion_r389523487
##########
File path: docs/sql-ref-datetime-pattern.md
##########
@@ -0,0 +1,212 @@
+---
+layout: global
+title: Datetime patterns
+displayTitle: Datetime Patterns for Formatting and Parsing
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+Spark uses the blow letters in date and timestamp parsing and formatting:
+<table class="table">
+<tr>
+ <th> <b>Symbol</b> </th>
+ <th> <b>Meaning</b> </th>
+ <th> <b>Presentation</b> </th>
+ <th> <b>Examples</b> </th>
+</tr>
+<tr>
+ <td> <b>G</b> </td>
+ <td> era </td>
+ <td> text </td>
+ <td> AD; Anno Domini; A </td>
+</tr>
+<tr>
+ <td> <b>y</b> </td>
+ <td> year </td>
+ <td> year </td>
+ <td> 2020; 20 </td>
+</tr>
+<tr>
+ <td> <b>D</b> </td>
+ <td> day-of-year </td>
+ <td> number </td>
+ <td> 189 </td>
+</tr>
+<tr>
+ <td> <b>M</b> </td>
+ <td> month-of-year </td>
+ <td> number/text </td>
+ <td> 7; 07; Jul; July; J </td>
+</tr>
+<tr>
+ <td> <b>d</b> </td>
+ <td> day-of-month </td>
+ <td> number </td>
+ <td> 28 </td>
+</tr>
+<tr>
+ <td> <b>Y</b> </td>
+ <td> week-based-year </td>
+ <td> year </td>
+ <td> 1996; 96 </td>
+</tr>
+<tr>
+ <td> <b>w</b> </td>
+ <td> week-of-week-based-year </td>
+ <td> number </td>
+ <td> 27 </td>
+</tr>
+<tr>
+ <td> <b>W</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 4 </td>
+</tr>
+<tr>
+ <td> <b>E</b> </td>
+ <td> day-of-week </td>
+ <td> text </td>
+ <td> Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>e</b> </td>
+ <td> localized day-of-week </td>
+ <td> number/text </td>
+ <td> 2; 02; Tue; Tuesday; T </td>
+</tr>
+<tr>
+ <td> <b>F</b> </td>
+ <td> week-of-month </td>
+ <td> number </td>
+ <td> 3 </td>
+</tr>
+<tr>
+ <td> <b>a</b> </td>
+ <td> am-pm-of-day </td>
+ <td> text </td>
+ <td> PM </td>
+</tr>
+<tr>
+ <td> <b>h</b> </td>
+ <td> clock-hour-of-am-pm (1-12) </td>
+ <td> number </td>
+ <td> 12 </td>
+</tr>
+<tr>
+ <td> <b>K</b> </td>
+ <td> hour-of-am-pm (0-11) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>k</b> </td>
+ <td> clock-hour-of-am-pm (1-24) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>H</b> </td>
+ <td> hour-of-day (0-23) </td>
+ <td> number </td>
+ <td> 0 </td>
+</tr>
+<tr>
+ <td> <b>m</b> </td>
+ <td> minute-of-hour </td>
+ <td> number </td>
+ <td> 30 </td>
+</tr>
+<tr>
+ <td> <b>s</b> </td>
+ <td> second-of-minute </td>
+ <td> number </td>
+ <td> 55 </td>
+</tr>
+<tr>
+ <td> <b>S</b> </td>
+ <td> fraction-of-second </td>
+ <td> fraction </td>
+ <td> 978 </td>
+</tr>
+<tr>
+ <td> <b>z</b> </td>
+ <td> time-zone name </td>
+ <td> zone-name </td>
+ <td> Pacific Standard Time; PST </td>
+</tr>
+<tr>
+ <td> <b>O</b> </td>
+ <td> localized zone-offset </td>
+ <td> offset-O </td>
+ <td> GMT+8; GMT+08:00; UTC-08:00; </td>
+</tr>
+<tr>
+ <td> <b>X</b> </td>
+ <td> zone-offset 'Z' for zero </td>
+ <td> offset-X </td>
+ <td> Z; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>x</b> </td>
+ <td> zone-offset </td>
+ <td> offset-x </td>
+ <td> +0000; -08; -0830; -08:30; -083015; -08:30:15; </td>
+</tr>
+<tr>
+ <td> <b>Z</b> </td>
+ <td> zone-offset </td>
+ <td> offset-Z </td>
+ <td> +0000; -0800; -08:00; </td>
+</tr>
+<tr>
+ <td> <b>'</b> </td>
+ <td> escape for text </td>
+ <td> delimiter </td>
+ <td></td>
+</tr>
+<tr>
+ <td> <b>''</b> </td>
+ <td> single quote </td>
+ <td> literal </td>
+ <td> ' </td>
+</tr>
+</table>
+
+The count of pattern letters determines the format.
+
+- Text: The text style is determined based on the number of pattern letters used. Less than 4 pattern letters will use the short form. Exactly 4 pattern letters will use the full form.
+
+- Number: If the count of letters is one, then the value is output using the minimum number of digits and without padding. Otherwise, the count of digits is used as the width of the output field, with the value zero-padded as necessary. The following pattern letters have constraints on the count of letters. Only one letter 'F' can be specified. Up to two letters of 'd', 'H', 'h', 'K', 'k', 'm', and 's' can be specified. Up to three letters of 'D' can be specified.
+
+- Number/Text: If the count of pattern letters is 3 or greater, use the Text rules above. Otherwise use the Number rules above.
+
+- Fraction: Outputs the nano-of-second field as a fraction-of-second. The nano-of-second value has nine digits, thus the count of pattern letters is from 1 to 9. If it is less than 9, then the nano-of-second value is truncated, with only the most significant digits being output.
Review comment:
Thanks for the comment, update the fraction section in 621a00e.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597086726
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-595712911
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597079510
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24349/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27830:
[SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-596525648
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119557/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27830: [SPARK-31030][SQL]
Backward Compatibility for Parsing and formatting Datetime
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27830: [SPARK-31030][SQL] Backward Compatibility for Parsing and formatting Datetime
URL: https://github.com/apache/spark/pull/27830#issuecomment-597227176
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119620/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org