You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/06/01 08:45:48 UTC
[GitHub] [spark] yaooqinn opened a new pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
yaooqinn opened a new pull request #28692:
URL: https://github.com/apache/spark/pull/28692
# What changes were proposed in this pull request?
This PR switches the default Locale from the `US` to `GB` to change the behavior of the first day of the week from Sunday-started to Monday-started as same as v2.4
### Why are the changes needed?
#### cases
{code:sql}
spark-sql> select to_timestamp('2020-1-1', 'YYYY-w-u');
2019-12-29 00:00:00
spark-sql> set spark.sql.legacy.timeParserPolicy=legacy;
spark.sql.legacy.timeParserPolicy legacy
spark-sql> select to_timestamp('2020-1-1', 'YYYY-w-u');
2019-12-30 00:00:00
{code}
#### reasons
These week-based fields need Locale to express their semantics, the first day of the week varies from country to country.
From the Java doc of WeekFields
```java
/**
* Gets the first day-of-week.
* <p>
* The first day-of-week varies by culture.
* For example, the US uses Sunday, while France and the ISO-8601 standard use Monday.
* This method returns the first day using the standard {@code DayOfWeek} enum.
*
* @return the first day-of-week, not null
*/
public DayOfWeek getFirstDayOfWeek() {
return firstDayOfWeek;
}
```
But for the SimpleDateFormat, the day-of-week is not localized
```
u Day number of week (1 = Monday, ..., 7 = Sunday) Number 1
```
Currently, the default locale we use is the US, so the result moved a day backward.
For other countries, please refer to [First Day of the Week in Different Countries|http://chartsbin.com/view/41671]
With this change, it restores the first day of week calculating for functions when using the default locale.
### Does this PR introduce _any_ user-facing change?
Yes, but the behavior change is used to restore the old one of v2.4
### How was this patch tested?
add unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638147190
> please also create a PR to explain what gets changed in the meaning of u in the migration guide.
@bart-samwel maybe you were confused by this. The meaning was already changed when we switch to the java 8 formatter API. In the legacy formatter, `u` means `Day number of week (1 = Monday, ..., 7 = Sunday)`, which is number, in the new formatter, we replace `u` with `e`, but `e` is `localized day-of-week`, which is number/text.
There is no such a pattern letter in the java 8 formatter API, which has the same meaning of the legacy `u`, the `e` is the closest. Parsing is OK as we ban all week-based pattern letters in parsing, see https://github.com/apache/spark/pull/28706
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637228101
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638360031
This breaks the Jenkins Java 11 job, because Java 11 changes the behavior of the GB locale (not sure about other locales, but US locale doesn't change) when formatting AM/PM by lower-casing it, see https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-hive-2.3-jdk-11/539/testReport/org.apache.spark.sql/SQLQueryTestSuite/datetime_legacy_sql/ . I can't believe it but that's the reality...
Note: users can still hit this Java 11 behavior change if they set locale to GB manually. It's true for Spark 2.4 as well, but Spark 2.4 doesn't support Java 11.
I'm reverting it from master/3.0, we need to find another way to fix the reported behavior change (or accept it).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637339913
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637720341
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637226731
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637227803
**[Test build #123413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123413/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637207235
**[Test build #123410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123410/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637528725
**[Test build #123433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123433/testReport)** for PR 28692 at commit [`f94c72b`](https://github.com/apache/spark/commit/f94c72be5edc5846b64bef8ed81a24a9109716c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bart-samwel commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
bart-samwel commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638117331
@yaooqinn @cloud-fan Does this locale setting change anything _else_ in the date parsing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637335864
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638391198
Thank you for recovering Java 11!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636709285
**[Test build #123369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123369/testReport)** for PR 28692 at commit [`ab0517d`](https://github.com/apache/spark/commit/ab0517dbb8935775bdb4026ba3c832b6bd534d1a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636709285
**[Test build #123369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123369/testReport)** for PR 28692 at commit [`ab0517d`](https://github.com/apache/spark/commit/ab0517dbb8935775bdb4026ba3c832b6bd534d1a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637528725
**[Test build #123433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123433/testReport)** for PR 28692 at commit [`f94c72b`](https://github.com/apache/spark/commit/f94c72be5edc5846b64bef8ed81a24a9109716c8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637335864
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638179022
Hi @bart-samwel, we are improving test coverage for the datetime patterns. Can you help review here https://github.com/apache/spark/pull/28718/files#diff-e342ba37e6a036a13fa8373de2ea9470R1047 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #28692:
URL: https://github.com/apache/spark/pull/28692#discussion_r433218901
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -117,7 +117,16 @@ class LegacySimpleDateFormatter(pattern: String, locale: Locale) extends LegacyD
object DateFormatter {
import LegacyDateFormats._
- val defaultLocale: Locale = Locale.US
+ /**
+ * This is change from Locale.US to GB, because:
+ * The first day-of-week varies by culture.
+ * For example, the US uses Sunday, while the United Kingdom and the ISO-8601 standard use Monday.
Review comment:
It seems that we don't need care about the other differences, e.g. currency symbol
FYI,
http://www.localeplanet.com/java/en-GB/index.html and http://www.localeplanet.com/java/en-US/index.html
the timeZone is not the same, but it's a separate field we don't get it from Locale right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637335200
**[Test build #123413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123413/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636852348
**[Test build #123369 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123369/testReport)** for PR 28692 at commit [`ab0517d`](https://github.com/apache/spark/commit/ab0517dbb8935775bdb4026ba3c832b6bd534d1a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638139165
As I mentioned in: https://github.com/apache/spark/pull/28692#issuecomment-637226576
There are only 2 localized pattern letter: `e: localized day-of-week` and `O: localized zone-offset`. @yaooqinn tested it locally, there is no difference between US and GB for these 2 pattern letters, except that GB treats Monday as the first day of week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638360031
This breaks the Jenkins Java 11 job, because Java 11 changes the behavior when formatting AM/PM by lower-casing it, see https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-hive-2.3-jdk-11/539/testReport/org.apache.spark.sql/SQLQueryTestSuite/datetime_legacy_sql/ . I can't believe it but that's the reality...
I'm reverting it from master/3.0, we need to find another way to fix the reported behavior change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637214555
**[Test build #123410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123410/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637339211
**[Test build #123421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123421/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637050742
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #28692:
URL: https://github.com/apache/spark/pull/28692
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637227803
**[Test build #123413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123413/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28692:
URL: https://github.com/apache/spark/pull/28692#discussion_r433852445
##########
File path: sql/core/src/test/resources/sql-tests/inputs/datetime.sql
##########
@@ -164,3 +164,6 @@ select from_csv('26/October/2015', 'date Date', map('dateFormat', 'dd/MMMMM/yyyy
select from_unixtime(1, 'yyyyyyyyyyy-MM-dd');
select date_format(timestamp '2018-11-17 13:33:33', 'yyyyyyyyyy-MM-dd HH:mm:ss');
select date_format(date '2018-11-17', 'yyyyyyyyyyy-MM-dd');
+
+-- SPARK-31879: the first day of week
+select to_timestamp('2020-01-01', 'YYYY-ww-uu');
Review comment:
can we use formatting in the test? so this doesn't conflict with https://github.com/apache/spark/pull/28706
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638360031
This breaks the Jenkins Java 11 job, because Java 11 changes the behavior of the GB locale (not sure about other locales, but US locale doesn't change) when formatting AM/PM by lower-casing it, see https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-hive-2.3-jdk-11/539/testReport/org.apache.spark.sql/SQLQueryTestSuite/datetime_legacy_sql/ . I can't believe it but that's the reality...
I'm reverting it from master/3.0, we need to find another way to fix the reported behavior change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638365287
Thanks for taking care of this. It looks like the culture in GB has been changed a lot during JDK8 to 11, lol...
Since we have banned 'u'(localized day-of-week) for parsing, maybe 'E' is better than 'e' for substitution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638360031
This breaks the Jenkins Java 11 job, because Java 11 changes the behavior of the GB locale (not sure about other locales, but US locale doesn't change) when formatting AM/PM by lower-casing it, see https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-hive-2.3-jdk-11/539/testReport/org.apache.spark.sql/SQLQueryTestSuite/datetime_legacy_sql/ . I can't believe it but that's the reality...
I'm reverting it from master/3.0, we need to find another way to fix the reported behavior change (or accept it).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638360031
This breaks the Jenkins Java 11 job, because Java 11 changes the behavior of the GB locale (not sure about other locales, but US locale doesn't change) when formatting AM/PM by lower-casing it, see https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7-hive-2.3-jdk-11/539/testReport/org.apache.spark.sql/SQLQueryTestSuite/datetime_legacy_sql/ . I can't believe it but that's the reality...
Note: users can still hit this Java 11 behavior change if they set locale to GB manually.
I'm reverting it from master/3.0, we need to find another way to fix the reported behavior change (or accept it).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636888699
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #28692:
URL: https://github.com/apache/spark/pull/28692#discussion_r433258038
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -117,7 +117,16 @@ class LegacySimpleDateFormatter(pattern: String, locale: Locale) extends LegacyD
object DateFormatter {
import LegacyDateFormats._
- val defaultLocale: Locale = Locale.US
+ /**
+ * This is change from Locale.US to GB, because:
+ * The first day-of-week varies by culture.
+ * For example, the US uses Sunday, while the United Kingdom and the ISO-8601 standard use Monday.
Review comment:
We have `O`, `OOOO` and `ZZZZ` that are localized, but there are decided by the `zoneID`(spark.sql.session.timeZone), not related to Locate here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636888059
**[Test build #123378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123378/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637486548
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637627749
Oops right I meant Locale, not TZ.
OK makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637486548
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638617420
> > Thanks for taking care of this. It looks like the culture in GB has been changed a lot during JDK8 to 11, lol...
> > Since we have banned 'u'(localized day-of-week) for parsing, maybe 'E' is better than 'e' for substitution.
>
> I'm thinking that since we can't properly support the old behavior of `u`, we're better off forbidding it, because otherwise we're doing a silent behavior change. The error can suggest using `e` or `E` instead, with a warning about the potential differences.
FYI, https://github.com/apache/spark/pull/28719#issuecomment-638585185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637529393
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638147190
> please also create a PR to explain what gets changed in the meaning of u in the migration guide.
@bart-samwel maybe you were confused by this. The meaning was already changed when we switch to the java 8 formatter API. In the legacy formatter, `u` means `Day number of week (1 = Monday, ..., 7 = Sunday)`, which is number, in the new formatter, we replace `u` with `e`, but `e` is `localized day-of-week`, which is number/text.
There is no such a pattern letter in the java 8 formatter API, which has the same meaning of the legacy `u`, the `e` is the closest.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637207448
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638147190
> please also create a PR to explain what gets changed in the meaning of u in the migration guide.
@bart-samwel maybe you were confused by this. The meaning already changed when we switch to the java 8 formatter API. In the legacy formatter, `u` means `Day number of week (1 = Monday, ..., 7 = Sunday)`, which is number, in the new formatter, we replace `u` with `e`, but `e` is `localized day-of-week`, which is number/text.
There is no such a pattern letter in the java 8 formatter API, which has the same meaning of the legacy `u`, the `e` is the closest.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637538510
Just for my info, why was Monday the first day of week in 2.4 then? we didn't use TZ to determine it?
Also is that the right answer from a SQL standard or Hive perspective?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637228101
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636992427
Wow, really? Monday is considered the first day of week in the US locale?
I'm a little uneasy about defaulting to GB, as it might have other implications.
What does the root locale do here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637206698
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637975864
@yaooqinn please also create a PR to explain what gets changed in the meaning of `u` in the migration guide. I'm merging this PR to master/3.0 first, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637207235
**[Test build #123410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123410/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637529393
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636853352
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637207448
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637720341
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637336891
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637608841
> why was Monday the first day of week in 2.4 then?
In Spark version 2.4 and earlier, datetime parsing and formatting are performed by the old Java 7 `SimpleDateFormat` API, the ` Monday the first day of week ` thing is decided by SimpleDateFormat. Since Spark 3.0, we switch to the new Java 8 `DateTimeFormatter` to use the Proleptic Gregorian calendar, which is required by the ISO and SQL standards.
> we didn't use TZ to determine it?
The `first day of week` is a date field, not a time field, TZ is short for time zone actually.
> Also is that the right answer from a SQL standard
The `first day of week` is not directly decided by ANSI SQL Framework but from its normative reference ISO standard. It is Monday start in ISO. The day of week`DateTimeFormatter` become localized and by default we use `Locale.US` where a week starts from Sunday. We have to restore this for backward compatibility. There may more solutions options provided at SPARK-31879
> or Hive perspective?
I think the backward compatibility between 3.0 and 2.4 shall go first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637339913
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637339211
**[Test build #123421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123421/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636888059
**[Test build #123378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123378/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28692:
URL: https://github.com/apache/spark/pull/28692#discussion_r433253854
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -117,7 +117,16 @@ class LegacySimpleDateFormatter(pattern: String, locale: Locale) extends LegacyD
object DateFormatter {
import LegacyDateFormats._
- val defaultLocale: Locale = Locale.US
+ /**
+ * This is change from Locale.US to GB, because:
Review comment:
Let's make the doc shorter
```
Before Spark 3.0, the first day-of-week is always Monday. Since Spark 3.0, it depends on the locale.
We pick GB as the default locale instead of US, to be compatible with Spark 2.x, as US locale uses
Sunday as the first day-of-week. See SPARK-31879.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637214619
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123410/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637719261
**[Test build #123433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123433/testReport)** for PR 28692 at commit [`f94c72b`](https://github.com/apache/spark/commit/f94c72be5edc5846b64bef8ed81a24a9109716c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637219105
Locale.ROOT was my first choice but it didn't work. However, even it works, I don't think it's a good idea for a distributed system.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636853352
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636710088
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637337298
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637226576
It's a bit unfortunate that ROOT locale also uses Sunday as the first day. I checked the pattern string doc: https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html , there are only 2 localized pattern letter: `e: localized day-of-week` and `O: localized zone-offset`. US and DB have no difference for these two.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637214611
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28692:
URL: https://github.com/apache/spark/pull/28692#discussion_r433205399
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -117,7 +117,16 @@ class LegacySimpleDateFormatter(pattern: String, locale: Locale) extends LegacyD
object DateFormatter {
import LegacyDateFormats._
- val defaultLocale: Locale = Locale.US
+ /**
+ * This is change from Locale.US to GB, because:
+ * The first day-of-week varies by culture.
+ * For example, the US uses Sunday, while the United Kingdom and the ISO-8601 standard use Monday.
Review comment:
is this the only difference between US and en-GB?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636710088
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637335881
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123413/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637216537
good idea using ROOT locale! @yaooqinn can you try it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637219782
```java
+ val defaultLocale: Locale = Locale.ROOT
def defaultPattern(): String = s"${DateFormatter.defaultPattern} HH:mm:ss"
diff --git a/sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out b/sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out
index a9a3bccadc..94fcc3b4ad 100644
--- a/sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/ansi/datetime.sql.out
@@ -1032,4 +1032,4 @@ select to_timestamp('2020-01-01', 'YYYY-ww-uu')
-- !query schema
struct<to_timestamp(2020-01-01, YYYY-ww-uu):timestamp>
-- !query output
-2019-12-30 00:00:00
+2019-12-29 00:00:00
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637214611
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637049925
**[Test build #123378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123378/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637050742
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637050750
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/123378/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28692:
URL: https://github.com/apache/spark/pull/28692#discussion_r433250246
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateFormatter.scala
##########
@@ -117,7 +117,16 @@ class LegacySimpleDateFormatter(pattern: String, locale: Locale) extends LegacyD
object DateFormatter {
import LegacyDateFormats._
- val defaultLocale: Locale = Locale.US
+ /**
+ * This is change from Locale.US to GB, because:
+ * The first day-of-week varies by culture.
+ * For example, the US uses Sunday, while the United Kingdom and the ISO-8601 standard use Monday.
Review comment:
is there any localized timezone related pattern letter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-636888699
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bart-samwel commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
bart-samwel commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-638422820
> Thanks for taking care of this. It looks like the culture in GB has been changed a lot during JDK8 to 11, lol...
>
> Since we have banned 'u'(localized day-of-week) for parsing, maybe 'E' is better than 'e' for substitution.
I'm thinking that since we can't properly support the old behavior of `u`, we're better off forbidding it, because otherwise we're doing a silent behavior change. The error can suggest using `e` or `E` instead, with a warning about the potential differences.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28692: [SPARK-31879][SQL] Using GB as default Locale for datetime formatters
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #28692:
URL: https://github.com/apache/spark/pull/28692#issuecomment-637485490
**[Test build #123421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/123421/testReport)** for PR 28692 at commit [`3339274`](https://github.com/apache/spark/commit/33392744c4c9f2ffebe8e05411908cea76de17cc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org