You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2020/06/03 06:54:00 UTC
[jira] [Resolved] (SPARK-31892) Disable week-based date filed for
parsing
[ https://issues.apache.org/jira/browse/SPARK-31892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-31892.
---------------------------------
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 28706
[https://github.com/apache/spark/pull/28706]
> Disable week-based date filed for parsing
> -----------------------------------------
>
> Key: SPARK-31892
> URL: https://issues.apache.org/jira/browse/SPARK-31892
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 3.0.0, 3.1.0
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Major
> Fix For: 3.0.0
>
>
> It's an un-fixable behavior change to fill the gap between SimpleDateFormat and DateTimeFormater and backward-compatibility for different JDKs.
> A lot of effort has been made to prove this at https://github.com/apache/spark/pull/28674
> The existing behavior itself in 2.4 is confusing, e.g.
> {code:sql}
> spark-sql> select to_timestamp('1', 'w');
> 1969-12-28 00:00:00
> spark-sql> select to_timestamp('1', 'u');
> 1970-01-05 00:00:00
> {code}
> the 'u' here seems not to go to the Monday of the first week in week-based form or the first day of the year in non-week-based form but go to the Monday of the second week in week-based form.
> {code:sql}
> spark-sql> select to_timestamp('2020 2020', 'YYYY yyyy');
> 2020-01-01 00:00:00
> spark-sql> select to_timestamp('2020 2020', 'yyyy YYYY');
> 2019-12-29 00:00:00
> spark-sql> select to_timestamp('2020 2020 1', 'YYYY yyyy w');
> NULL
> spark-sql> select to_timestamp('2020 2020 1', 'yyyy YYYY w');
> 2019-12-29 00:00:00
> {code}
> I think we don't need to introduce all the weird behavior from Java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org