[jira] [Updated] (SPARK-31892) Disable week-based date filed for parsing

["Kent Yao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-31892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kent Yao updated SPARK-31892:
-----------------------------
    Summary: Disable week-based date filed for parsing  (was: Disable week-base date filed for parsing)

> Disable week-based date filed for parsing
> -----------------------------------------
>
>                 Key: SPARK-31892
>                 URL: https://issues.apache.org/jira/browse/SPARK-31892
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.0.0, 3.1.0
>            Reporter: Kent Yao
>            Priority: Major
>
> It's an un-fixable behavior change to fill the gap between SimpleDateFormat and DateTimeFormater and backward-compatibility for different JDKs.
> A lot of effort has been made to prove this at https://github.com/apache/spark/pull/28674
> The existing behavior itself in 2.4 is confusing, e.g. 
> {code:sql}
> spark-sql> select to_timestamp('1', 'w');
> 1969-12-28 00:00:00
> spark-sql> select to_timestamp('1', 'u');
> 1970-01-05 00:00:00
> {code}
> the 'u' here seems not to go to the Monday of the first week in week-based form or the first day of the year in non-week-based form but go to the Monday of the second week in week-based form.
> {code:sql}
> spark-sql> select to_timestamp('2020 2020', 'YYYY yyyy');
> 2020-01-01 00:00:00
> spark-sql> select to_timestamp('2020 2020', 'yyyy YYYY');
> 2019-12-29 00:00:00
> spark-sql> select to_timestamp('2020 2020 1', 'YYYY yyyy w');
> NULL
> spark-sql> select to_timestamp('2020 2020 1', 'yyyy YYYY w');
> 2019-12-29 00:00:00
> {code}
> I think we don't need to introduce all the weird behavior from Java.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org