[jira] [Commented] (SPARK-40791) The semantics of `F` in `DateTimeFormatter` have changed

["Yang Jie (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-40791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619613#comment-17619613 ] 

Yang Jie commented on SPARK-40791:
----------------------------------

[~hyukjin.kwon] 

I found that the difference was caused by a Java bug fix. Before Java 19, `F` represented `ALIGNED_DAY_OF_WEEK_IN_MONTH`, which is different from `java.text.SimpleDateFormatter`'s behavior. Therefore, it was changed to `ALIGNED_WEEK_OF_MONTH` in Java 19, but only changed in Java 19.

Although only the parse method is mentioned here, the behavior of the format method has also been changed.

 
 * https://bugs.openjdk.org/browse/JDK-8282081
 * [https://github.com/openjdk/jdk/commit/733c7907b0059cc734fd1aa5b8d31f9c3e2e3079]

 

> The semantics of `F` in `DateTimeFormatter` have changed
> --------------------------------------------------------
>
>                 Key: SPARK-40791
>                 URL: https://issues.apache.org/jira/browse/SPARK-40791
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: Yang Jie
>            Priority: Major
>
> {code:java}
> val createSql =
>   """
>     |create temporary view v as select col from values
>     | (timestamp '1582-06-01 11:33:33.123UTC+080000'),
>     | (timestamp '1970-01-01 00:00:00.000Europe/Paris'),
>     | (timestamp '1970-12-31 23:59:59.999Asia/Srednekolymsk'),
>     | (timestamp '1996-04-01 00:33:33.123Australia/Darwin'),
>     | (timestamp '2018-11-17 13:33:33.123Z'),
>     | (timestamp '2020-01-01 01:33:33.123Asia/Shanghai'),
>     | (timestamp '2100-01-01 01:33:33.123America/Los_Angeles') t(col)
>     | """.stripMargin
> sql(createSql)
> withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> false.toString) {
>   val rows = sql("select col, date_format(col, 'F') from v").collect()
>   // scalastyle:off
>   rows.foreach(println)
> } {code}
>  
> Before Java 19, the result is 
>  
> {code:java}
> [1582-05-31 19:40:35.123,3]
> [1969-12-31 15:00:00.0,3]
> [1970-12-31 04:59:59.999,3]
> [1996-03-31 07:03:33.123,3]
> [2018-11-17 05:33:33.123,3]
> [2019-12-31 09:33:33.123,3]
> [2100-01-01 01:33:33.123,1] {code}
> Java 19
>  
> {code:java}
> [1582-05-31 19:40:35.123,5]
> [1969-12-31 15:00:00.0,5]
> [1970-12-31 04:59:59.999,5]
> [1996-03-31 07:03:33.123,5]
> [2018-11-17 05:33:33.123,3]
> [2019-12-31 09:33:33.123,5]
> [2100-01-01 01:33:33.123,1] {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org