[jira] [Created] (SPARK-39017) Change Java8 datetime support to configurable

["Weicheng Wang (Jira)" <ji...@apache.org>]

Weicheng Wang created SPARK-39017:
-------------------------------------

             Summary: Change Java8 datetime support to configurable 
                 Key: SPARK-39017
                 URL: https://issues.apache.org/jira/browse/SPARK-39017
             Project: Spark
          Issue Type: Brainstorming
          Components: SQL
    Affects Versions: 3.2.1
            Reporter: Weicheng Wang


*Background:*

In the Spark 3.1.0 there is an improvement introduced to enable Java 8 datetime API by default.

It stops user set this configuration, *spark.sql.datetime.java8API.enabled,* in command line or configuration file when using Spark SQL shell or Spark Thrift Server.

The only way to set it is in the SQL Session using SET command like:
{code:java}
spark-sql> SET spark.sql.datetime.java8API.enabled=false {code}
There are a few issues related to this improvement 
 * [https://github.com/apache/iceberg/issues/2530]
 * [https://github.com/delta-io/delta/issues/760]

There is a solution in 3.2.0 for it by having *LocalDateConverter* uses *DateConverter* and in the *DateConverter* it handles both *LocalDate* and *Date* types.

 

*Discussion:*

I think we should give back the ability to user to set this option in configuration file and command line. Both of the improvement defeats the reason why we have a java8API option configurable.

 

Please advise

Thanks

 

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org