You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Maxim Gekk (Jira)" <ji...@apache.org> on 2020/03/31 17:47:00 UTC

[jira] [Created] (SPARK-31318) Split Parquet/Avro configs for rebasing dates/timestamps in read and in write

Maxim Gekk created SPARK-31318:
----------------------------------

             Summary: Split Parquet/Avro configs for rebasing dates/timestamps in read and in write
                 Key: SPARK-31318
                 URL: https://issues.apache.org/jira/browse/SPARK-31318
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Maxim Gekk


Currently, Spark provides 2 SQL configs to control rebasing of dates/timestamps in Parquet and Avro datasource: 

spark.sql.legacy.parquet.rebaseDateTime.enabled

spark.sql.legacy.avro.rebaseDateTime.enabled

The configs control rebasing in read and in write. That's can be inconvenient for users who want to read files saved by Spark 2.4 and earlier versions, and save dates/timestamps without rebasing.

The ticket aims to split the configs, and introduce separate SQL configs for read and for write. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org