You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sachit Murarka <co...@gmail.com> on 2022/09/20 13:26:52 UTC
Query regarding Proleptic Gregorian Calendar Spark3
Hi All,
I am getting below error , I read the document and understood that we need
to set 2 properties
spark.conf.set("spark.sql.parquet.int96RebaseModeInRead","CORRECTED")
spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite","CORRECTED")
is this the only way or is there any other way to handle this behaviour?
Caused by: org.apache.spark.SparkUpgradeException: You may get a different
result due to the upgrading of Spark 3.0:
writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z
into Parquet INT96
files can be dangerous, as the files may be read by Spark 2.x or legacy
versions of Hive
later, which uses a legacy hybrid calendar that is different from Spark
3.0+'s Proleptic
Gregorian calendar. See more details in SPARK-31404. You can set
spark.sql.parquet.int96RebaseModeInWrite to 'LEGACY' to
rebase the datetime values w.r.t. the calendar difference during writing,
to get maximum
interoperability. Or set spark.sql.parquet.int96RebaseModeInWrite to
'CORRECTED' to write the datetime values as it is,
if you are 100% sure that the written files will only be read by Spark 3.0+
or other
systems that use Proleptic Gregorian calendar.
Kind Regards,
Sachit Murarka
Re: Query regarding Proleptic Gregorian Calendar Spark3
Posted by Sachit Murarka <co...@gmail.com>.
Reposting once.
Kind Regards,
Sachit Murarka
On Tue, Sep 20, 2022 at 6:56 PM Sachit Murarka <co...@gmail.com>
wrote:
> Hi All,
>
> I am getting below error , I read the document and understood that we need
> to set 2 properties
> spark.conf.set("spark.sql.parquet.int96RebaseModeInRead","CORRECTED")
> spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite","CORRECTED")
>
> is this the only way or is there any other way to handle this behaviour?
>
> Caused by: org.apache.spark.SparkUpgradeException: You may get a different
> result due to the upgrading of Spark 3.0:
> writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z
> into Parquet INT96
> files can be dangerous, as the files may be read by Spark 2.x or legacy
> versions of Hive
> later, which uses a legacy hybrid calendar that is different from Spark
> 3.0+'s Proleptic
> Gregorian calendar. See more details in SPARK-31404. You can set
> spark.sql.parquet.int96RebaseModeInWrite to 'LEGACY' to
> rebase the datetime values w.r.t. the calendar difference during writing,
> to get maximum
> interoperability. Or set spark.sql.parquet.int96RebaseModeInWrite to
> 'CORRECTED' to write the datetime values as it is,
> if you are 100% sure that the written files will only be read by Spark
> 3.0+ or other
> systems that use Proleptic Gregorian calendar.
>
>
> Kind Regards,
> Sachit Murarka
>