You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Simeon Simeonov (Jira)" <ji...@apache.org> on 2021/04/19 01:59:00 UTC
[jira] [Comment Edited] (SPARK-35097) Add column name to
SparkUpgradeException about ancient datetime
[ https://issues.apache.org/jira/browse/SPARK-35097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324643#comment-17324643 ]
Simeon Simeonov edited comment on SPARK-35097 at 4/19/21, 1:58 AM:
-------------------------------------------------------------------
[~maxgekk] thanks for creating this issue; it came from a problem we discovered.
Reporting the column name alone in the exception message would not be sufficient for fast root cause analysis in many situations. Imagine the case where the column name is something like {{date}}, which is common across many tables, and a job that uses lots of different tables. To aid users, one needs to narrow down the source of the problem by either (a) providing user stack trace information or, if that is for some reason impossible or very difficult, (b) provide information about the Parquet source with the issue (path, plan info, etc.).
[~angerszhuuu] can either (a) or (b) above be added to your PR?
was (Author: simeons):
[~maxgekk] thanks for creating this issue; it came from a problem we discovered.
Reporting the column name alone in the exception message would not be sufficient for fast root cause analysis in many situations. Imagine the case where the column name is something like {{date}}, which is common across many tables, and a job that uses lots of different tables. To aid users, one needs to narrow down the source of the problem by either (a) providing user stack trace information or, if that is for some reason impossible or very difficult, (b) provide information about the Parquet parquet source with the issue (path, plan info, etc.).
[~angerszhuuu] can either (a) or (b) above be added to your PR?
> Add column name to SparkUpgradeException about ancient datetime
> ---------------------------------------------------------------
>
> Key: SPARK-35097
> URL: https://issues.apache.org/jira/browse/SPARK-35097
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Max Gekk
> Priority: Major
>
> The error message:
> {code:java}
> org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from Parquet files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.legacy.parquet.datetimeRebaseModeInRead to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. Or set spark.sql.legacy.parquet.datetimeRebaseModeInRead to 'CORRECTED' to read the datetime values as it is.
> {code}
> doesn't have any clues of which column causes the issue. Need to improve the message and add column name to it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org