You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/04 17:12:55 UTC
[GitHub] [hudi] SubashRanganathan opened a new issue, #6305: Hudi Delta Streamer unable to read Older Dates
SubashRanganathan opened a new issue, #6305:
URL: https://github.com/apache/hudi/issues/6305
Hudi Delta Streamer unable to read dates that are older than older than 1900-01-01.The workaround fix for this is to set the following spark configurations :
spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED.
This options work fine when I try to create Hudi table with PySpark. However, when I run CDC process with DeltaStreamer I still continue to get this error. Please note that I cannot use the hudi- transformer class becuase for transformer class to be applied, delta streamer should read the source files. Delta streamer is not able to read the source files.
The error message is "An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
You may get a different result due to the upgrading of Spark 3.0: reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from Parquet INT96 files can be ambiguous,
as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+s Proleptic Gregorian calendar.
See more details in SPARK-31404.
You can set spark.sql.legacy.parquet.int96RebaseModeInRead to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading.
Or set spark.sql.legacy.parquet.int96RebaseModeInRead to 'CORRECTED' to read the datetime values as it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6305: Hudi Delta Streamer unable to read Older Dates
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6305:
URL: https://github.com/apache/hudi/issues/6305#issuecomment-1229261834
@brskiran1 : can you try out https://github.com/apache/hudi/pull/6352 and let us know if the issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] brskiran1 commented on issue #6305: Hudi Delta Streamer unable to read Older Dates
Posted by GitBox <gi...@apache.org>.
brskiran1 commented on issue #6305:
URL: https://github.com/apache/hudi/issues/6305#issuecomment-1216927166
@alexeykudinkin @nsivabalan can you please let me know what should be tried to see if the issue is resolved? or can you please. I have responded in the ticket #6278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6305: Hudi Delta Streamer unable to read Older Dates
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6305:
URL: https://github.com/apache/hudi/issues/6305#issuecomment-1216129737
CC @rmahindra123
gentle ping @alexeykudinkin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #6305: Hudi Delta Streamer unable to read Older Dates
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #6305:
URL: https://github.com/apache/hudi/issues/6305#issuecomment-1299697380
Since we got the patch landed, going ahead and closing the issue out.
feel free to open new issue if you need further assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #6305: Hudi Delta Streamer unable to read Older Dates
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #6305:
URL: https://github.com/apache/hudi/issues/6305#issuecomment-1207684814
@alexeykudinkin based on the discussion in Slack, is there a solution to it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] brskiran1 commented on issue #6305: Hudi Delta Streamer unable to read Older Dates
Posted by GitBox <gi...@apache.org>.
brskiran1 commented on issue #6305:
URL: https://github.com/apache/hudi/issues/6305#issuecomment-1207143961
@alexeykudinkin one more issue here related to dates older than 1900-01-01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #6305: Hudi Delta Streamer unable to read Older Dates
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #6305: Hudi Delta Streamer unable to read Older Dates
URL: https://github.com/apache/hudi/issues/6305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #6305: Hudi Delta Streamer unable to read Older Dates
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on issue #6305:
URL: https://github.com/apache/hudi/issues/6305#issuecomment-1216920765
@nsivabalan let's merge this one w/ https://github.com/apache/hudi/issues/6278
I've put up https://github.com/apache/hudi/pull/6352 to address this, but didn't hear back from the original reporter(s) whether they were able to try it out and if it resolved the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org