[GitHub] [hudi] szingerpeter commented on issue #7724: failed to read timestamp from hive

["szingerpeter (via GitHub)" <gi...@apache.org>]

szingerpeter commented on issue #7724:
URL: https://github.com/apache/hudi/issues/7724#issuecomment-1400435894

   i tried setting
   ```
   spark.sql.avro.datetimeRebaseModeInWrite=LEGACY
   ```
   
   as well, but i get the same error:
   ```
   Caused by: java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot be cast to org.apache.hadoop.hive.serde2.io.TimestampWritable
           at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.getPrimitiveJavaObject(WritableTimestampObjectInspector.java:39)
   ```
   
   the reason i need to set `spark.sql.avro.datetimeRebaseModeInWrite` is, because i was getting the following error:
   ```
   org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: 
   writing dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z into Avro
   files can be dangerous, as the files may be read by Spark 2.x or legacy versions of Hive
   later, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic
   Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.avro.datetimeRebaseModeInWrite to 'LEGACY' to
   rebase the datetime values w.r.t. the calendar difference during writing, to get maximum
   interoperability. Or set spark.sql.avro.datetimeRebaseModeInWrite to 'CORRECTED' to write the datetime values as it is,
   if you are 100% sure that the written files will only be read by Spark 3.0+ or other
   systems that use Proleptic Gregorian calendar.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org