[GitHub] [hudi] cxzl25 commented on pull request #5287: [HUDI-3849] AvroDeserializer supports AVRO_REBASE_MODE_IN_READ configuration

[GitBox <gi...@apache.org>]

cxzl25 commented on PR #5287:
URL: https://github.com/apache/hudi/pull/5287#issuecomment-1095301076

   Using Spark3.2.0+Hudi 0.11 Snapshot, the following exception was encountered.
   This problem can be solved if `AvroDeserializer` supports SQLConf configuration.
   
   ```java
   You can set the SQL config
   'spark.sql.avro.datetimeRebaseModeInRead' or the datasource option 'datetimeRebaseMode' to 'LEGACY' to rebase the datetime values
   w.r.t. the calendar difference during reading. To read the datetime values as it is,
   set the SQL config 'spark.sql.avro.datetimeRebaseModeInRead' or the datasource option 'datetimeRebaseMode' to 'CORRECTED'.
          
   	at org.apache.spark.sql.errors.QueryExecutionErrors$.sparkUpgradeInReadingDatesError(QueryExecutionErrors.scala:474)
   	at org.apache.spark.sql.execution.datasources.DataSourceUtils$.newRebaseExceptionInRead(DataSourceUtils.scala:161)
   	at org.apache.spark.sql.execution.datasources.DataSourceUtils$.$anonfun$creteTimestampRebaseFuncInRead$1(DataSourceUtils.scala:203)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$7(AvroDeserializer.scala:145)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$newWriter$7$adapted(AvroDeserializer.scala:143)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1(AvroDeserializer.scala:360)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$1$adapted(AvroDeserializer.scala:356)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2(AvroDeserializer.scala:378)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$getRecordWriter$2$adapted(AvroDeserializer.scala:374)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.$anonfun$converter$4(AvroDeserializer.scala:85)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.AvroDeserializer.deserialize(AvroDeserializer.scala:103)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.Spark3HoodieAvroDeserializer.doDeserialize(Spark3HoodieAvroDeserializer.scala:37)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.HoodieAvroDeserializerTrait.deserialize(HoodieAvroDeserializerTrait.scala:29)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.HoodieAvroDeserializerTrait.deserialize$(HoodieAvroDeserializerTrait.scala:28)
   	at org.apache.hudi.spark.org.apache.spark.sql.avro.Spark3HoodieAvroDeserializer.deserialize(Spark3HoodieAvroDeserializer.scala:24)
   	at org.apache.hudi.HoodieMergeOnReadRDD$$anon$3.hasNext(HoodieMergeOnReadRDD.scala:262)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org