You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "vinishjail97 (via GitHub)" <gi...@apache.org> on 2025/04/01 20:59:40 UTC

[I] Hudi 0.x writes INT64 in parquet for Avro LogicalTypes.LocalTimestampMillis, needs 1.x upgrade [incubator-xtable]

vinishjail97 opened a new issue, #672:
URL: https://github.com/apache/incubator-xtable/issues/672

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Please describe the bug 🐞
   
   When adding supporting for timestamp_ntz for delta and iceberg, found a bug where hudi 0.x is writing INT64 for LocalTimestampMillis and LocalTimestampMicros.  
   
   ```
   ############ Column(timestamp_millis_nullable_field) ############
   name: timestamp_millis_nullable_field
   path: timestamp_millis_nullable_field
   max_definition_level: 1
   max_repetition_level: 0
   physical_type: INT64
   logical_type: Timestamp(isAdjustedToUTC=true, timeUnit=milliseconds, is_from_converted_type=false, force_set_converted_type=false)
   converted_type (legacy): TIMESTAMP_MILLIS
   compression: UNCOMPRESSED (space_saved: 0%)
   
   ############ Column(timestamp_micros_nullable_field) ############
   name: timestamp_micros_nullable_field
   path: timestamp_micros_nullable_field
   max_definition_level: 1
   max_repetition_level: 0
   physical_type: INT64
   logical_type: Timestamp(isAdjustedToUTC=true, timeUnit=microseconds, is_from_converted_type=false, force_set_converted_type=false)
   converted_type (legacy): TIMESTAMP_MICROS
   compression: UNCOMPRESSED (space_saved: 0%)
   
   ############ Column(timestamp_local_millis_nullable_field) ############
   name: timestamp_local_millis_nullable_field
   path: timestamp_local_millis_nullable_field
   max_definition_level: 1
   max_repetition_level: 0
   physical_type: INT64
   logical_type: None
   converted_type (legacy): NONE
   compression: UNCOMPRESSED (space_saved: 0%)
   
   ############ Column(timestamp_local_micros_nullable_field) ############
   name: timestamp_local_micros_nullable_field
   path: timestamp_local_micros_nullable_field
   max_definition_level: 1
   max_repetition_level: 0
   physical_type: INT64
   logical_type: None
   converted_type (legacy): NONE
   compression: UNCOMPRESSED (space_saved: 0%)
   ```
   
   https://github.com/apache/incubator-xtable/pull/647/files#r2023648267 
   
   Hudi 1.x has fixed the problem, adding this GH issue to revert the schema mapping back to timestamp_ntz in AvroSchemaConverter.
   https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala#L152 
   
   ### Are you willing to submit PR?
   
   - [x] I am willing to submit a PR!
   - [ ] I am willing to submit a PR but need help getting started!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org