You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/23 08:58:21 UTC

[GitHub] [spark] gengliangwang commented on pull request #30902: [SPARK-33888][SQL] Add support for TimeMillis logicalType to TimestampType

gengliangwang commented on pull request #30902:
URL: https://github.com/apache/spark/pull/30902#issuecomment-750019909


   Hi @saikocat ,
   
   thanks for reporting the issue. 
   As per Avro spec https://avro.apache.org/docs/1.8.2/spec.html#Time+%28millisecond+precision%29:
   ```
   Time (microsecond precision)
   The time-micros logical type represents a time of day, with no reference to a particular calendar, time zone or date, with a precision of one microsecond.
   
   A time-micros logical type annotates an Avro long, where the long stores the number of microseconds after midnight, 00:00:00.000000.
   
   Timestamp (millisecond precision)
   The timestamp-millis logical type represents an instant on the global timeline, independent of a particular time zone or calendar, with a precision of one millisecond.
   
   A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds from the unix epoch, 1 January 1970 00:00:00.000 UTC.
   ```
   So, we can't map the time-millis/time-micros type to Timestamp type. Instead, we should map it as `CalendarIntervalType`.
   In Spark, `CalendarIntervalType` is internal type and we can't use it for table creatation. So we didn't support the Avro logical type time-millis and time-micros.
   In the current stage, I am +0 for supporting it.
   cc @cloud-fan 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org