[GitHub] [iceberg] davseitsev commented on issue #2122: Spark can't read Iceberg table created from Presto

[GitBox <gi...@apache.org>]

davseitsev commented on issue #2122:
URL: https://github.com/apache/iceberg/issues/2122#issuecomment-767443068


   According to Spark docs `TimestampType` represents values comprising values of fields year, month, day, hour, minute, and second, with the session local time-zone. The timestamp value represents an absolute point in time.  So it's more like `LocalDateTime`. Why can't  Iceberg expose `timestamp without timezone` as Spark `timestamp`?
   
   As far as I understand the problem is that Spark has only one timestamp type but Iceberg has two types which causes some inconsistency in type mapping between Spark and Iceberg.
   
   In my opinion it could be solved in a following way:
   1. For read access there is no problem, both Iceberg types (`timestamp`, `timestamptz`) could be exposed as Spark `timestamp`.
   2. For insert there is no problem because we know target type defined in the table metadata and can choose correct one
   3. For table creation we have inconsistency which could be solved by configuration property. Spark has such kind of property to choose how to store timestamp fields in parquet (`spark.sql.parquet.outputTimestampType=(INT96 | TIMESTAMP_MICROS | TIMESTAMP_MILLIS)`). Why can't we have such logic for Iceberg?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org