[GitHub] [iceberg] rdblue commented on pull request #1127: ORC: fix date metrics to adjust milliseconds to local timezone

[GitBox <gi...@apache.org>]

rdblue commented on pull request #1127:
URL: https://github.com/apache/iceberg/pull/1127#issuecomment-650416548


   Thanks for the summary, @edgarRd. I find this behavior concerning. I think it will help to cover some background, just to cover the context for specific comments.
   
   As a storage layer, Iceberg should always read and return the exact data value that was written. For example, with floating point values, there is no "close enough" and Iceberg should read the same bits that were written. That's why we use `assertEquals` instead of assertions that a value is within some floating point error.
   
   For date/time types, Iceberg should never modify a value. Concerns like translating from some time zone into the UTC value to store for `timestamptz` must be handled by processing engines, which handle calendars time zone offsets, and other concerns. Iceberg stores the UTC value without modification, and the data type tells the engine how to produce a value to write and consume a value that's read. This is why the internal representation is microseconds from the Unix epoch in UTC. That's unambiguous.
   
   It looks like ORC uses representations that are timezone-dependent in some cases, like returning, and possibly calculating, stats. I think that it is good that we don't use these in the Iceberg readers or writers, and my first approach to fixing this would be to avoid having data passed using those objects.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org