[GitHub] [iceberg] pvary commented on pull request #6426: Flink: add fixed field type for DataGenerators test util

[GitBox <gi...@apache.org>]

pvary commented on PR #6426:
URL: https://github.com/apache/iceberg/pull/6426#issuecomment-1352618114

   > @pvary regarding your other comments, I am not sure how to proceed yet.
   > 
   > > Null values
   > 
   > for null values, would optional primitive types at top level be enough?
   
   I would use a schema, where the fields are optional. I would like to test that null primitive values, and null map/list/struct is handled correctly.
   Also I have seen cases where the null value inside a struct was causing errors. So I would add either a different test, or one field added to the existing records where the value is null
   
   > > Edge cases, like year differences based on TZ, or non valid timestamps based on TZ
   > 
   > not very clear
   
   Test  records where the timestamp and the same timestamp in UTC are different years.
   Test records where the timestamp is invalid because of the summer time / winter time change - on second thought, I am not entirely sure about this. We might need a quick check if it is possible to do or the java classes will prevent the creation of these timestamps.
   Another idea is, very old, or timestamps in the far future. I have seen issues with timestamp before epoch, and also timestamps before the calendar change (before around 1400?). Also I have seen issues with timestamps around year 9999, which moved to 10000 when converted to UTC.
   
   What is the range of the timestamps supported by Iceberg anyways.
   
   That said, I would create 2 sets of test records:
   - common tests
   - extensive tests - with the edge cases
   And run the tests with extensive tests only in specific cases
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org