[GitHub] [arrow-rs] crepererum commented on issue #254: u64::MAX does not roundtrip through parquet

[GitBox <gi...@apache.org>]

crepererum commented on issue #254:
URL: https://github.com/apache/arrow-rs/issues/254#issuecomment-831964686


   Some more context: the logical type `u64` is stored as the physical "`i64` w/o sign bit". As per [format specs](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#unsigned-integers):
   
   > If a stored value is larger than the maximum allowed by the annotation, the behavior is not defined and can be determined by the implementation. Implementations must not write values that are larger than the annotation allows.
   
   so in theory we could map the logical range `(i64:MAX as u64)..u64::MAX` to the physical range `i64:MIN..0i64`. From looking at the C++ code I think (!!!) this is also what C++ is doing:
   
   https://github.com/apache/arrow/blob/0ee3b90cac8f8eb3bf512e51a7f941fcead026d9/cpp/src/parquet/arrow/reader_internal.cc#L307-L322
   
   aka: "reinterpret signed as unsigned ints".
   
   Then the only bit that might be tricky (or not in which case a to-be-written test will pass) is to get the statistics right.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org