You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/04 08:04:49 UTC

[GitHub] [arrow-rs] crepererum opened a new issue #254: u64::MAX does not roundtrip through parquet

crepererum opened a new issue #254:
URL: https://github.com/apache/arrow-rs/issues/254


   **Describe the bug**
   `u64::MAX` gets truncated to 0 when storing to parquet and reading back.
   
   **To Reproduce**
   Add the following test:
   
   ```rust
   #[test]
   fn u64_min_max() {
       let values = Arc::new(UInt64Array::from_iter_values(vec![u64::MIN, u64::MAX]));
       one_column_roundtrip("u64_min_max_single_column", values, false);
   }
   ```
   
   **Expected behavior**
   All values either roundtrip correctly or some error is produced - at least they should not be silently truncated.
   
   **Additional context**
   Tested on commit `8f030db53d9eda901c82db9daf94339fc447d0db`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] crepererum commented on issue #254: u64::MAX does not roundtrip through parquet

Posted by GitBox <gi...@apache.org>.
crepererum commented on issue #254:
URL: https://github.com/apache/arrow-rs/issues/254#issuecomment-832463486


   BTW: I'm working on that issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] nevi-me closed issue #254: u64::MAX does not roundtrip through parquet

Posted by GitBox <gi...@apache.org>.
nevi-me closed issue #254:
URL: https://github.com/apache/arrow-rs/issues/254


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] crepererum commented on issue #254: u64::MAX does not roundtrip through parquet

Posted by GitBox <gi...@apache.org>.
crepererum commented on issue #254:
URL: https://github.com/apache/arrow-rs/issues/254#issuecomment-831964686


   Some more context: the logical type `u64` is stored as the physical "`i64` w/o sign bit". As per [format specs](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#unsigned-integers):
   
   > If a stored value is larger than the maximum allowed by the annotation, the behavior is not defined and can be determined by the implementation. Implementations must not write values that are larger than the annotation allows.
   
   so in theory we could map the logical range `(i64:MAX as u64)..u64::MAX` to the physical range `i64:MIN..0i64`. From looking at the C++ code I think (!!!) this is also what C++ is doing:
   
   https://github.com/apache/arrow/blob/0ee3b90cac8f8eb3bf512e51a7f941fcead026d9/cpp/src/parquet/arrow/reader_internal.cc#L307-L322
   
   aka: "reinterpret signed as unsigned ints".
   
   Then the only bit that might be tricky (or not in which case a to-be-written test will pass) is to get the statistics right.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org