[GitHub] [arrow-rs] tustvold commented on issue #2320: optimize the validation of decimal: compare data using the byte array instead of i128 or string

[GitBox <gi...@apache.org>]

tustvold commented on issue #2320:
URL: https://github.com/apache/arrow-rs/issues/2320#issuecomment-1207186118

   I find it unexpected that a variable length byte array outperforms a fixed-length i128?
   
   > reduce construct the decimal128 and decimal258 which contain the useless precision/scale for validation.
   
   I think it would help to take a step back here, as I at least am very confused as to what is being validated when. For most array types we have an invariant that `ArrayData` only contains valid data, this means we can elide validation on access. This does not appear to be the case for `DecimalArray`, which I personally think is a bug.
   
   For example, `BasicDecimal::new` is not `unsafe`, `BasicDecimalArray::from_fixed_size_binary_array` doesn't perform validation, etc... If we can make their an invariant of valid data, and consistently enforce this, we can then look to elide validation checks on access


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org