[GitHub] [arrow-rs] tustvold opened a new issue #1217: Dictionary Uniqueness and Sortedness

[GitBox <gi...@apache.org>]

tustvold opened a new issue #1217:
URL: https://github.com/apache/arrow-rs/issues/1217


   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   A schema Field currently has a `dict_is_ordered` field. Much like `dict_id` this is non-trivial to use in practice (#1206), as it is part of the schema which may need to remain constant across multiple RecordBatch, that would wish to use different dictionaries. It is also not visible to compute kernels.
   
   There doesn't appear to be a notion of if a dictionary contains unique values.
   
   **Describe the solution you'd like**
   
   We would ideally like to associated dictionary properties with the dictionary arrays themselves. This would allow compute kernels to exploit their properties, in addition to avoiding schema incompatibility issues.
   
   There is an assumption in many places that the arrays themselves are sugar on top of `ArrayData`, with it possible to construct a DictionaryArray from a `ArrayData`. As such I would like to propose:
   
   * Remove dict_is_ordered from Field
   * Add dict_is_ordered bool to ArrayData (default false)
   * Add dict_is_unique bool to ArrayData (default false)
   
   The various compute kernels can then progressively be updated to exploit these properties.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org