[I] arrow-ord: support partitioning on nested types [arrow-rs]

["asubiotto (via GitHub)" <gi...@apache.org>]

asubiotto opened a new issue, #7130:
URL: https://github.com/apache/arrow-rs/issues/7130

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   Using datafusion to run a window function partitioned by a nested data type column results in a nested comparison error during execution:
   ```
   InvalidArgumentError("Nested comparison: Struct([Field { name: \"f1\", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) IS DISTINCT FROM Struct([Field { name: \"f1\", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]) (hint: use make_comparator instead)")
   ```
   This is a feature request to add nested partitioning support to the partition kernel:
   https://github.com/apache/arrow-rs/blob/d4b9482f5dee47a0f2f7afc129db83b8ac8df406/arrow-ord/src/partition.rs#L126
   
   **Describe the solution you'd like**
   `partition` shells out to `distinct`, which does not support nested comparisons:
   https://github.com/apache/arrow-rs/blob/d4b9482f5dee47a0f2f7afc129db83b8ac8df406/arrow-ord/src/cmp.rs#L179-L181
   My proposal is to add a check for nested type columns and use `make_comparator` to check for value distinctness instead.
   
   **Describe alternatives you've considered**
   - Expanding nested array fields to primitive arrays. This seems costly
   - Allowing nested comparisons in `compare_op` for certain op types where null ordering semantics don't matter (which is the case here I think). This is another option, but it seems like the proposed approach is a more general solution which can be swapped out if performance becomes an issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] arrow-ord: support partitioning on nested types [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb closed issue #7130: arrow-ord: support partitioning on nested types
URL: https://github.com/apache/arrow-rs/issues/7130


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org