You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/27 08:45:38 UTC
[GitHub] [arrow-datafusion] tustvold opened a new issue, #4386: Make Binary Dictionary Operations Optional
tustvold opened a new issue, #4386:
URL: https://github.com/apache/arrow-datafusion/issues/4386
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The dyn_cmp_dict and dyn_arith_dict features of arrow enable binary operations involving dictionary arrays and other dictionary arrays, or scalar arrays.
They are, however, extremely expensive both from a compilation time, and code size perspective. As the kernels must be generated for the combinatorial explosion of all dictionary key and value types.
They are also exceedingly rare in practice, as almost all queries instead use thr scalar variant, i.e. add scalar to dictionary, compare dictionary against scalar, etc...
**Describe the solution you'd like**
I would like a feature flag, e.g. binary_dict_op, that is not enabled by default, and enables the arrow features. The three or so tests that happen to need this, can then be gated on this feature flag
**Describe alternatives you've considered**
We could not do this
**Additional context**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] retikulum commented on issue #4386: Make Binary Dictionary Operations Optional
Posted by GitBox <gi...@apache.org>.
retikulum commented on issue #4386:
URL: https://github.com/apache/arrow-datafusion/issues/4386#issuecomment-1331748181
Hi. I added this on purpose (but without knowing it is extremely expensive) to pass `test_dictionary_type_to_array_coersion` test case. The following error was generated before enabling it:
```rust
Error: ArrowError(CastError("Comparing array of type Dictionary(Int32, Utf8) with array of type Dictionary(Int32, Utf8) requires \"dyn_cmp_dict\" feature"))
```
It seems great to me for enabling the feature flag rather than enabling it by default. Thanks for noticing me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold closed issue #4386: Make Binary Dictionary Operations Optional
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #4386: Make Binary Dictionary Operations Optional
URL: https://github.com/apache/arrow-datafusion/issues/4386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org