You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2025/02/24 09:23:55 UTC
[I] Minify Kernel [arrow-rs]
tustvold opened a new issue, #7186:
URL: https://github.com/apache/arrow-rs/issues/7186
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
Often people want to store arrow data in memory, and in some cases are potentially willing to pay a premium for reduced memory usage.
Currently various array types can have "bloated" memory footprints, for example:
* Arrays can be sliced, with potentially unreferenced data buffers and child arrays
* Dictionaries can contain duplicate entries
* View arrays can contain unreferenced data
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
I would like to propose a minify kernel in arrow-select that allows performing this minification. It should take a non-exhaustive/build-pattern `MinifyOptions` struct to allow controlling how this logic is performed. Ideally it would be possible to move the logic that currently resides in the IPCWriter over to use this kernel.
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-4847684376
That is a pretty nice / complete story @Jefffrey 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-4784980915
> also in minify some people might want to cleanup all non empty nulls and reduce the unused space but this specific option as oppose to only clean top level is different that only match for some data types.
The way I read the original issue, ideally this behaviour would be composable via the options. e.g. could specify only want to zero size nulls but don't minimise sliced data, so only lists/strings would be affected.
(Though in implementation it could get a bit tricky)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-4783620495
> also, if minify is for reduce memory, in case the buffer is shared, calling minify will just increase memory as it will require data copy (which is still valuable)
I think this is the case for almost all kernels (they make a new array from an existing one, temporarily increasing memory usage)
> also in minify some people might want to cleanup all non empty nulls and reduce the unused space but this specific option as oppose to only clean top level is different that only match for some data types.
I was thinking minify would always create a new Array (with new allocations, etc) that consumes the minimum amount of new space (so for lists and other structured types it would avoid copying unused data for child arrays, as proposed in https://github.com/apache/arrow-rs/pull/9970).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "adriangb (via GitHub) via github" <gi...@arrow.apache.org>.
adriangb commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-6788428732
This came up again when looking at spilling and the various places in datafusion we want to perform this sort of operation. The structure proposed by @Jefffrey sounds good to me. I don't see any blockers to implementing this. I'd rather have it in 1 place in arrow-rs w/ some options than have it organically re-appear in multiple places in datafusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-4831596088
The implementation **will** be tricky and can make it harder for the user to understand how each config will affect the other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "Jefffrey (via GitHub) via github" <gi...@arrow.apache.org>.
Jefffrey commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-6798071006
I was planning to take this on but time got away from me; I probably won't do it myself but can review, so unassigning myself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-4843376782
I'm thinking along the lines like this:
```rust
// default = all true
struct MinifyOptions {
// general
minimize_all_buffers: bool,
recursive: bool,
// specialized
zero_size_nulls: bool,
compact_views: bool,
propagate_null_mask: bool,
deduplicate_dictionary: bool,
compact_runs: bool,
compact_dense_union: bool,
}
```
### minimize_all_buffers
If any buffer that the array holds (values, null buffer) is sliced, copy into a new buffer with only the sliced portion to get rid of unreferenced data. Also can do minor optimization to change null buffer from `Some` to `None` if null count is 0 (could happen if slicing).
### recursive
If array is nested, apply same minify options to all children. This could get a bit tricky, but I'll get to that below.
### zero_size_nulls
For byte/list/map types, pretty much what https://github.com/apache/arrow-rs/pull/9970 is doing.
### compact_views
Performing gc on list/byte view types.
### propagate_null_mask
For struct & fixed size list, for the null slots in this parent types we can try rebuild the children to have nulls there too to ensure minimum possible memory footprint. For example if the child is a string array, we can ensure its null (or at least zero sized) where the parents have null slots.
### deduplicate_dictionary
For dictionaries, deduplicate the values and also remove any nulls in the values array (and instead specify at key level).
### compact_runs
For run arrays, ensure we have minimum possible runs (in case we have an array with two runs of the same value in succession).
### compact_dense_union
Ensure children arrays & offsets are minimal possible, similar to views.
## Interaction of options
Each of the specialized options are mutually exclusive from the others (without considering recursion yet). The only option that would affect them all would be `minimize_all_buffers`. It would be possible to have `minimize_all_buffers: false` but still specify `zero_size_nulls: true`. This would still have to rebuild some buffers of the variable types, but not necessarily all of the buffers (e.g. can copy null buffer as is), which is why the general option specifies **all** buffers.
Recursive option is a bit tricky since some of the specialized options would need to rebuild the child anyway, so need to ensure rebuild in proper order. e.g. if zero sizing nulls, could zero size nulls first then again minify the children (rebuild children twice) or minify children first and rely on that minification to automatically zero size the nulls, etc.
## Overall
Personally I'm not sure how concerned we should be about the granularity of the controls on this minification behaviour; I assume most use cases would be fine with defaults 🤔
In terms of overall memory footprint, as long as the docstring properly explains that whilst it can create an equivalent array with a smaller memory footprint, it does not necessarily reduce overall memory footprint since it does copy data into new buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-4781707387
Although I like the idea of minify kernel with builder pattern non-exhaustive options, some options are not relevant for all array types.
for example, the cleanup non empty nulls, this is not relevant to `Primitive/Struct/Boolean` and possibly more
also in minify some people might want to cleanup all non empty nulls and reduce the unused space but this specific option as oppose to only clean top level is different that only match for some data types.
also, if minify is for reduce memory, in case the buffer is shared, calling minify will just increase memory as it will require data copy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Minify Kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7186:
URL: https://github.com/apache/arrow-rs/issues/7186#issuecomment-4772147124
This came up as part of
- https://github.com/apache/arrow-rs/pull/9970
It also comes up regularly in DataFusion and we have various workarounds for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org