You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/29 15:11:58 UTC
[GitHub] [arrow-rs] alamb opened a new issue, #3407: ArrayData`get_slice_memory_size` or similar
alamb opened a new issue, #3407:
URL: https://github.com/apache/arrow-rs/issues/3407
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
As part of implementing an Arrow flight client, we have observed that there is a maximum message size for gRPC messages, and thus while encoding the Arrow flight client has a way to limit the size of the messages (see https://github.com/apache/arrow-rs/pull/3391)
However, apparently there is currently no way to get the memory size used by a sliced (aka the array only points to a portion of the overall underlying allocation), which will result in overly conservative re-splitting in the arrow-flight client
**Describe the solution you'd like**
As suggested by @tustvold on https://github.com/apache/arrow-rs/pull/3391#discussion_r1058288317
> It is worth noting that this method does not take into account already sliced arrays, we likely need to add a `get_slice_memory_size` or something to `ArrayData` to provide this, it should be relatively straightforward
**Describe alternatives you've considered**
We could leave the users of the client to worry about batch splitting
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #3407: ArrayData`get_slice_memory_size` or similar
Posted by GitBox <gi...@apache.org>.
tustvold closed issue #3407: ArrayData`get_slice_memory_size` or similar
URL: https://github.com/apache/arrow-rs/issues/3407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] askoa commented on issue #3407: ArrayData`get_slice_memory_size` or similar
Posted by GitBox <gi...@apache.org>.
askoa commented on issue #3407:
URL: https://github.com/apache/arrow-rs/issues/3407#issuecomment-1376160042
I'll give this a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #3407: ArrayData`get_slice_memory_size` or similar
Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #3407:
URL: https://github.com/apache/arrow-rs/issues/3407#issuecomment-1381538537
`label_issue.py` automatically added labels {'arrow'} from #3501
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #3407: ArrayData`get_slice_memory_size` or similar
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3407:
URL: https://github.com/apache/arrow-rs/issues/3407#issuecomment-1376383261
Thank you @askoa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #3407: ArrayData`get_slice_memory_size` or similar
Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #3407:
URL: https://github.com/apache/arrow-rs/issues/3407#issuecomment-1381538589
`label_issue.py` automatically added labels {'arrow-flight'} from #3391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org