You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ariesdevil (via GitHub)" <gi...@apache.org> on 2024/04/09 04:03:12 UTC
[PR] feat: support slice view data on writing IPC [arrow-rs]
ariesdevil opened a new pull request, #5610:
URL: https://github.com/apache/arrow-rs/pull/5610
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Part of https://github.com/apache/arrow-rs/issues/5506.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Support slice view data on writing IPC.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Support slice view data on writing IPC.
# Are there any user-facing changes?
No.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support slice view data on writing IPC [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #5610:
URL: https://github.com/apache/arrow-rs/pull/5610#issuecomment-2046111670
> I believe the intention of this code is to avoid writing out data buffers that aren't referenced by any views, perhaps as a result of slicing.
That is my interpretation too -- @ariesdevil can you confirm?
> I think this is probably better handled as a more general "compact" operation, that also handles pruning the data buffers themselves. I believe this is tracked by https://github.com/apache/arrow-rs/issues/5513
Yes, that is my suggestion. The rationale for a separate gc operation is:
1. Since it is potentially expensive I think the user should decide when to do the check, rather than on each IPC call. I think some applications might have different tradeoffs between compacting / pruning and saving IPC space.
2. `gc` is more generally useful for operations other than IPC (consolidating memory down after a filter, for example)
Having a variation of `gc` that simply prunes unreferenced buffers (but not copy) might also be interesting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support slice view data on writing IPC [arrow-rs]
Posted by "ariesdevil (via GitHub)" <gi...@apache.org>.
ariesdevil commented on PR #5610:
URL: https://github.com/apache/arrow-rs/pull/5610#issuecomment-2045126666
Maybe we can just slice views here and hand over the data buffer to gc for processing. cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support slice view data on writing IPC [arrow-rs]
Posted by "ariesdevil (via GitHub)" <gi...@apache.org>.
ariesdevil commented on PR #5610:
URL: https://github.com/apache/arrow-rs/pull/5610#issuecomment-2046501107
Make sense, thanks @alamb @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support slice view data on writing IPC [arrow-rs]
Posted by "ariesdevil (via GitHub)" <gi...@apache.org>.
ariesdevil closed pull request #5610: feat: support slice view data on writing IPC
URL: https://github.com/apache/arrow-rs/pull/5610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: support slice view data on writing IPC [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5610:
URL: https://github.com/apache/arrow-rs/pull/5610#issuecomment-2044682118
I believe the intention of this code is to avoid writing out data buffers that aren't referenced by any views, perhaps as a result of slicing. I think this is probably better handled as a more general "compact" operation, that also handles pruning the data buffers themselves. I believe this is tracked by https://github.com/apache/arrow-rs/issues/5513
I'm not sure whether the IPC writer should be doing anything smart here?
Perhaps @alamb has thoughts on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org